January 28, 2016 Simon Raper

From Zero to D3

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

Some time ago I was asked to run a short course to take analysts from javascript newbies to being capable of putting together interactive charts using d3. That’s a big ask, from them and from me! However it did work, or at the very least it set people off in the right direction while giving them some immediate and satisfying results.

I’m sharing the materials here as they may be of use to anyone who wants to get up and running quickly.

It’s two pronged attack. In the first session we go bottom up and run through how html, javascript, css and svgs all fit together. We then use that knowledge to build a simple scatterplot. In the second session we go top down using what we learnt from our basic example to unpick some more complex code and substitute in our own data. This gets us quite a long way.

I’m sharing the videos I made, the presentation that I talk though and the exercises that we went through. This is the material from session one. Session two to follow shortly.

Details of more courses run by Coppelia can be found here. If you are interested in onsite training please get in touch ([email protected])

The videos

Please watch the following short videos before the workshop class.

The javascript revolution

 

How it all fits together

 

The slides

Download the full presentation or watch the full screen version here

 

The Workshop Tasks

Download the pdf from here.

We will be building a simple interactive scatterplot using d3. Although it is simple it will have three of the features very common to javascript visualisations:

  • It will be interactive
  • It will use animation
  • By using tooltips it will be capable of representing additional information without cluttering the graph.

You can see what we are working towards here

We will assume no HTML or javascript skills so we’ll start slowly and build things up.

Please watch the video: How it all fits together before going any further

 

Step 1: Introducing JS fiddle

JS fiddle allows us to experiment with html and javascript without going through the bother of setting up a coding environment.

Note be sure to hit update to regularly save your work!

  1. Open up jsfiddle and log in. For your reference you can see all my fiddles here
  2. Click in the top right corner to create a new fiddle
  3. You now have four windows
    1. Top left is for your html
    2. Top right is for your css
    3. Bottom left is for your javascript
    4. Bottom right is your output
  4. Let’s give it a go. Type <h1>Demo Scatterplot</h1> and hit run.
  5. Now let’s style it by pasting code in the css section. The web is full of nice styles to borrow. I got the following from here
     

     h1 {
         color: #1abc9c;
         font-family:  'Raleway';
         font-weight: normal;
         font-size: 1.75em;
         letter-spacing: .2em;
         line-height: 1.1em;
         margin:0px;
         text-align: center;
         text-transform: uppercase;
     }
    

 

Step 2: Setting up a page using divs

Now we really need a more sophisticated layout if we are going to incorporate things like drop downs and info panels so let’s use the div html tag to to divide up our page. Add the following to the html window

<div id ="container">
    <div id ="header">
        <h1>Demo Scatterplot</h1>
    </div>
    <div id ="sidebar">
    </div>
    <div id ="content">
    </div>
    <div id ="footer">
    </div>
</div>

Notice the nested structure of the divs. We are creating something that looks like this

DivStructure

Hit run.

Nothing much has happened but that’s because we need to style our divs. It’s part of the genius of HTML that so much of the design is accomplished via css. So replace the css code with this:

h1 {
    color: #ffffff;
    font-family:  'Raleway';
    font-weight: normal;
        font-size: 1.75em;
    letter-spacing: .2em;
    line-height: 1.1em;
    margin:0px;
    text-align: center;
    text-transform: uppercase;
}
#container {
    width:800px;
}
#header {
    width:800px;
    height: 50px;
    padding: 5px;
    background: #1abc9c;
}
#sidebar {
    height:400px;
    width:150px;
    background: #1abc9c;
    float:left;
}
#content {
    background-color:#ffffff;
    height:400px;
    width:650px;
    float:left;
}
#footer {
    width:800px;
    height: 50px;
    background: #1abc9c;
    float:left;
}

This has set the properties of the divs including their dimensions, colour and how they stack together (using the float property). It’s not very sophisticated stylistically but it is starting more like the end product.

 

Step 3: Introducing SVGs

We introduced the concept of SVGs in the theory video. Now let’s add them to our page (ultimately we will be manipulating them using d3)

Inside our content div we add

<div id ="content">
    <svg height="400" width="650">
      <circle cx="100" cy="100" r="10"/>
      <circle cx="200" cy="250" r="10"/>
      <circle cx="300" cy="180" r="10"/>
    </svg>
</div>

Note the x, y co-ordinates (cx and cy) are given as off sets from the top lefthand corner of the svg box.

Just as we did with the header element we can style our circle using css. Add the following code to the css window

circle {
    stroke: #1abc9c;
    stroke-width: 2;
    fill: white;
}

Your work should now look like this

 

Step 4: Introducing json

This is all very well but we would like our circle positions to be related to some data so that when change the data the circles move. We are going to include this data in json form in the javascript window.

circle_data = [{"x": 100,
               "y": 100,
               "r": 10},
              {"x": 200,
               "y": 250,
               "r": 10},
              {"x": 300,
               "y": 180,
               "r": 10}]

 

Step 5: And now d3

d3 is going to allow us to connect the data to the svg circles but first you need to ensure that you have access to the d3 library. To do this go to Frameworks and Extensions on the lefthand side of jsfiddle and select the D3 library.

Next, delete the contents of the svg in your html code. With d3 you are going add this dynamically. So your code should now look like this:

    <div id ="content">
        <svg height="400" width="650">
        </svg>
    </div>

Now add the following code to your javascript window

svg = d3.select("svg");
diag_circles = svg.selectAll("circle")
diag_circles.data(circle_data)
.enter()
.append("circle")
.attr("cx", function(d){return d.x})
.attr("cy", function(d){return d.y})
.attr("r", function(d){return d.r});

Let’s dissect the code.

  • The first line svg = d3.select("svg"); picks out the svg element that we left our html and assigns it to the svg variable.
  • The next line diag_circles = svg.selectAll("circle") selects all the circles in svg (note there aren’t an yet so this might seem a bit weird but think of it as an empty container)
  • The next part uses a technique known as chaining. It is easier to understand if you flatten the code so it looks like this diag_circles.data(circle_data).enter().append("circle").attr("cx", function(d){return d.x}).attr("cy", function(d){return d.y}).attr("r", function(d){return d.r}); Wherever you see a dot a method is being applied to the object that has been produced up to that point. so
    1. We start off with the diag_circles object (all the svg circles even though there are none yet!) and apply the data method to it which binds circle data to the svgs, thus producing a new object
    2. We apply the enter method to this new object (we’ll come back to what this does later but for now note that it in turn creates a new object.)
    3. Next we apply the append method that creates svg circles
    4. Finally we successively use the attr method set the attributes of the circles. So for example attr("cx", function(d){return d.x}) sets the x co-ord of each circle by going off to the data set and bringing back the value corresponding to x. It takes the first x in the data and uses it to set the x co-ord of the first circle and so on.

If that all sounds a bit confusing then it’s because it is. D3 contains some counterintuitive ideas that pay off in the long run but are hard to get your head around initially. If you think of it as mapping the data onto the properties of svgs then you’ve basically got it!

So now try altering the data and hitting run. The circles should move.

Note if you are using Chrome you can as usual right click on the circles and select inspect element to see the svg that has been created by d3.

Your work should now look like this.

 

Step 6: Some more complex data

To make things more interesting we will start using a data set with more than two dimensions. Here is some data I’ve extracted from the Top Trumps horror set (many happy eighties memories).

240810cards

The data will be stored in json format in your javascript window

horror = [{
    "name": "ape man",
        "physicalstrength": 20,
        "fearfactor": 7,
        "killingpower": 12,
        "horrorrating": 4,
        "cluster": 1
}, {
    "name": "godzilla",
        "physicalstrength": 3,
        "fearfactor": 5,
        "killingpower": 4,
        "horrorrating": 8,
        "cluster": 2
}, {
    "name": "the ghoul",
        "physicalstrength": 7,
        "fearfactor": 12,
        "killingpower": 2,
        "horrorrating": 7,
        "cluster": 1
}, {
    "name": "the freak",
        "physicalstrength": 9,
        "fearfactor": 2,
        "killingpower": 1,
        "horrorrating": 17,
        "cluster": 1
}, {
    "name": "dracula",
        "physicalstrength": 4,
        "fearfactor": 17,
        "killingpower": 5,
        "horrorrating": 16,
        "cluster": 2
}, {
    "name": "the hangman",
        "physicalstrength": 2,
        "fearfactor": 4,
        "killingpower": 13,
        "horrorrating": 8,
        "cluster": 2
}, {
    "name": "incredible melting man",
        "physicalstrength": 3,
        "fearfactor": 8,
        "killingpower": 15,
        "horrorrating": 6,
        "cluster": 1
}, {
    "name": "the thing",
        "physicalstrength": 19,
        "fearfactor": 4,
        "killingpower": 11,
        "horrorrating": 7,
        "cluster": 1
}];

We need to adjust the d3 code to read from this data set now rather than the circles data. This is just a few tweaks.

svg = d3.select("svg");
diag_circles = svg.selectAll("circle");
diag_circles.data(horror)
.enter()
.append("circle")
.attr("cx", function(d){return d.physicalstrength*10})
.attr("cy", function(d){return d.fearfactor*10})
.attr("r", 10);

Note we are now hard coding the circle radius and we are mapping our x and y axes onto physicalstrength and fearfactor respectively (multiplied by 10 so that we can see them properly)

 

Step 7: Adding in a tool tip

It would be nice if some of the other dimension were represented so first let’s colour the circles by cluster.

diag_circles.data(horror)
.enter()
.append("circle")
.attr("cx", function(d){return d.physicalstrength*10})
.attr("cy", function(d){return d.fearfactor*10})
.attr("r", 10)
.style("stroke", function(d){if (d.cluster === 1) {return "#1abc9c";} else {return "gray";}}) ;

Notice the if then else syntax for javascript

Scatter point labels are messy so let’s add the names in using a tool tip. To do this we going to borrow some code written by labratsrevenge.

  1. Go to External Resources and add the following url http://labratrevenge.com/d3-tip/javascripts/d3.tip.v0.6.3.js. This will allow your page to read their javascript code.
  2. Modify your javascript code so that it now includes the set up of the tooltip and mouseover properties for the circles
         //Select the svg
         svg = d3.select("svg");
         //Define and set up the tooltip
         var tip = d3.tip()
             .attr('class', 'd3-tip')
             .offset([-10, 0])
             .html(function (d) {
             return d.name;
         });
         svg.call(tip);
         //Set up the circles
         diag_circles = svg.selectAll("circle");
         diag_circles.data(horror)
         .enter()
         .append("circle")
         .attr("cx", function(d){return d.physicalstrength*10})
         .attr("cy", function(d){return d.fearfactor*10})
         .attr("r", 10)
         .style("stroke", function(d){if (d.cluster === 1) {return "#1abc9c";} else {return "gray";}})
         .on('mouseover', tip.show)
         .on('mouseout', tip.hide);
    
  3. Finally style your tooltip in the css section
     .d3-tip {
         font-family:  'Raleway';
         line-height: 1;
         padding: 1px;
         background: transparent;
         color: #000;
         border-radius: 2px;
     }
    

Your code should look like this

 

Step 8: Some intelligent axes

There’s still a lot to do. Our plot has no axes and the origin is still in the top left corner. Using d3 we can set up some dynamic axes that take their properties from the data.

In d3 an axis has

  • a domain: the values it is capable of representing (we want this to be a bit wider than the max and min of the values in the data)
  • a range: beginning and end points of the axis line as it is drawn within the svg.

So for example the origin 0 might in fact lie at the x point 100 in the svg. This is worked out in the following code

//Set constants
var w = 420,
    h = 400,
    padding = 30;
//Create adaptable scales
var xScale = d3.scale.linear()
.domain([0, d3.max(horror, function(d) {return d.physicalstrength;})])
    .range([padding, w - padding]);
var yScale = d3.scale.linear()
    .domain([0, d3.max(horror, function(d) {return d.fearfactor;})])
    .range([h - padding, padding]);

See if you can work out what is happening

To put our circles in the right place we need to apply these scales to the data when we set the circles co-ords. So substitute in these lines

.attr("cx", function(d){return xScale(d.physicalstrength)})
.attr("cy", function(d){return yScale(d.fearfactor)})

Next draw the axes by adding this code at the bottom of your javascript

//Set up the axes
var xAxis = d3.svg.axis()
    .scale(xScale)
    .orient("bottom");
var yAxis = d3.svg.axis()
    .scale(yScale)
    .orient("left");
svg.append("g")
    .attr("class", "axis")
    .attr("transform", "translate(0," + (h - padding) + ")")
    .call(xAxis);
svg.append("g")
    .attr("class", "axis")
    .attr("transform", "translate(" + padding + ",0)")
    .call(yAxis);

Finally you need to style them. Add this to the css:

.axis path, .axis line {
    fill: none;
    stroke: grey;
    shape-rendering: crispEdges;
}
.axis text {
    font-family: 'Raleway';
    font-size: 11px;

Everything should now look like this with the axes cleverly adjusted to the data. To check that they are auto scaling trying changing one of the fear factor values

 

Step 9: Adding drop downs

We want to be able to select which variable is being represented on each axis. Our drop downs for this can be created in html. Add the following inside your side bar div:

       X-axis:
<select id="xAxisVar" onchange="reDraw()" >
     <option></option>
     <option value="physicalstrength" selected="selected">physical strength</option>
     <option value="fearfactor" >fear factor</option>
     <option value="killingpower">killing power</option>
     <option value="horrorrating">horror rating</option>
</select>
<br>
Y-axis:
<select id="yAxisVar" onchange="reDraw()">
     <option></option>
     <option value="physicalstrength">physical strength</option>
     <option value="fearfactor" selected="selected">fear factor</option>
     <option value="killingpower">killing power</option>
     <option value="horrorrating">horror rating</option>
</select>

Now your sidebar will need a restyle and your drop downs also

#sidebar {
    height:400px;
    width:140px;
    background: #1abc9c;
    float:left;
    font-family:  'Raleway';
    font-size: 11px;
    padding: 5px;
}
select, button {
    margin: 5px;
    font-family:  'Raleway'
    font-size: 20px;
}

The result should look like this Don’t worry that they are not doing anything yet. They launch a function that we have still to define.

 

Step 10: Flipping the axes

To complete the program we add a function that grabs the values from the drop downs and then updates the x and y values of the circles. This is what the code below does. Notice the transition function. This creates a smooth animated transition of the circles from one location to another. The duration of this transition is set at 1000 milliseconds.

function reDraw() {
    xVar = document.getElementById('xAxisVar').value;
    yVar = document.getElementById('yAxisVar').value;
    d3.selectAll("circle")
        .transition()
        .duration(1000)
        .attr("cx", function (d) {
        return eval("xScale(d." + xVar + ");");
    })
        .attr("cy", function (d) {
        return eval("yScale(d." + yVar + ");");
    });
}

For this to work we need to change some of the jsfiddle settings. Change the second drop down on the left hand side to No wrap - in <body>

Finally in you code change the domain of the axes to

.domain([0, d3.max(horror, function (d) {
    return d3.max([d.physicalstrength, d.fearfactor, d.killingpower, d.horrorrating]);
})])

(This ensures that the axes are always big enough no matter what the variable is selected)

The final version looks like this

 

Step 11: How would you implement this?

Having developed your interactive chart in jsfiddle it would be nice to be able to run it as a stand alone web page. This is nice and easy since it is a matter of pasting your code into this template and opening the file with a browser. Note how the libraries for d3 and the tooltip are included.

<html>
    <head>
        <script src="http://d3js.org/d3.v3.min.js"></script>
        <script src="http://labratrevenge.com/d3-tip/javascripts/d3.tip.v0.6.3.js"></script>
        <style>
        <!--Paste your css code here-->
        </style
    </head>
    <body>
    <!--Paste your html code here-->
        <script>
        <!--Paste your javascript code here-->
        </script>
    </body>
</html>

About the Author

Simon Raper I am an RSS accredited statistician with over 15 years’ experience working in data mining and analytics and many more in coding and software development. My specialities include machine learning, time series forecasting, Bayesian modelling, market simulation and data visualisation. I am the founder of Coppelia an analytics startup that uses agile methods to bring machine learning and other cutting edge statistical techniques to businesses that are looking to extract value from their data. My current interests are in scalable machine learning (Mahout, spark, Hadoop), interactive visualisatons (D3 and similar) and applying the methods of agile software development to analytics. I have worked for Channel 4, Mindshare, News International, Credit Suisse and AOL. I am co-author with Mark Bulling of Drunks and Lampposts - a blog on computational statistics, machine learning, data visualisation, R, python and cloud computing. It has had over 310 K visits and appeared in the online editions of The New York Times and The New Yorker. I am a regular speaker at conferences and events.

Machine Learning and Analytics based in London, UK