Interactive Mapping

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

Animated World Map

In the past I have used expensive GIS software, or ggmap in R, to produce maps and visualise geographic data. I wanted to see what javascript offered to make more interactive maps. After a little research I found a couple of options Leaflet and good old D3.js. I challenged myself to make an animated map that plots all the places I’ve been to over time.

Read more

Dendrograms in R2D3

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

Hi, I’m Andrew and this is my first post for Coppelia! If you like the look of this feel free to visit my blog dinner with data (and see what happens when a data scientist hits the kitchen!)

I was excited by James’s last post on the new package R2D3, and I thought I would try to help further develop the package. This is a great new package, built by James Thomson (and in collaboration with myself and Simon Raper at Coppelia) that utilises D3 visualisations inside R. You can quickly create very striking visualisations with a just a few lines of code. This has recently been shared with a recent post, but since then a couple of updates have been made to increase the functionality.

In particular to the function D3Dendro, which creates dendrograms based on a hclust object in R. I had been working on a number of alternatives to the usual static dendrogram found in the package so far, so I thought I would add these in and describe them below.

I have created two new distinct functionalities:

  • Collapsible nodes
  • Radial output (rather than the more traditional ‘linear’ dendrogram)

You can clone the package from James’s github repository or run the following in R:


I will include the example in the original post, so you can easily compare the differences.

Original dendrogram:

hc < - hclust(dist(USArrests), "ave") JSON<-jsonHC(hc) D3Dendro(JSON, file_out="USArrests_Dendo.html")

Read more

Introducing R2D3

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

R2D3 is a new package for R I’ve been working on. As the name suggests this package uses R to produce D3 visualisations. It builds on some work I previously blogged about here.

There are some similar packages out there on CRAN already. Notably rjson and d3Network. However I found with these packages that they covered parts of the process (creating a json or creating a D3) but not the whole process and not ensuring the json was in the right format for the D3. So that was the thinking with this package. I was the aiming to create an end to end process for converting R objects into D3 visualisations. When i mentioned it to [email protected] he was keen to contribute. So we’ve been collaborating on it over the last few weeks. Its by no means finished, but I think it contains enough that its worth sharing.

Read more

An A to Z of extra features for the D3 force layout

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

Since d3 can be a little inaccessible at times I thought I’d make things easier by starting with a basic skeleton force directed layout (Mike Bostock’s original example) and then giving you some blocks of code that can be plugged in to add various features that I have found useful.

The idea is that you can pick the features you want and slot in the code. In other words I’ve tried to make things sort of modular. The code I’ve taken from various places and adapted so thank you to everyone who has shared. I will try to provide the credits as far as I remember them!

Basic Skeleton

Here’s the basic skeleton based on Mike Bostock’s original example but with a bit of commentary to remind me exactly what is going on

A is for arrows

If you are dealing with a directed graph and you wish to use arrows to indicate direction your can just append this bit of code Read more

Converting a dendrogram into a graph for a D3 force directed layout

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

I wrote this code for a project that didn’t work out but I thought I’d share. It takes the dendrogram produced by hclust in R and converts it into json to be used in a D3 force directed graph (slicing the dendrogram near the top to create a couple of clusters). The dendrogram in R looks like this

Dendrogram of clustering on US arrests

Dendrogram of clustering on US arrests

And the end result in D3 is this Read more

Finding neighbours in a D3 force directed layout

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

I’ve adapted the code from this stackoverflow answer (from D3’s creator Mike Bostock) so that you can highlight neighbours in a force directed layout. Just doubleclick on a node to fade out any non-immediate neighbours and double click again to bring them back.

Lazy D3 on some astronomical data

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

I can’t claim to be anything near an expert on D3 (a JavaScript library for data visualisation) but being both greedy and lazy I wondered if I could get some nice results with minimum effort. In any case the hardest thing about D3 for a novice to the world of web design seems to be getting started at all so perhaps this post will be useful for getting people up and running.

A Reingold–Tilford Tree of the IVOA Astronomical Object Ontology using D3

A Reingold–Tilford Tree of the IVOA Astronomical Object Ontology using D3

The images above and below are visualisations using D3 of a classification hierarchy for astronomical objects provided by the IVOA (International Virtual Observatory Alliance). I take no credit for the layout. The designs are taken straight from the D3 examples gallery but I will show you how I got the environment set up and my data into the graphs. The process should be replicable for any hierarchical dataset stored in a similar fashion.

A dendrogram of the same data

A dendrogram of the same data

Even better than the static images are various interactive versions such as the rotating Reingold–Tilford Tree, the collapsible dendrogram and collapsible indented tree . These were all created fairly easily by substituting the astronomical object data for the data in the original examples. (I say fairly easily as you need to get the hierarchy into the right format but more on that later.)

First the environment.

If you are an R user with little experience of building web pages then you’ll probably find yourself squinting at the D3 documentation wondering how you get from the code to the output. With just a browser to read the JavaScript and an editor (notepad if you like but preferably a specialist HTML editor) you can get through the most of the tutorials referred to in Mark’s previous post . Doing this locally on your own pc works ok because the data visualised in the tutorials is mostly hard coded into the JavaScript. However once you want to start referring to data contained in external files some sensible security restrictions on what files your browser can access will block your attempts. The options are to turn these off (not advised) or switch to working on a webserver.

You can either opt for one of the many free hosting services or if you are feeling more adventurous you can follow Mark’s posts on setting up a Linux instance on Amazon Web Services and then follow Libby Hemphill’s instructions for starting up Apache and opening the port. Finally, going with the latter, I use FileZilla to transfer over any data I need to my Linux instance. See this post for getting it to work with your authentication key.

This should leave you in the following situation:

  • You have a working web server and an IP address that will take you to an HTML index page from which you can provide links to your D3 documents.
  • You have a way of transferring work that you do locally over to your web server.
  • You have a whole gallery of examples to play around with

Rather than creating my own D3 scripts I’m just going too substitute my own data into the examples. The issue here is that the hierarchical examples take as an input a nested JSON object. This means something like:

 &quot;name&quot;: &quot;Astronomical Object&quot;,
 &quot;children&quot;: [
   &quot;name&quot;: &quot;Planet&quot;,
   &quot;children&quot;: [
     &quot;name&quot;: &quot;Type&quot;,
     &quot;children&quot;: [
      {&quot;name&quot;: &quot;Terrestrial&quot;},
      {&quot;name&quot;: &quot;Gas Giant&quot;}

The issue is that our data looks like this:

1. Planet
1.1. [Type]
1.1.1. Terrestrial
1.1.2. Gas Giant
1.2.  [Feature]
1.2.1. Surface Mountain Canyon Volcanic Impact Erosion Liquid Ice
1.2.2. Atmosphere Cloud Storm Belt Aurora

To put this into the right format I’ve used Python to read in the file as csv (with dot as a delimiter) and construct the nested JSON object. Here’s the full python code:

import csv, re
#Open the file and read as dot delimited
def readLev(row):
    pattern = re.compile(&quot;[a-zA-Z]&quot;)
    for col in row:
        if !=None:
    return level
with open('B:PythonFilesPythonInOutAstroObjectAstro.csv', 'rb') as csvfile:
    astroReader = csv.reader(csvfile, delimiter='.')
    astroJson='{&quot;name&quot;: &quot;Astronomical Object&quot;'
    for row in astroReader:
        #Identify the depth
        if currentLev&gt;prevLev:
            astroJson=astroJson+', &quot;children&quot;: [{&quot;name&quot;: &quot;' + row[currentLev]+'&quot;'
        elif currentLev&lt;prevLev:
            astroJson=astroJson+', &quot;size&quot;: 1}'+']}'*jump+', {&quot;name&quot;: &quot;' + row[currentLev]+'&quot;'
        elif currentLev==prevLev:
            astroJson=astroJson+', &quot;size&quot;: 1},{&quot;name&quot;: &quot;' + row[currentLev]+'&quot;'
    astroJson=astroJson+ '}]}]}'
    print astroJson

Since only the level of indentation matters to this process it could be repeated with on any data that has the form

* Planet
** [Type]
*** Terrestrial
*** Gas Giant

So that’s it. You’ll see that it in the source code for the hierarchical examples there is a reference to flare.json. Substitute in a reference to your own file containing the outputted JSON object and be sure to include that file in the directory on your web server.

Of course it’s a poor substitute for learning the language itself as that enables you to construct your own innovative visualisations but it gets you started.

D3 – another acronym to learn

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

D3 (which stands for data driven documents ) has been getting a lot of traction over the last few months, with more and more interactive and animated visualisations using this JavaScript library. The author of the library, Mike Bostock, is very active in both developing the library and also in providing a constant stream of examples using the language.

For anyone coming from a ggplot2 background, there are a lot of similarities with certain parts of D3 and the attributes that can be given to objects. There is a fair amount more code to write, but there are a lot of great tutorials out there which give very good introductions to creating good looking visualisations. There’s a bit of a learning curve to using it, but there are lots of tutorials out there which make it easy to get started, for example, here, here and here.

The general idea behind the language is to encode objects, for example, a circle, with various attributes like colour, size and  location. These could either be hard coded, based on a function (e.g. the sine function) or based on a dataset. The additional nice feature of the software is the built in ability to transition between states (e.g. moving a circle from location A to location B) which would normally be difficult to do, and probably appear quite jerky.

For anyone coming from a Tableau background, D3 provides a lot more flexibility in what can be achieved, but is a very code based solution and not a BI solution.

From my experience so far, a specialist HTML editor like Astana Studio is a good place to start for a few reasons, including colour coding for the HTML and also having a built in local web server so that you can quickly preview the results of your code writing.

As an aside, you’ll need a web server of some sort for any D3 code to work (It took me a frustrating hour to figure this out). The alternative to using the built in Astana server is something like using the python simple http server, but I find this a lot more cumbersome.

Oh, and sorry for not having any examples built into the post – doesn’t like having custom JavaScript in posts.

Machine Learning and Analytics based in London, UK