Converting a dendrogram into a graph for a D3 force directed layout

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

I wrote this code for a project that didn’t work out but I thought I’d share. It takes the dendrogram produced by hclust in R and converts it into json to be used in a D3 force directed graph (slicing the dendrogram near the top to create a couple of clusters). The dendrogram in R looks like this

Dendrogram of clustering on US arrests

Dendrogram of clustering on US arrests

And the end result in D3 is this Read more

Thorstein Veblen and Hard Coding

Tweet about this on TwitterShare on LinkedInShare on FacebookGoogle+Share on StumbleUponEmail to someone

It is still quite common to hear the career progression of an analyst described as one upwards from the hard graft of coding and “getting your hands dirty” towards the enviable heights of people management and strategic thinking. Whenever I hear this it reminds me of the book Conspicuous Consumption by the American economist Thorstein Veblen. It examines, in a very hypothetical way, the roots of economic behaviour in some of our basic social needs: to impress others, to dominate and to demonstrate status. It’s not a happy book.

His principal concept is the distinction between exploit and drudgery.


The institution of a leisure class is the outgrowth of an early discrimination between employments, according to which some employments are worthy and others unworthy. Under this ancient distinction the worthy employments are those which may be classed as exploit; unworthy are those necessary employments into which no appreciable element of exploit enters.

He sees this division arising early in history as honour and status are assigned to those are successful in making others do what they want and “at the same time, employment in industry becomes correspondingly odious, and, in the common-sense apprehension, the handling of the tools and implements of industry falls beneath the dignity of able-bodied men. Labour becomes irksome.”

We’re no longer hairy barbarians using the vanquished for foot-stools but, as Veblen points out, the distinction persists no matter how illogical and unprofitable. Coding is still somehow viewed as necessarily inferior to the boardroom meeting even if it is the genius piece of code that makes or breaks a business.

Attitudes are changing, but still so slowly that there is a desperate shortage of people with both the experience and the hands-on skills to get things done.

So if you are a good analyst, and you love what you do, then this is my advice to you: when they come to lure you from your lovely pristine scripts, resist. Stay where you are. The 21st century is going to need you!

Machine Learning and Analytics based in London, UK