Mark Hansen's Blog

I had a lot of fun making this visualization. d3.js + Backbone.js + CoffeeScript is an amazing combination.

It’s all totally standards-compliant HTML5. Boxes are drawn with inline SVG, and transitions are run in CSS for hardware acceleration (edit: they aren’t by default, you have to do a little extra work for that, see comments). Popovers are Twitter Bootstrap.

It’s crazy how much the number of stolen cars has picked up in the last few days approaching Christmas.

A Code Walkthrough

As an example of how to do these things with coffeescript, d3, and backbone, here’s the code used to generate the weekDays graph, showing the number of stolen vehicles per weekday.

The Backbone Bits

The whole widget is in one javascript class that takes care of itself - WeekGraph. It extends from Backbone.View, and its constructor hooks it up with an HTML element - @el, and a data model - @model. The view then listens for ‘change’ events coming off the model, and rerenders whenever the model changes. In this way, the view and model stay in sync.

WeekGraph = Backbone.View.extend
  height: 150
  width: 420
  initialize: ->
    @model.bind "change", => @render()

That was the wiring (decoupling and dependency injection if you’re into that).

The d3 Bits

The render() method is where the real magic happens, rendering that data model into the HTML element, so we can see it on screen.

First, separate the stolen vehicles data by weekday. This is accomplished with d3.nest().

  render: ->
    weekDays = d3.nest()
      .key((d) -> d.dayOfWeek)
      .entries(@model.get "selectedVehicles")

We need to separate the weekdays on the graph horizontally across the x axis.

x is a function that does this. You could just use a linear function, but d3 has scale function helpers to make changing the domain and range of these functions really easy, without having to manipulate equations when you want to change the width.

The d3 Scales

x = d3.scale.linear()
    .domain([0, 6]) # weekdays
    .range([0, 6 * (cellPadding + cellSize)])

Also, the bars need to have a height, decided by the number of vehicles stolen that day. y is a linear scale that turns vehicles stolen into a pixel height for the bar.

I’m also making the bar darker when there’s more vehicles stolen that day. d3 can also make scales that interpolate between colors. This seemed a bit magic to me, but really d3 just has a list of color strings, and turns them into RGB colors and interpolates them.

maxWeekday = _.max(weekDays, (d) -> d.values.length)
maxPerWeek = maxWeekday.values.length

height = d3.scale.linear()
    .domain([0, maxPerWeek])
    .range([0, @height])

color = d3.scale.linear()
    .domain([0, maxPerWeek])
    .range(["white", "darkblue"])

The d3 Data-DOM Join

Now let’s select the SVG element, set the width and height, and do a relational join between our weekDay data and the bar DOM elements. I’ll explain in a second.

bars = d3.select(@el)
    .attr("width", @width)
    .attr("height", @height)
    .selectAll("rect.weekDay")
    .data(weekDays)

This data() relational join operator gives us three selections:

The Updating Elements

1: update. data that already existed in the DOM, but might have changed.

So we update the DOM. Use a nice transition of 1000ms. We can pass functions that have (d, i) arguments. d is the data associated with that DOM element, i is the data index. Transitions are all interpolated nicely.

bars
    .transition()
    .duration(1000)
    .attr("x", (d, i) -> x(i))
    .attr("y", (d, i) => @height - height(d.values.length))
    .attr("height", (d, i) -> height(d.values.length))
    .attr("fill", (d) -> color(d.values.length))

The New Elements

2: enter(). Like an actor entering the stage. Data that doesn’t have a DOM element yet. So we append elements to the DOM for each new datum, and set the location, size, class, and fill according to the data values.

bars.enter()
    .append("svg:rect")
    .attr("x", (d, i) -> x(i))
    .attr("y", (d, i) => @height - height(d.values.length))
    .attr("height", (d, i) -> height(d.values.length))
    .attr("width", cellSize)
    .attr("class", "weekDay")
    .attr("fill", (d) -> color(d.values.length))

The Exiting Elements

3: exit(). Like an actor exiting the stage. Elements that don’t match up with any data we have. The elements used to match up with some data, but they don’t any more. So we remove the elements from the DOM.

bars.exit()
    .remove()

The Popovers

Finally, I hooked up a twipsy overlay on mouseover. This uses twitter bootstrap’s twipsy plugin. It demonstrates how each DOM element contains a property __data__ that contains the d3 data.

@$ is this.$, a jQuery selector scoped to the local backbone element. It’s handy to make sure one view doesn’t accidentally attach handlers to another view’s elements.

    @$("rect.weekDay").twipsy
      title: ->
        weekDayName(@__data__.values[0].date) + "<br>" +
        @__data__.values.length + " Reports"
      html: true
      placement: "above"

I hope this has given you a taste of how general and powerful d3 is. It’s a really enjoyable library to use, and it combines well with Backbone and CoffeeScript. Highly recommended.

visualizations

Mouseover the bars for a visualisation of the colors of the stolen cars.

The Database

A few days ago, the NZ Police launched a public database of stolen cars. It’s a snapshot taken from the Police Vehicle of Interest database, updated 3x daily.

"It will be a resource for security guards, insurance companies, moteliers, scrap metal dealers and community policing patrols."

...

"It also has obvious benefits for people buying second-hand vehicles, garages that service vehicles and service stations where petrol thefts can regularly be associated with stolen vehicles."

...

"Potentially, it gives police many more pairs of eyes out there. People can do their own checking and then report it to Police."

This sounds like a goldmine of interesting data. I grabbed the list of all the cars stolen in the last six months. It comes in a handy CSV file. For example, here’s the first four vehicles:

Plate,Color,Make,Model,Year,Type,Date,Region
007579,Blue,Yamaha,TTR230,2007,Trail Bike,2011-07-14,Bay of Plenty
0BOOST,Blue,Honda,ACCORD,2003,Saloon,2011-06-19,Counties/Manukau
1045Z,Orange,Trailer,HOMEMADE,1972,Trailer,2011-11-07,Eastern
1079Y,Silver,Trailer,ABEL K8SSA,2001,Trailer,2011-07-14,Waitemata
...

So I decided to have a crack at visualising the data. Above is the first interactive graphic I’ve published - the most popular makes of stolen vehicles.

Visualisation Tech

This visualisation was a great excuse to play around with some new tech. Rendering is done with SVG, controlled by the excellent d3 visualisation library. Popovers are care of Twitter Bootstrap, which rocks, although I did have to put in a pull request with fixing the popovers for SVG.

Sorry Android 2.3 users, your browser doesn’t do SVG, so you’ll need to use a desktop browser to see.

I was very impressed with d3’s flexibility and conciseness, and I’m looking forward to using it on more projects. I may blog about how awesome the d3 experience was later. If you’re into making visualisations, I heartily recommend checking d3 out.

visualizations

I don’t write a lot of straight HTML any more - I prefer the rapid prototyping you can do with the likes of Haml, without having to worry about always matching your start and end tags.

However, the main disadvantage of haml is that it needs to be compiled to HTML. This is an extra step in development that slows down iteration.

Previously I used a watchr script, spinning up a process that watches for new haml files and automatically compiles them. This isn’t as seamless as I’d like, because I have to remember to spin up this process every time I want to write some haml. Also, watchr won’t pick up new files without a restart.

But I don’t want to auto compile every haml file. My rails or jekyll templates are supposed to be substituted into templates, not rendered as-is. I want a way to opt in to auto-compilation, preferably only once per directory.

To opt-in, my script checks if a file named “.autohaml” exists in the current directory. Create a file named “.autohaml” to opt in.

I’ve added this autocompile code to my vimrc. I’ve tried to use the minimum of vimscript possible, doing most of the logic using python.

" Auto compile .haml files on save, but only
" if there's a .autocompilehaml file in the cwd.
" Depends on a `haml` executable. `sudo gem install haml`

au BufWritePost *.haml call HamlMake()

function! HamlMake()
    py << ENDOFPYTHON
import os
import vim

in_file = vim.current.buffer.name
dirname = os.path.dirname(in_file)
if os.path.exists(dirname + "/.autohaml"):
    out_file = in_file[0:-5] + ".html"
    os.system("haml %s > %s" % (in_file, out_file))

ENDOFPYTHON
endfunction

vim haml