Once Manta decided to stop using lineman in favor of grunt, we had to figure how the heck to make the switch and ensure that our build process was still the same.

Steal the underpants

I think there was skepticism on the part of some that it would even be possible, but I was optimistic because I had been thinking about it and had sort of planned out the steps in my mind. There were three main elements that would need addressed in order to migrate to grunt: the build process (see below), the fake api (coming soon), and the tests (coming soon).

Migrating the build process

This was actually the trickiest part of the migration up front. I had a fairly solid idea of how to migrate the other items (though there were more challenges after the fact with those parts), but our build process was still a bit of a mystery. As I mentioned in the previous post, the nice thing about lineman is that it wraps up a (typically) messy process and abstracts it away so you don't have to worry about it. But that also means it's difficult to get insight into the process to understand it. Combine this with the fact that I was still pretty green with grunt, and you can understand why this was a dubious undertaking. So really, the first step was trying to understand everything that our build process was currently doing. This involved a lot of reading our own configuration setup, diving into the lineman source, perusing the output of lineman config (which dumps the raw underlying grunt config) and lineman config --process (which dumps the interpolated config), and just observing the logs generated by lineman run and lineman build.

The other tricky thing was that I needed to keep the changelog from becoming unreviewable for the other developers (I don't know that I succeeded in this - I am known for ridiculously long pull requests). I knew, for example, that I would be migrating our node_modules directory up a level, and that I would be blowing away a number of lineman plugins and adding quite a few grunt plugins. That alone would be enough to make github choke on the PR file diff (we check our dependencies in, which, for a private web app, I absolutely recommend). So until the pull request was ready to merge, I did not commit any node_module changes.

At this point, it's probably briefly worth mentioning how task-master works. Task-master is a tool that wraps, not really so much grunt itself as the grunt configuration specific to a project. Instead of saying:

module.exports = function(grunt) {  
  // Call grunt.loadNpmTasks two dozen times
    // excessive configuration

    // . . . time passes

    // yes, still configuring


  // Setup 35+ aliases

  // Do your taxes

  // etc.?

You can just say:

module.exports = require('task-master');  

Since task-master is a function that takes grunt as a parameter, that's literally all you need. I won't hash out the whole of the readme here, but the tl;dr version is that each task's configuration is a separate file inside the tasks directory. So the configuration for grunt-contrib-watch would be held in tasks/watch.js (or .json or .coffee or .yml) and would look something like this:

module.exports = {  
  // Target
  js: {
    files: ['<%= files.js %>'],
    tasks: ['jshint', 'concat_sourcemap:js'],
    options: {
      spawn: false
  // ... other targets

Task-master has a few other bells and whistles as well, but the ability to modularize one's build process is the entire reason I wrote it in the first place.

So I started working through the text output of the command lineman run, mapping the same tasks to an alias in grunt called run (I wanted the change to be as minimal as possible from a user standpoint, so I kept many of the task names the same). I won't walk through every single conversion, as that would be tedious and make this post extremely long, but the basic pattern was this:

1. Figure out what task ran next

This was actually the hardest part because, again, lineman abstracted those details away, but also because some parts of the build were dependent on others, at least in lineman, but I had a feeling they didn't need to be. One goal I knew I had in converting everything was a faster build time because our build process was slow. So I was planning to use grunt-concurrent, which meant that I needed to keep things as decoupled as possible. But obviously, we needed to avoid race conditions and some things really did depend on other things. For instance, we generate a template cache for angular using grunt-angular-templates and concatenate that to our main javascript app. And we obviously couldn't concatenate it before it existed.

2. Decide on a matching grunt plugin and install it.

This was typically as easy as seeing what plugin lineman was using under the hood. I did occasionally look to see what other plugins were available. For instance, lineman uses grunt-concat-sourcemap, but I had used grunt-contrib-concat in other projects and felt like it would be worthwhile to try the plugin supported by the core grunt team. Plus, I strongly and inexplicably dislike snake case in javascript - e.g. "concat_sourcemap". It turns out, however, that grunt-concat-sourcemap is much, much faster than grunt-contrib-concat, so we stuck with that one in the end.

This step was partially complicated by the fact that several core lineman tasks are custom tasks. However, they are pretty much all "copy x to y" type tasks, like images:dev, webfonts:dev, and pages:dev. They have different names, but they're really just copying files around. So in those cases, I had to figure out what files were copied and add them to a single copy:dev task that used grunt-contrib-copy. And since they were all in different directories and needed separate destinations, I had to use the files array format, which I dislike severly. Why can't grunt just figure out what I mean? Surely a single boolean option specifying one-to-one file copies is plausible.

3. Figure out what configuration it ran with

This probably sounds easy (just use the same config as lineman), and it wasn't terrible in most cases, but there were a few instances where the sheer number of options or the complexity of those options or even which version of a plugin the options belonged to made it more challenging. The ones that come to mind are grunt-contrib-uglify because I wanted to keep our javascript package as small as possible, but I didn't know enough about how minification worked, and there were many options; grunt-browserify because some of the options we were using had been deprecated; and anything that involved generating sourcemaps (grunt-contrib-less and grunt-concat-sourcemap for example) because, again, I just didn't know enough.

This step was also complicated by needing to figure out which files each task operated on and where it should put them when it was done. Lineman has a "files" configuration file that let's you group files together and then reference them via interpolation, like files: ['<%= files.js %>']. This is actually just the normal grunt functionality, but lineman abstracts this into a separate file. Task-master does this through a context file (where you can add any generic configuration that multiple grunt tasks use), so I added a _taskmaster.context.js that specified a paths object and a files object. Basically, paths was a collection of common locations in the repository (like node_modules, bower_components, and reports/coverage) and files was a collection of globstar patterns matching files for various tasks (e.g. all the frontend javascript files to be concatenated and minified). Here's a sample from that file:

var path = require('path');  
var root = path.resolve(__dirname, '..');

module.exports = {  
  paths: {
    root: root,
    client: root + '/client',
    server: root + '/server',
    generated: root + '/client/generated',
    app: root + '/client/app',
    // . . .
  files: {
    js: {
      vendor: [
        "<%= paths.vendor %>/js/jquery.js",
        "<%= paths.vendor %>/js/jquery.*.js",
        "<%= paths.vendor %>/js/modernizr.js",
        "<%= paths.vendor %>/js/lodash.js",
        "<%= paths.vendor %>/js/bootstrap.js",
        "<%= paths.vendor %>/js/angular-file-upload-shim.js",
        "<%= paths.vendor %>/js/angular.js",
        "<%= paths.vendor %>/js/**/*.js"
    // other sets of files

4. Optimize.

This involved a lot trial and error, and even now, almost six months after our migration, I still believe there may be one lingering race condition related to browserify because we occasionally see inexplicable reference errors on travis. The nice thing about optimizing a build process is that it doesn't have to happen all at once. I'm still optimizing some parts. I strongly recommend grunt-concurrent, which can really help speed things up by running multiple tasks in parallel. It doesn't have any mechanism for running sets of tasks in parallel, but with aliases, you can accomplish the same thing. Here's our concurrent:dev task:

  dev: ['jsdev', 'instrument:client', 'less:dev', 'brew', 'copy:dev'],

All these tasks will run in parallel (if you run them with a processor with enough cores to do so), but jsdev is actually an alias to:

['ngtemplates:dev', 'browserify:dev', 'concat_sourcemap:dev']

So essentially, those tasks will run in series (because they depend on one another) as part of the larger concurrent process.


A special side note that led to some complications . . . figuring out what files to watch and what tasks to run when those files changed. In our lineman configuration, we were watching everything it seemed (needing to run ulimit -n 1024 in .bashrc was a common problem). And we were restarting our server (and basically rerunning the entire lineman run process) whenever any of the files changed. But we really didn't need the server to restart when only the client-side assets changed or when the tests changed (etc). I think I was able to boil it down to only the critical tasks in the end (again using a combination of grunt-concurrent and aliases), but there were some bugs, specifically around race conditions. One change I made when moving away from lineman was to get rid of the dist directory. Lineman basically did some tasks and put the results in client/generated, then did some things to those files and put the results in client/dist, and then did a few more things and put them in server/public. This felt unnecessary to me, so I cut out the middle directory and just used client/generated for dev builds and server/public for production ones. But this meant that a single file (e.g. our main app, app.js) might actually change several times during the build . . . and that is BAD for file watching. Test reruns would trigger part-way through builds and then tests would fail with Reference errors. That's since been resolved by building intermediate files in the client/generated directory.


The tl;dr version here is that this kind of migration just requires patience, trial and error, and a certain amount of optimism. There's no particularly magical formula for figuring out how to de-lineman-ify your build (unless maybe you're using the absolute defaults). The good news is that doing this kind of migration gave me a much more thorough understanding of our build process (and grunt as a build tool in general). But even aside from this, the pain and frustration was worth it because in the end we were able to speed our build process up from 10 seconds to just over 4 (this was a combination of a. doing fewer things, b. doing them in parallel, and c. using jit-grunt - a default for task-master.), and we simplified our abstraction. In some ways, task-master is still an abstraction, but it doesn't introduce anything that's not part of grunt itself; it just modularizes it. So we gained speed, simplicity, and, probably most importantly, flexibility. We have been able to make many other build process optimizations and enhancements because we took the time to migrate.