Introducing file-manifest

This is not really an introduction, as file-manifest has existed for almost two years. File-manifest is a node module I wrote originally with Justin Searls for the Manta platform when we were in the process of migrating it from a monolithic Perl application to a Node.js and Java service-based infrastructure, and then later made enhancements to and published on npm. It is similar to require-dir but does a lot more (such as pattern matching, automatic recursion, and custom map-reducing and file-loading). The goal here is to make you say, "Huh, that sounds interesting. I'll give it a try," but even if you stop at "that sounds interesting", I'll still be satisfied.

How it came to be

Manta uses hogan.js for rendering server-side templates, along with hogan-express to make it (more easily) compatible with express. Hogan-express requires you to register an object of partials in your app settings, and at first, we did this manually, like this:

app.set('partials', {  
  foo: 'partials/foo',
  bar: 'partials/bar'
  /* etc. */
});

As you can imagine, this can get out of hand quickly. We now (two years later) have 124 separate partials, and we probably should have more as we have not been extremely diligent in refactoring common logic. How would you like to have something like that cluttering up your express app? And of course, if you do this manually, every time you add a partial you have to remember to add it to this object, or it won't be available to hogan. How many times do you think the average developer would forget to do this and then wonder why their page wasn't working? So we needed a way to automatically build this object on app start up using the files in the views/partials directory. File-manifest was created to solve this problem.

What it does

File-manifest is actually three modules that depend on each other downward. The first thing that was obvious when we started working on this module was that it needed to be able to easily get just the files from a directory, as well as know which "files" were actually directories, in order to recurse into them. Node doesn't have anything like this built in. The best you can do is fs.readdir combined with fs.stat. So we started by writing kindly (although, it was not called that or published at this point), which abstracts the process of reading files from a directory and grouping them by "kind" (hence the name). So if you say kindly.get('views/partials'), you get an object back with a files property and a directories property, which are just arrays of the files and directories in "views/partials." Second, we needed something to call kindly recursively, based on the directories returned. There are a number of file walkers out there for node.js, but none of them really appealed to us for one reason or another, so we wrote our own, called pedestrian (get it? . . . it walks). And finally, we needed the consumer of this file walker that would know what to do with all the files it found (and incidentally "requiring them" is only one of the things you can do). This functionality is now published as file-manifest. Of course, we could've written all this into a single module, but in general, I subscribe to the small, atomic module theory. There's nothing wrong with a small module that does one thing and does it well. It creates small building blocks that can then be more easily combined in other ways (even ways you didn't anticipate). I've personally used both kindly and pedestrian in other projects, apart from their "parent" file-manifest.

How It Works

File-manifest can operate both synchronously and asynchronously. If you pass a callback function as your final argument (with the signature function(err, manifest), it will be asynchronous. The following examples, however, will all be synchronous, as they are a bit clearer and more concise that way. The simplest usage of file-manifest is to recursively require a directory. Just pass that directory to the generate function like this:

var fm = require('file-manifest');  
var manifest = fm.generate('dir');  

But you can also change the way the files are read (the default is require, but fs.readFile support is also built in):

var manifest = fm.generate('dir', { require: 'readFile' });  

But you can also pass any function. For instance, you could load yaml files as objects:

var yaml = require('yamljs');  
var manifest = fm.generate('config', { require: function(options, file) {  
  return yaml.loadFile(file.fullPath);
}});

You can also name the resulting keys differently. The default is camel-cased (with the full subdirectory path), but you can use dashes too:

var manifest = fm.generate('dir', { namer: 'dash' });  

You can get only files matching a pattern:

var manifest = fm.generate('dir', { patterns: '**/*.json' });  

Or exclude some files:

var manifest = fm.generate('dir', { patterns: ['**/*.json', '!config*'] });  

Or don't recurse:

var manifest = fm.generate('dir', { patterns: '*.json' });  

Or provide a completely custom implementation:

var manifest = fm.generate('dir', { memo: [] }, function(options, manifest, fileObj) {  
  return memo.concat(fileObj.relativeName);
});

Use Cases

In short, you can use file-manifest pretty much anytime you need to do something with all the files in a directory, regardless of what you want to do or what types of files they are. File-manifest is flexible. It won't complain. Here are some more specific things you can do (and I have done) with file-manifest.

1) Modularize your express application. Create a routes directory, and export an instance of the express router, like this:

var router = module.exports = express.Router();  
router.get('/bar');  

Then create a route manifest in your main express app.

var fm = require('file-manifest');  
var routes = fm.generate('routes');  
var express = require('express');  
var app = express();

app.use('/foo', routes.bar);  

2) Similarly, create a directory of express middleware functions:

module.exports = function(req, res, next) {  
  res.header('Access-Control-Allow-Origin', '*');
  next();
};

And generate a middleware manifest.

var middleware = fm.generate('middleware');

app.use(middleware.crossDomain);  

3) Dynamically generate a hogan partials object (yeah, that thing that started all of this in the first place).

var partials = fm.generate('views/partials', { namer: 'dash', require: function(options, file) { return 'partials/' + file.relativePath; } });  

Notice how we use the require option to not actually even require the files, but just to generate a location that the partial points to.

4) Collect all the JSON files for a fake server of a particular type. For instance, we collect all the member files in our fake api in order to create a quick dev login.

var members = fm.generate('data/member', '**/*.json');  

5) Generate a recursive data manipulator for putting template data into a form usable by an opinionated templating engine (such as hogan). This is a somewhat complicated and domain-specific example (which I won't try to explain in depth), but the point is to illustrate how you can do very custom things.

var dataBuilders = fm.generate('../data-builders', '**/*.js', function(options, manifest, file) {  
  var dirs = file.relativeName.split('/').slice(0, -1);
  var objPath = dirs.length ? dirs.join('.') + '.builders' : 'builders';
  _.ensure(manifest, objPath, []);
  var builder = require(file.fullPath);
  var arrayMethod = file.name === 'index' ? 'unshift' : 'push';
  _.safe(manifest, objPath)[arrayMethod](builder);
  return manifest;
});

This results in an object that looks something like this (where the functions are the exports of the files in data-builders):

{
  builders: [
    function(){/*...*/},
    function(){/*...*/},
  ],
  foo: {
    builders: [
      function(){/*...*/},
      function(){/*...*/},
    ],
    bar: {
      builders: [
        function(){/*...*/},
        function(){/*...*/},
      ]
    }
  }
}

6) Require all your code prior to running tests in order to generate accurate coverage reports.

  fm.generate('lib', ['**/*.js', '!exclusions.js']);

7) Generate a list of specs to run under minijasminenode, based on a command line flag.

  var files = fm.generate('spec', { memo: [], patterns: grunt.option('match'), reducer: function(options, manifest, file) {
    manifest.push(path.relative(process.cwd(), file.fullPath));
    return manifest;
  }});

8) Modularize your gulpfile by putting each task in a separate module in the gulp directory:

require('file-manifest').generate('./gulp', ['*.js', '!config.js']);  

Note that, in this case, I don't need to do anything with the object that comes back. I mostly want to require the files inside gulp for the side effects of registering tasks.

9) Create a manifest of mongoose models and then throw them on req so you can access them in your routes and middleware.

var models = fm.generate('models');  
app.use(function(req, res, next) {  
  req.models = models;
  next();
});

10) Collect all the markdown posts for a blog in a particular directory:

var marked = require('marked');  
var fs = require('fs');  
var rawPosts = fm.generate('posts', { patterns: '**/*.md', memo: [], reducer: function(options, manifest, file) {  
  var contents = fs.readFileSync(file.fullPath);
  var markdown = marked(content);
  manifest.push(markdown);
  return manifest;
}});

Conclusion

That's really just a sample of what file-manifest can do, but you get the idea: collect all the files in a directory and process them in some way. After writing it, I wasn't sure how often we'd use it (other than to solve our hogan problem), but it turns out, it's fairly common to need to do something with all the files in a directory. Feel free to share your own use cases in the comments below!

comments powered by Disqus