Pre-compiling Node.js Modules

Automated Publishing Toolchains for Node.js Modules

I recently worked on a pretty cool feature for autocannon that makes it easier for users to download and use it. The feature is the pre-compilation of the native Node.js modules that autocannon depends on. This means that autocannon users don’t have to compile dependencies themselves. Therefore, they don’t need to have build tools installed, and this lowers the entry barrier for using Autocannon.

Autocannon itself only has a single native module that needs to be compiled, and that is native-hdr-histogram. To do this, I needed to switch out node-gyp for node-pre-gyp, and “publish” compiled versions of the Node.js modules on AWS S3.

Our goal was to give users running node 4, 5 & 6 the ability to download these native Node.js modules, without the need to compile them. To do this, we needed to compile the binaries for each Node.js version for Windows 64 and 32 bit, Linux 64 and 32 bit and OSx 64 bit only. That means on a build and publish, we needed to compile 15 binaries and publish them to some form of publicly available cloud storage.

CI automation to the rescue.

Prerequisites

To move forward with this work, you will need to create an Amazon S3 bucket, and have an IAM user policy with a user configured with the ability to publish to that bucket. You will also need this user’s access key and secret access key. Read the Readme of node-pre-gyp over here and figure it out. For this, we created a bucket called autocannon-assets, in eu-west-1. These minor details are important to note. We use the url for this bucket (https://autocannon-assets.s3-eu-west-1.amazonaws.com) to build sane urls to download the binaries.

The Setup

node-gyp is a build tool which is used extensively in Node.js projects. node-pre-gyp is a wrapper around this. It adds functionality to build and publish Node.js modules to AWS S3 buckets and download those published Node.js modules when running npm install. It can fall back to compiling them the way node-gyp does. However, this is only when there is no published version available on S3 which is compatible with the user’s installed node version. In this case, users will need compatible build tools to build it.

With native-hdr-histogram, we aimed to automate as much of the publishing and building process as possible. To do this, we made heavy usage of CI tooling. For Linux and OSX support, we used Travis, and to support Windows we needed to add Appveyor to the mix. To support these CI tools, we needed to have scripts created for the auto publishing of binaries. Figuring out all the bits and bobs that the scripts needed took a while, and I’m pretty sure I killed Travis’s osx build infrastructure while working on this!

To run CI tools with the ability to publish to S3 buckets, we needed to embed our keys within the repository. This is a pretty big no no if you don’t want people getting access to your infrastructure. To safely embed the keys in the corresponding .travis.yml and appveyor.yml, you need to first encrypt them. There is a different path for encrypting these keys within each platform.
To encrypt these keys on Travis you must use a Ruby gem. To install this gem, simply do the following in the command line:

gem install travis

After it is installed, you then need to encrypt the keys with the following commands. (This is best done in the directory of the project; otherwise you must add -r owner/project to the command)

travis encrypt node_pre_gyp_accessKeyId=${node_pre_gyp_accessKeyId} travis encrypt node_pre_gyp_secretAccessKey=${node_pre_gyp_secretAccessKey}

This will then output something like secure: “…….. encrypted ………” which can be put into your .travis.yml like so:

env:
  global:
     - secure: xb95WkJu5sab……………
     - secure: wugo4xhy7hO5SErSxy……………

On Appveyor, the project owner needs to be the one to encrypt the keys. To do this, you must be signed in as the project owner, and navigate to the encrypt data page. On this page, you simple copy and paste your access key and secret access key into the box and get the results. To then use these keys, embed them into your appveyor.yml like so:

environment:
  node_pre_gyp_accessKeyId:
    secure: BB4XzOLXvG+XMmXyCVwqaFF/hKJnYvjIACQMD423QvY=
  node_pre_gyp_secretAccessKey:
    secure: BFW25HQ0Jdq8IpysC59Y2mh1J9IsXwWB+FTElHVSYVl5xDwwZPHrXwctZQLzKrw9

You can read the readme of node-pre-gyp for travis and appveyor if you want to know more.

We didn’t always want to publish binaries when they were run through CI though, so we needed to put a safeguard in place. That was to only publish module binaries when the the commit message contained [publish binary]. We prepend version bumps with this, to trigger building and publishing in the CI server. eg, [publish binary] 0.3.0 would cause a publish.
There is already an existing module which essentially has scripts which does everything that we needed, which is the node-sqlite3 module by mapbox. This module had many scripts which I was able to repurpose for our uses under the BSD license.

Basically, if you want do something similar, take the /scripts/ folder from native-hdr-histogram, along with the Makefile, and copy and modify the appveyor.yml and .travis.yml files as you need. It looks scarier than it is, they’re already configured to handle node 4-6 for you. You might need to change the test target in the Makefile depending on your setup.

You also need to modify your package.json, and add a binary attribute. The module_path, remote_path, and package_name can be copied as is, but the host must be changed to match your S3 bucket URL. The module_name is the name of the compiled binary, I named it “histogram” for simplicity.

Additionally, using node-gyp requires you to create a binding.gyp file. When using node-pre-gyp, you must modify this file to include a “target” with the exact same content as this one, and you should modify your normal build target to use the module_name value used in your package.json, by setting its “target_name” to “<(module_name)”.

Your index.js, or rather, the main file that you export from your package.json should require the native file for exporting, using node-pre-gyp. It should end up similar to the one from here, assuming your binary section was similar to mine.

Now, when you want to publish a binary, all you need to do is add [publish binary] to your commit and push it to Github. It will take a while, because appveyor is kinda ever so slightly slow, but in the meantime if a user does download your native module without the published version available, they will simply build it themselves locally.

Conclusion

So, I’m sorry this was super long, technical, and didn’t have many examples, but it is one of the cooler technical things I have worked on recently that I wanted to share with people. Using this, we can create really nice automated publishing toolchains for any native Node.js modules and isn’t that what everyone wants?

You may find these articles about Node.js modules helpful as well.

• Node.js Modules – NearForm bootcamp series
• The future of Native Modules in Node.js