This post originated from an RSS feed registered with Ruby Buzz by Aslak Hellesøy. Original Post: Replacing NPM with Make Feed Title: Aslak Hellesøy's uncommon sense Feed URL: http://aslakhellesoy.com/rss Feed Description: Ruby-related blog posts from Aslak Hellesøy Latest Ruby Buzz Posts Latest Ruby Buzz Posts by Aslak Hellesøy Latest Posts From Aslak Hellesøy's uncommon sense

Package management at a high level is a fairly simple concept. A software package can be uploaded to—and downloaded from—a central location.

The Node.js Package Manager NPM does this, and a whole lot more. If you run npm help you’ll be presented with 64 different commands. I have been using NPM for about a year, and I have only ever used 4 or 5 of these. If someone did a study of NPM and who uses what commands I expect they would find a power-law relationship.

As you might expect from a feature-rich tool like this, the codebase is significant. Almost 10.000 lines of code spread across over 100 files according to CLOC. 1.800 issues have been closed since the project’s inception, and there are 100 open ones. These numbers illustrate that NPM is a very complex piece of software.

If all you need is uploading and downloading files there must be a simpler way.

It should Just Work

I work in a place where we use a lot of different programming languages. It’s important that a developer can clone a git repository and have the code “just work” without the need to learn and install a lot of additional software.

The ruby ecosystem is notoriously bad about this. There are many incompatible implementations of ruby. Then there are tools like rvm, bundler and rubygems. For someone who only occasionally works with Ruby this is a huge barrier, especially since there are so many moving parts and things tend to break nilly willy.

One of the really nice things about Node.js is that the executable is a single node binary with everything inside it. We often add a precompiled node binary to our git repositories. This removes a lot of tool complexity, and we reduce the risk of developers using different (and incompatible) versions of Node.js. This is especially important for a young project like Node.js where the APIs change frequently.

Node.js 0.6.3 and newer bundle NPM. This doesn’t mean NPM is bundled inside the node binary. Remember - NPM is a hundred files. Checking in NPM alongside node isn’t quite as convenient. If all you need is to pull down NPM packages there is a much simpler solution:

Make

Downloading NPM packages with Make and cURL

Below is a typical Makefile from one of the Node.js projects I work on. It pulls down NPM modules from the NPM registry as well as alternative locations. We sometimes pull them from a github tag, sometimes from an internal HTTP server (for our proprietary modules).

node_modules: Makefile
    @rm -rf $@
    @mkdir -p $@
    $(call get_src_module,$@,\
    https://github.com/tastapod/node-imap/tarball/bruno-merge)
    $(call get_npm_module,$@,log,1.2.0)
    $(call get_npm_module,$@,connect,1.8.2)
    # We have to manually fetch connect's dependencies
    $(call get_npm_module,$@,qs,0.4.0)
    $(call get_npm_module,$@,mime,1.2.4)
    $(call get_npm_module,$@,formidable,1.0.8)
    @touch $@

get_npm_module = $(call get_node_module,$1,$2,\
    http://registry.npmjs.org/$2/-/$2-$3.tgz,package)

get_src_module = $(call get_node_module,$1,$2,$3,)

get_node_module = @set -e; mkdir -p $1/$2; \
    echo "Fetching module $2 "; \
    curl -L --silent $3 | tar xzf - \
    --strip=1 --directory=$1/$2 $4

You can find the code in this gist (with tabs intact).

Publishing NPM packages with Make and scp

We have some proprietary Node.js libraries that we make available on a HTTP server. We considered installing our own NPM Registry Server, but quickly decided that this was more complex than it needs to be.

We settled for an nginx server and scp for uploading packages. Make to the rescue again:

NAME    := $(shell node -e "console.log(JSON.parse(require(\
'fs').readFileSync('package.json', 'utf8')).name)")
VERSION := $(shell node -e "console.log(JSON.parse(require(\
'fs').readFileSync('package.json', 'utf8')).version)")
TARBALL := $(NAME)-$(VERSION).tgz

npm-publish:
    @rm -Rf package
    @mkdir package
    @cp -R lib package/lib
    @cp package.json package
    @tar czf $(TARBALL) package
    @scp $(TARBALL) npm@npm:/home/npm/repo

clean:
    rm -Rf *.tgz

.PHONY: npm-publish clean

You can find the code in this gist (with tabs intact).

Conclusion

I really appreciate the hard work that goes into the Node.js ecosystem, including NPM. I just wish things were simpler.

Thanks to Joe Walnes for coming up with the ingenious idea to download NPM packages with Make and cURL and Dan North for showing me.