Lessons from Production 21 Feb 2014

Adefy is the first Node JS project I’ve had to deploy into production, and the largest chunk of technology I’ve ever worked on. We have seperate codebases for our Platform (by far the largest), Android engine, WebGL/Canvas engine, AJS middleware, static website, and Developer Center.

The sheer amount of code that has to be deployed, and has to work perfectly is staggering. Dealing with it and creating a stable production environment has taught me many things which have changed the way I look at web app development. Yes, feature development is still a core component (you have to actually build the app), but in order to sleep well at night you need to trust that the code you send out to production is as damn near to perfect as you can get it.

How can I sleep at night? Tests

Tests are sexy. There’s no going around that. How do you know that one hacked up feature you and your buddies implemented two months ago at the bar still works, after refactoring DB access for 1/3rd of your codebase? Tests.

Disturbing lack of tests

Tests tests tests.

Before we merge anything into production, we run our growing collection of server and client-side unit tests. If the tests all pass, then we merge and push the production branch, run the tests again in production, then finally restart all running instances.

I’m not advocating TDD though. Writing tests first and logic second seems profoundly backwards to me. Instead, write your logic, but commit it with new tests. If you are ever not 100% confident about deploying to production, then you are likely missing some tests!

Don’t make deploys a ritual

The process of actually shipping new code should be as automated as possible. Beyond just confidence that the code works, you need to know that it’ll make it into production safely and predictably. Deploying should have as few points of human interaction as possible. If after initiating the deploy (whether by running a script, or pushing to a specific branch), you can’t immediately walk away from your PC and grab a pizza (or a drink!), something is horribly wrong.

Deploying to production

We currently use a combination of fabfiles, grunt, and server-side git hooks to run our deploys, but are moving towards a pure git + ansible combo. So although the way we do things isn’t perfect, I’ve seen enough of the light to preach it. Use ansible!

Do production Node right

You shouldn’t be running a lone Node instance in production. Use the node cluster module to spawn an instance for each CPU on the server, and pass over static file serving to Nginx. This frees up the instances to do actual important processing.

The Adefy platform is an angular client app; this means we can serve all client files purely with Nginx, and rely on the node instances only for API calls.

Do hot reloads! Don’t instantly kill your app server(s) for an update. Doing so drops active requests, which is a big no-no when you’re going for 100% uptime. Use something like PM2 to run and reload your app. PM2 also handles clustering for you, so that’s a neat added bonus.

What does all this mean? Automate!

Overall, you should automate and test as much as possible. Testing is just the automation of piece of mind. Get your codebase to the point where it maintains itself, and be smart about your production environment.

If you haven’t written tests before, try out a Mocha and Chai combo, and use something like PhantomJS for client-side testing. For easily testing an API, supertest and superagent are EPIC :D