Zero downtime deploy with Dokku

TL;DR we hacked Dokku to check-list new deployments before switching traffic over, eliminating down time. Here's our fork.

Dokku is a wonderful tool for quickly deploying apps to production, but it's missing zero downtime deploys. Not anymore!

Even our svelte Node.js app needs a few seconds to fully load the main code, and open connections to back-ends servers, before it can start handling requests.

The thing with Dokku: it switches traffic over to the new container before the server loads complete, so each deploy has several seconds of downtime.

That's best case scenario. When we screw deployment (usually some configuration issue that's different in production), even the quickest hands in the west, you still get a few minutes of downtime.

And so it became necessary that I learn how to catch and handle exceptions in Bash (yay, traps!) and shift a few lines of code around.

Checks Your Engine

The main piece is a new plugin that checks the newly deployed server.

The plugin runs through a list of checks, each check tells it to request a given URL and verify its contents. If all checks pass, then it's safe to switch traffic to the new server.

Checks looks something like this:

# Run these checks against server to verify it works correctly
# Each check takes the form of:
#   <pathname> <expected contents of page>
#
# For example:
#   /about     Our Amazing Team

/                       My Amazing App
/stylesheets/index.css  .body
/scripts/index.js       $(function()
/images/logo.png

You don't have to fuss with the expected content, but it helps against cases where templates don't load properly, LESS styles fail to compile, etc.

Depending on your application, these checks can also warm up the cache, making sure the first live requests are served with dispatch.

You add the checks to a file called CHECKS in the app's root directory. For example, if your app is called web, you'll need to create the file /home/dokku/web/CHECKS.

Eliminate The Dead Zone

Besides a plugin, there are also some changes to the main Dokku script. The reason for doing both: you can run a different deploy check plugin if you don't like mine.

The changes to Dokku work as follows:

1. Create a new container, deploy the app, start the web server (not much changed here)
2. Run the check-deploy hooks, which runs through the CHECKS list
3. Run the post-deploy hooks, which update the Nginx configuration and route traffic to the new server
4. Now get rid of the old container

If the checks fail, traffic is still going to the old container, so we just delete the new container and fail the push.

And that's all there is to it: Dokku with zero downtime deploy.

Oh, HTTPS

If you want to use our fork of Dokku, please note that we also changed the behavior for HTTPS traffic.

If you have an SSL certificate, it will accept HTTPS requests, but also allow HTTP requests. Dokku's current behavior is to redirect all HTTP requests to HTTPS.

The reason for this change: IE. We would like to serve all our resources over HTTPS, but we do allow our customers to embed client-side JavaScript on their site, fetches content from our servers.

IE's implementation of CORS (XDR), for reasons that defy logic or consistency with real web browsers, won't access HTTPS resources from pages loaded over HTTP.

To Conclude

Install:

git clone git@github.com:broadly/dokku.git
sudo make install

Add some checks to the file $DOKKU_ROOT/$APP/CHECKS.

Worry less, sleep better, look younger.