Recently, I needed to write a collection of upstart jobs to start the backend for a project at work. Upstart is a great tool for starting jobs in a particular sequence, but getting everything just right was challenging.

The startup dependencies for the system are shown in Figure 1. The application uses Redis, Celery, and a Node.js server to provide asynchronous communications and task execution for a Django application. Nginx and uWSGI are used to provide access to the Django application.

deps

Figure 1 – Required startup dependencies.

The Configuration Files

Starting the redis-server

The first component of the system to start is the Redis key-value store. The application uses Redis as a communications hub for Celery tasks, the Node.js server, and the Django application. The actual configuration of Redis is omitted, and the upstart file follows:

Upstart files aren’t exactly scripts but are instead configuration files that are broken down into a set of stanzas. Some of the stanzas may be repeated, while others appear at most once. The first stanza in this file, on line 5, is the description stanza which is a quoted one-liner that provides a brief description of the task. The next two stanzas, start on and stop on are used to indicate when a job starts and stops respectively. In this case, it is desirable for redis-server to start when the system is running in multi-user mode with run levels 2-5. Similarly, redis-server should stop when the system is halting, in single-user mode, or rebooting, which are run levels 0,1, and 6 respectively.

The next stanza is the expect stanza that lets upstart know how a job started via an exec or script stanza is going to fork. If an expect stanza is provided it must have as its body one of forkdaemon, or stop. The fork option tells upstart that the job will fork once, while the daemon option indicates that the job will fork twice. The stop option indicates that the job will raise a SIGSTOP when it is ready to go. In the case that your job doesn’t fork, don’t use expect. If your job forks more than twice, you’ll need to get creative since upstart doesn’t readily handle that situation.

The respawn stanza will restart the job if it died unexpectedly. Be warned that if your job’s expect stanza is set incorrectly, the respawn stanza can lead to unexpected behavior. It is important to be able to cleanly start and stop the upstart job before setting the restart stanza. Failure to get the expect stanza correct leads to general annoyance with upstart.

The next four env stanzas create environment variables for the tool. In this example the user and group for execution are provided as well as the directory specification for a work directory. In addition to being defined within the upstart file, environment variables may be passed in as an option to the jobs control tools or passed between related jobs. For more details see the environment variables documentation.

The pre-start stanza defines either a single action with an exec or a script block that is executed before the task is kicked off. In this example a directory must be in place for the redis-server to stash its work files.

The last stanza is an exec stanza that kicks off the redis-server process. Since the redis-server needs to be run as a non-privileged user the start-stop-daemon was an easy way to make this happen. You will notice that there is no “start” stanza, nor is there a “stop” stanza. Starting is done in an unadorned exec or script stanza. Stopping the job is accomplished via a signal sent to the started job.

 Starting celeryd

The next component that needs to be started is the Celery distributed task queue to manage periodic and long-running tasks. In previous versions of celery there had been a celeryd script that could be used to start the server. In the current version (v3.1) the celeryd script is deprecated, so in keeping with a policy of not building new tools with deprecated tools, a new startup process was needed. Celery provides a tool called multi that we’ll use to start the task runners. The upstart configuration file follows:

Typically upstart keeps track of the PID of the job being run so it can kill it later. Additionally, the PID is used to determine if the job is actually running. When using the celery multi tool, the PID that upstart would capture belongs to a launcher that dies once the workers are started. In order to control the workers without directly launching them, the pre-start and pre-stop stanzas are used along with a dummy job that just sleeps. The pre-start stanza starts the celery multi tool that starts up the required workers and then dies leaving PID files for each worker behind in the $CELERYD_PID_FILE. Similarly, the pre-stop stanza stops the celery workers using the PID files stored in $CELERYD_PID_FILE. If the script stanza given in lines 24-30 were omitted, upstart would not detect that a job was running and consequently would never execute the pre-stop since the job would not be in the STARTED state.

The other interesting portions of this script are the start on and stop on stanzas. The start on stanza specifies that the celeryd job may start after the redis-server job has started. This ordering is important since celery is using the redis-server for managing its job queues. The stop on stanza ensures that the celery works are shut down before the redis-server.

 Starting celerybeat

After celeryd has been started, the celerybeat job may start. The celerybeat jobs uses the celery beat tool which is responsible for starting periodic tasks on the celery workers. The upstart configuration follows:

 

 Starting asyncserver

After redis-server has been started the asyncserver job can start. The asyncserver job launches a javascript program running on Node.js that provides a Socket.io based communications relay. The upstart configuration follows:

 

 Starting uwsgi

Once the asyncserver and celerybeat jobs have been started, the uwsgi job can start. The uwsgi job starts a uWSGI server that in this case provides an interface to a Django web application.   The upstart configuration follows:

While the upstart configuration is minimal, the start on and stop on stanzas are interesting. In order for the uWSGI jobs to be successful it depends on redis-server, celery-beat, and asyncserver all being started first. While the application also requires celeryd to have been started, that is implied by the dependency on celerybeat.

 Starting celeryflower

The celery flower tool is for monitoring and controlling jobs in the celery task queue. The celeryflower job can start once the celeryd job has started.  The upstart configuration follows:

 Starting nginx

The last job to start is nginx. Nginx is a HTTP and proxy server that is used to provide access to all of the other components and serve static content. The configuration of Nginx is beyond the scope of this post. The upstart configuration follows:

Verifying Dependencies

After loading all of the upstart configuration files, it’s a good idea to check that the dependencies are correct. One of the nicer tools provided is the initctl2dot helper that will generate an annotated dependency graph using Graphviz. Invoking the initctl2dot as:

produces:

Figure 2 – Startup dependencies returned by initctl2dot.

Figure 2 – Startup dependencies returned by initctl2dot.

Final Thoughts

Upstart does solve a number of problems, and makes it relatively easy to start well-behaved jobs. When a job doesn’t work exactly in the manner that upstart expects, for example celery worker in celeryd, things can get a little strange. In the course of writing these configurations the Upstart Cookbook was useful, and trouble shooting was facilitated by digging into the logs in /var/log/upstart.