Doing something before systemd shuts your supervisord down

If you are running your server applications via supervisord on a Linux distro running systemd, you may find this post useful.

Problem Scenario

An example scenario to help us establish the utility for this post is as follows:


What we want to do is prevent in-flight requests being dropped when a system is shutting down as part of a power off cycle (AWS instance termination, for example). We can do so in two ways:

  1. Our server application is intelligent enough to not exit (and hence halt instance shutdown) if a request is in progress
  2. We hook into the shutdown process above so that we stop new requests from coming in once the shutdown process has started and give our application server enough time to finish doing what it is doing.

The first approach has more theoretical “guarantee” around what we want, but can be hard to implement correctly. In fact, I couldn’t get it right even after trying all sorts of signal handling tricks. Your mileage may vary of course and if you have an example you have, please let me know.

So, I went ahead with the very unclean second approach:

When you are using a software like linkerd as your RPC proxy, even long-lived connections are not a problem since linkerd will see that your service instance is unhealthy, so it will not proxy any more requests to it.

Proposed solution implementation

The proposed solution is a systemd unit - let’s call it drain-connections which is defined as follows:

#  cat /etc/systemd/system/drain-connections.service

Description=Shutdown hook to run before supervisord is stopped
After=supervisord.service networking.service

ExecStop=/usr/local/bin/consul maint -enable
ExecStop=/bin/sleep 300



Let’s go over the key systemd directives used above in the Unit section:

  1. After ensures that drain-connections is started after supervisord, but stopped before supervisord
  2. PartOf ensures that drain-connections is stopped/restarted whenever supervisord is stopped/restarted

The Service section has the following key directives:

  1. Type=oneshot (learn more about it here)
  2. The first ExecStop first takes the service instance out of the pool by enabling consul maintenance mode
  3. The second ExecStop then gives our application 300 seconds to stop finishing what it is currently doing
  4. The TimeoutSec parameter override systemd default timeout of 90 seconds to 301 seconds so that the earlier sleep of 300 seconds can finish

In addition, we setup supervisord systemd unit override as follows:

# /etc/systemd/system/supervisord.service.d/supervisord.conf


This ensures that drain-connections service gets started when supervisord is started.


Let’s see how the above fits in to our scenario:

What if drain-connections is stopped first? That is okay, because that will execute the necessary commands we would want to be executed. Then, supervisord can be stopped which will stop our application server, but the drain-connections unit has already done its job by that time.