python-systemd-tutorial

Writing a systemd Service in Python

Many Linux distributions use [systemd] to manage the system's services (or daemons), for example to automatically start certain services in the correct order when the system boots.

Writing a systemd service in Python turns out to be easy, but the complexity of systemd can be daunting at first. This tutorial is intended to get you started.

When you feel lost or need the gritty details, head over to the [systemd documentation][systemd], which is pretty extensive. However, the docs are distributed over several pages, and finding what you're looking for isn't always easy. A good place to look up a particular systemd detail is [systemd.directives], which lists all the configuration options, command line parameters, etc., and links to their documentation.

Aside from this README.md file, this repository contains a basic implementation of a Python service consisting of a Python script (python_demo_service.py) and a systemd unit file (python_demo_service.service).

The systemd version we're going to work with is 229, so if you're using a different version (see systemctl --version) then check the systemd documentation for things that may differ.

System and User Services

systemd supports both system and user services. System services run in the system's own systemd instance and provide functionalities for the whole system and all users. User services, on the other hand, run in a separate systemd instance tied to a specific user.

Even if your goal is to develop a system service it is a good idea to start with a user service, because it allows you to focus on getting the service up and running before dealing with the complexities of setting up a system service. Most of this tutorial targets user services, but there's a section at the end on how to go from a user service to a system service once you're ready.

Creating a User Service

Unit Files

To create a systemd service you need to create a corresponding unit file, which is a plain-text, ini-style configuration file. For this tutorial we will use a simple self-contained unit file, see [systemd.unit] for advanced approaches.

Unit files for user services can be put in several places. Some of these require root access, but there are multiple possible places in your home directory. As far as I can tell, there is no established default choice for these, so for this tutorial we are going to use ~/.config/systemd/user/.

Therefore, store the following unit description as ~/.config/systemd/user/python_demo_service.service:

[Unit]
# Human readable name of the unit
Description=Python Demo Service

Once you have done this, systemd will find our service:

$ systemctl --user list-unit-files | grep python_demo_service
python_demo_service.service         static

The unit options for systemd services are documented in [systemd.service].

Connecting the Service to a Python Script

We can now start to write the actual Python code for the service. Let's start small with a script that simply prints a message every 5 seconds. Store the following script as python_demo_service.py in a directory of your choice:

if __name__ == '__main__':
    import time

    while True:
        print('Hello from the Python Demo Service')
        time.sleep(5)

To link our service to our script, extend the unit file as follows:

[Unit]
Description=Python Demo Service

[Service]
# Command to execute when the service is started
ExecStart=/usr/bin/python path/to/your/python_demo_service.py

Manually Starting and Stopping the Service

Now our service can be started:

$ systemctl --user start python_demo_service

Depending on your systemd version, you may need to reload the user daemon so that our service can be found and started.

$ systemctl --user daemon-reload

Note that this command returns immediately. This is because systemd has created a separate process that runs our script. This means that we don't have to care about the nasty details of correctly forking into a daemon process ourselves, since systemd does all the work for us. Yay!

We can check that our service is running:

$ systemctl --user status python_demo_service
● python_demo_service.service - Python Demo Service
   Loaded: loaded (/home/torf/.config/systemd/user/python_demo_service.service; static; vendor preset: enabled)
   Active: active (running) since So 2018-12-30 17:46:03 CET; 2min 35s ago
 Main PID: 26218 (python)
   CGroup: /user.slice/user-1000.slice/[email protected]/python_demo_service.service
           └─26218 /usr/bin/python /home/torf/projects/python-systemd-tutorial/python_demo_service.py

In the first line of the output we can see the Description from our unit file. The output also tells us the state of our service and the PID it is running as.

Obviously our service can also be stopped:

$ systemctl --user stop python_demo_service
$ systemctl --user status python_demo_service
● python_demo_service.service - Python Demo Service
   Loaded: loaded (/home/torf/.config/systemd/user/python_demo_service.service)
   Active: inactive (dead)

STDOUT and STDERR

You might have noticed that the output of our script's print calls did not show up on your terminal. This is because systemd detached the service process from that terminal and also redirected the process's STDOUT and STDERR streams.

One thing to remember is that in Python, STDOUT and STDERR are buffered. When running in a terminal, this means that output will only show up after a newline (\n) has been written. However, our service's STDOUT and STDERR are pipes, and in this case the buffer is only flushed once it is full. Hence the script's messages only turn up in systemd's logs after it has produced even more output.

To avoid this effect we need to disable the buffering of STDOUT and STDERR, and one possibility to do so is to set the PYTHONUNBUFFERED environment variable. This can be done directly in our unit file by adding the following line to the [Service] section:

Environment=PYTHONUNBUFFERED=1

As always when you change your unit file you need to tell systemd to reload its configuration, and (if your service is currently running), restart the service:

$ systemctl --user daemon-reload
$ systemctl --user restart python_demo_service

The output from our script should now show up in systemd's logs, which by default are redirected to syslog:

$ grep 'Python Demo Service' /var/log/syslog
Dec 30 18:05:34 leibniz python[26218]: Hello from the Python Demo Service

Another way to display your service's output is via

$ journalctl --user-unit python_demo_service

There are many more possible configurations for logging. For example, you can redirect STDOUT and STDERR to files instead. See [systemd.exec] for details.

Automatically Starting the Service during Boot

Many services are intended to be started automatically when the system boots. This is easy to achieve using systemd. First we need to attach our service to a suitable target: targets are special systemd units that are used for grouping other units and for synchronization during startup. See [systemd.target] for details about targets in general and [systemd.special] for a list of built-in targets.

For user services, the default.target is usually a good choice. Add the following to your unit file:

[Install]
WantedBy=default.target

Our service is now ready to be started automatically, but for that to actually happen we have to enable the service first:

$ systemctl --user enable python_demo_service
Created symlink from /home/torf/.config/systemd/user/default.target.wants/python_demo_service.service to /home/torf/.config/systemd/user/python_demo_service.service.

If you restart your system now then our service will be started automatically once you log in. After your last session is closed, your user's systemd instance (and with it, our service) will shutdown. You can make your user's systemd instance independent from your user's sessions (so that our service starts at boot time even if you don't log in and also keeps running until a shutdown/reboot) via

$ sudo loginctl enable-linger $USER

To disable autostart, simply disable your service:

$ systemctl --user disable python_demo_service
Removed symlink /home/torf/.config/systemd/user/default.target.wants/python_demo_service.service.

Note that simply enabling a service does not start it, but only activates autostart during boot-up. Similarly, disabling a service doesn't stop it, but only deactivates autostart during boot-up. If you want to start/stop the service immediately then you still need to do that manually as described above in addition to enabling/disabling the service.

To check whether your service is enabled, use

$ systemctl --user list-unit-files | grep python_demo_service
python_demo_service.service         enabled

Automatically Restarting the Service after Failure

As with any other software, your service might crash. In that case, systemd can automatically try to restart it. By default, systemd will not do that, so you have to enable this functionality in your unit file.

systemd has several options to precisely configure under which circumstances your service should be restarted. A good starting point is to set Restart=on-failure in the [Service] section of your unit file:

[Service]
...
Restart=on-failure

This tells systemd to restart your daemon when it exits with a non-zero exit code. Other settings for Restart and related options are documented in [systemd.service]. As always you need to run systemctl --user daemon-reload for these changes to become effective.

We can simulate a crash by killing our service using the SIGKILL signal:

$ systemctl --user --signal=SIGKILL kill python_demo_service

Afterwards, the logs will show that systemd restarted our service:

$ journalctl --user-unit python_demo_service
[...]
Jan 31 12:55:24 leibniz python[3074]: Hello from the Python Demo Service
Jan 31 12:55:29 leibniz python[3074]: Hello from the Python Demo Service
Jan 31 12:55:32 leibniz systemd[1791]: python_demo_service.service: Main process exited, code=killed, status=9/KILL
Jan 31 12:55:32 leibniz systemd[1791]: python_demo_service.service: Unit entered failed state.
Jan 31 12:55:32 leibniz systemd[1791]: python_demo_service.service: Failed with result 'signal'.
Jan 31 12:55:33 leibniz systemd[1791]: python_demo_service.service: Service hold-off time over, scheduling restart.
Jan 31 12:55:33 leibniz systemd[1791]: Stopped Python Demo Service.
Jan 31 12:55:33 leibniz systemd[1791]: Started Python Demo Service.
Jan 31 12:55:33 leibniz python[3089]: Hello from the Python Demo Service
Jan 31 12:55:38 leibniz python[3089]: Hello from the Python Demo Service
[...]

Notifying systemd when the Service is Ready

Often, a service needs to perform some initializiation before it is ready to perform its actual work. Your service can notify systemd once it has completed its initialization. This is particularly useful when other services depend on your service, since it allows systemd to delay starting these until your service is really ready.

The notification is done using the [sd_notify] system call. We'll use the [python-systemd] package to execute it, so make sure it is installed. Then add the following lines to our script:

if __name__ == '__main__':
    import time
    import systemd.daemon

    print('