`pasta` and `conmon` processes are sometimes left in calling process' control group
Container host
- Podman: 5.8.3
- conmon: 2.1.12
- pasta: commit
587980c - systemd: 257.13
- Running rootless
Development machine
- Podman: 5.8.3
I have a web service (herein called deployer) which invokes Podman (using Node.js' child_process.spawnSync()) to create, start and stop containers. The deployer service is managed as a systemd unit.
Note that in control group paths, I've used /.../ as a shorthand for /sys/fs/cgroup/user.slice/user-33.slice/[email protected]/.
Problem: When I attempt to stop or restart the deployer service, systemd hangs while waiting for all remaining processes in the control group /.../app.slice/deployer.service to exit. This never happens, since the conmon and pasta processes created by Podman when deployer starts containers are left in the control group. conmon is not moved to its own libpod-conmon-<CONTAINER_ID>.scope control group as it usually is.
After 60 seconds, systemd kills the conmon and pasta processes, killing the containers they support. With --restart=always set on the containers, they are restarted. That said, no pasta process is created when restarting them, making the containerised application inaccessible over the network.
When running Podman in an SSH session, the pasta process lands in the control group /.../user.slice/podman-<PID>.scope and becomes a child of the system instance of systemd (PID 1). This allows the shell to exit without killing pasta or waiting for it to exit. That said, this transient control group is only supposed to live as long as the podman process does, but is kept alive by pasta.
In some cases—I forget exactly which—the user instance of systemd complains in the journal about "left-over process[es]":
Aug 11 02:20:15 <HOST> systemd[993]: podman.service: Found left-over process 3323 (pasta) in control group while starting unit. Ignoring.
Aug 11 02:20:15 <HOST> systemd[993]: podman.service: Found left-over process 3927 (pasta) in control group while starting unit. Ignoring.
Aug 11 02:20:15 <HOST> systemd[993]: podman.service: Found left-over process 24491 (pasta) in control group while starting unit. Ignoring.
Aug 11 02:20:20 <HOST> systemd[993]: podman.service: Unit process 3323 (pasta) remains running after unit stopped.
Aug 11 02:20:20 <HOST> systemd[993]: podman.service: Unit process 3927 (pasta) remains running after unit stopped.
Aug 11 02:20:20 <HOST> systemd[993]: podman.service: Unit process 24491 (pasta) remains running after unit stopped.Possible fix: Explicitly move any spawned support processes (e.g., conmon, pasta, slirp4netns perhaps) to a dedicated control group like conmon sometimes is; libpod-support-<CONTAINER_ID>.scope may be a more accurate name.
I've tested several different methods of invoking Podman and recorded the parent processes and control groups of the spawned processes. I've left notes ("⚠️") wherever I think behaviour may not be correct.
Running Podman on development machine using remote system connection.
Details ⚠️podman→ SSH connection →sshd-session→podman.socket→podman.service→ ...podman run \ --connection=foo \ --rm \ --detach \ --publish=1234:1234 \ alpine sleep 1000 podman run \ --connection=foo \ --rm \ --tty \ --interactive \ --publish=1234:1234 \ alpineProcess Parent process Control group podmansystemd(user instance)/.../app.slice/podman.service conmonsystemd(user instance)/.../user.slice/libpod-conmon-<CONTAINER_ID>.scope pastasystemd(user instance)/.../app.slice/podman.service
⚠️ Being daemonless, Podman's service unit should only live as long as thepodmanprocess.sleep/shconmon/.../user.slice/libpod-<CONTAINER_ID>.scope/container Running Podman on the host via SSH session.
Details ⚠️podman run \ --rm \ --detach \ --publish=1234:1234 \ alpine sleep 1000 podman run \ --rm \ --tty \ --interactive \ --publish=1234:1234 \ alpineProcess Parent process Control group podmansshd-session→zsh/.../user.slice/podman-<PID>.scope conmonsystemd(PID 1)
⚠️ All other methods cause this process to be parented by the user instance of systemd, not PID 1./.../user.slice/libpod-conmon-<CONTAINER_ID>.scope pastasystemd(PID 1)
⚠️ See above./.../user.slice/podman-<PID>.scope
⚠️ Being transient, Podman's scope unit should only live as long as thepodmanprocess.sleep/shconmon/.../user.slice/libpod-<CONTAINER_ID>.scope/container Running Podman on the host from a user service (deployer).
Details ⚠️podman run \ --rm \ --detach \ --publish=1234:1234 \ alpine sleep 1000Process Parent process Control group podmannode/.../app.slice/deployer.service conmonsystemd(user instance)/.../app.slice/deployer.service
⚠️ This process prevents the deployer service from being restarted without killing containers.pastasystemd(user instance)/.../app.slice/deployer.service
⚠️ See above.sleep/shconmon/.../user.slice/libpod-<CONTAINER_ID>.scope/container Running Podman with
--cgroup-parenton the host from a user service (deployer).⚠️ This option seems to have no effect, even with an explicit
Details ⚠️--cgroupsoption. Is the example below incorrect?podman run \ --rm \ --detach \ --publish=1234:1234 \ --cgroups=enabled \ --cgroup-parent=user.slice \ alpine sleep 1000Process Parent process Control group podmannode/.../app.slice/deployer.service conmonsystemd(user instance)/.../app.slice/deployer.service
⚠️ This process prevents the deployer service from being restarted without killing containers.pastasystemd(user instance)/.../app.slice/deployer.service
⚠️ See above.sleep/shconmon/.../user.slice/libpod-<CONTAINER_ID>.scope/container Running Podman via
systemd-runon the host from a user service (deployer).This works but adds some complexity for the caller. deployer only requires this for calls to
Details ⚠️podman start, not other invocations of Podman.systemd-run \ --user \ --scope \ --slice=user.slice \ --unit=podman.scope \ podman run \ --rm \ --detach \ --publish=1234:1234 \ alpine sleep 1000Process Parent process Control group podmannode/.../user.slice/podman.scope conmonsystemd(user instance)/.../user.slice/podman.scope
⚠️ This works, but differs from the intended control group for this process.pastasystemd(user instance)/.../user.slice/podman.scope
⚠️ See above.sleep/shconmon/.../user.slice/libpod-<CONTAINER_ID>.scope/container Running Podman with option
--cgroups=spliton the host from a user service (deployer).⚠️ This alters the control group hierarchy, but does not move support processes to a separate control group to that which the user service runs in. Each invocation of Podman creates a new "runtime" control group and moves the user service into it, eventually leading to an infinite stack of control groups, e.g., ...deployer.service/runtime/runtime/runtime/[...]/runtime
Details ⚠️podman run \ --rm \ --detach \ --publish=1234:1234 \ --cgroups=split \ alpine sleep 1000Process Parent process Control group podmannode/.../app.slice/deployer.service conmonsystemd(user instance)/.../app.slice/deployer.service/runtime
⚠️ The parent control group is split, and this new "runtime" group is never removed. The user service is also moved to this new group and never moved back.pastasystemd(user instance)/.../app.slice/deployer.service/runtime
⚠️ See above.sleep/shconmon/.../app.slice/deployer.service/libpod-payload-<CONTAINER_ID>
The system states above were verified with these commands:
pstree --compact-not --hide-threads --show-pids --longpstree --compact-not --hide-threads --show-pids --long --show-parents <PID>systemctl --user statussystemctl --user status <PID>
In summary:
- Methods 1 and 2 probably should have the same result, but do not.
- Methods 3 and 4 have the same result.
- Method 5 has allowed me to work around this issue.
Source: podman-container-tools/podman