Systemd notify - PID not permitted
Author: DChalcraftCreated Mar 13, 2023Updated Mar 17, 2023
Ruby version: 3.1.2p20 Rails version: 7.0.4.2 Sidekiq / Pro / Enterprise version(s): sidekiq-7.0.6 / sidekiq-pro-7.0.7 / sidekiq-ent-7.0.5
sidekiq.yml
:concurrency: <%= ENV["RAILS_MAX_THREADS"]&.to_i || 5 %>
:queues:
- high
- default
- lowsudo sytemctl status sidekiq.service
...
● sidekiq.service - sidekiq
Loaded: loaded (/etc/systemd/system/sidekiq.service; enabled; vendor preset: enabled)
Active: activating (start) since Mon 2023-03-13 07:50:08 UTC; 27s ago
Main PID: 3788898 (einhorn)
Tasks: 11 (limit: 9362)
Memory: 258.4M
CPU: 5.079s
CGroup: /system.slice/sidekiq.service
├─3788898 "einhorn: /usr/local/bin/bundle exec sidekiq -e staging -c 1" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "">
└─3788908 "sidekiq 7.0.6 REDACTED [0 of 1 busy]" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
Mar 13 07:50:09 staging1-1 sidekiq[3788908]: [WORKER 3788908] INFO: About to exec ["/usr/local/bin/bundle", "exec", "sidekiq", "-e", "staging", "-c", "1"]
Mar 13 07:50:09 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:09.884Z pid=3788908 tid=295p8 INFO: Enabling systemd notification integration
Mar 13 07:50:11 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:11.471Z pid=3788908 tid=295p8 INFO: Filesystem mounts are ok!
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.032Z pid=3788908 tid=295p8 INFO: Booted Rails 7.0.4.2 application in staging environment
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.033Z pid=3788908 tid=295p8 INFO: Running in ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.033Z pid=3788908 tid=295p8 INFO: Sidekiq Pro 7.0.7 / Sidekiq Enterprise 7.0.5, commercially licensed.
Mar 13 07:50:14 staging1-1 systemd[1]: sidekiq.service: Got notification message from PID 3788908, but reception only permitted for main PID 3788898
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.037Z pid=3788908 tid=2buv8 INFO: SuperFetch[default] activated
Mar 13 07:50:14 staging1-1 sidekiq[3788898]: [MASTER 3788898] INFO: Received a manual ACK from 3788908
Mar 13 07:50:14 staging1-1 sidekiq[3788898]: [MASTER 3788898] INFO: Up to 1 / 1 manual ACKs
...sidekiq.service
# This file corresponds to a single Sidekiq process. Add multiple copies
# to run multiple processes (sidekiq-1, sidekiq-2, etc).
#
# Use `journalctl -u sidekiq -rn 100` to view the last 100 lines of log output.
#
[Unit]
Description=sidekiq
# start us only once the network and logging subsystems are available
After=syslog.target network.target nfs-server.service
RequiresMountsFor=/redacted /redacted
[Service]
Type=simple
WorkingDirectory=/**/**/**
ExecStart=/usr/local/bin/einhorn -m manual --reexec-as=/**/**/**/einhorn-redacted -- \
/usr/local/bin/bundle exec sidekiq -e {{rails_env}} -c {{sidekiq_worker_count}}
ExecReload=/usr/local/bin/einhornsh --execute upgrade
# Use `systemctl kill -s TSTP sidekiq` to quiet the Sidekiq process
# (However, note that `systemctl reload sidekiq` is preferable for rolling restarts)
Environment=RAILS_ENV={{rails_env}}
Environment=RAILS_LOG_TO_STDOUT=1
Environment=BOOM_DB_READ_TIMEOUT=120
Environment=MALLOC_ARENA_MAX=2
Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so
Environment=RAILS_MAX_THREADS=20
User=redacted
Group=redacted
UMask=redacted
RestartSec=1
Restart=on-failure
StandardOutput=journal
StandardError=journal
# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq
[Install]
WantedBy=multi-user.target
Possibly due to systemd expecting the notify to come from the Einhorn process rather than Sidekiq. Attemped to set NotifyAccess=all in sidekiq.service, but results in systemd killing it every 10 seconds.
Source: sidekiq/sidekiq