#5829·sidekiq

Systemd notify - PID not permitted

Author: DChalcraftCreated Mar 13, 2023Updated Mar 17, 2023

Ruby version: 3.1.2p20 Rails version: 7.0.4.2 Sidekiq / Pro / Enterprise version(s): sidekiq-7.0.6 / sidekiq-pro-7.0.7 / sidekiq-ent-7.0.5

sidekiq.yml

yaml
:concurrency: <%= ENV["RAILS_MAX_THREADS"]&.to_i || 5 %>
:queues:
  - high
  - default
  - low

sudo sytemctl status sidekiq.service

yaml
...
 sidekiq.service - sidekiq
     Loaded: loaded (/etc/systemd/system/sidekiq.service; enabled; vendor preset: enabled)
     Active: activating (start) since Mon 2023-03-13 07:50:08 UTC; 27s ago
   Main PID: 3788898 (einhorn)
      Tasks: 11 (limit: 9362)
     Memory: 258.4M
        CPU: 5.079s
     CGroup: /system.slice/sidekiq.service
             ├─3788898 "einhorn: /usr/local/bin/bundle exec sidekiq -e staging -c 1" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "">
             └─3788908 "sidekiq 7.0.6 REDACTED [0 of 1 busy]" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

Mar 13 07:50:09 staging1-1 sidekiq[3788908]: [WORKER 3788908] INFO: About to exec ["/usr/local/bin/bundle", "exec", "sidekiq", "-e", "staging", "-c", "1"]
Mar 13 07:50:09 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:09.884Z pid=3788908 tid=295p8 INFO: Enabling systemd notification integration
Mar 13 07:50:11 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:11.471Z pid=3788908 tid=295p8 INFO: Filesystem mounts are ok!
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.032Z pid=3788908 tid=295p8 INFO: Booted Rails 7.0.4.2 application in staging environment
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.033Z pid=3788908 tid=295p8 INFO: Running in ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.033Z pid=3788908 tid=295p8 INFO: Sidekiq Pro 7.0.7 / Sidekiq Enterprise 7.0.5, commercially licensed.
Mar 13 07:50:14 staging1-1 systemd[1]: sidekiq.service: Got notification message from PID 3788908, but reception only permitted for main PID 3788898
Mar 13 07:50:14 staging1-1 sidekiq[3788908]: 2023-03-13T07:50:14.037Z pid=3788908 tid=2buv8 INFO: SuperFetch[default] activated
Mar 13 07:50:14 staging1-1 sidekiq[3788898]: [MASTER 3788898] INFO: Received a manual ACK from 3788908
Mar 13 07:50:14 staging1-1 sidekiq[3788898]: [MASTER 3788898] INFO: Up to 1 / 1 manual ACKs
...

sidekiq.service

yaml
# This file corresponds to a single Sidekiq process.  Add multiple copies
# to run multiple processes (sidekiq-1, sidekiq-2, etc).
#
# Use `journalctl -u sidekiq -rn 100` to view the last 100 lines of log output.
#
[Unit]
Description=sidekiq
# start us only once the network and logging subsystems are available
After=syslog.target network.target nfs-server.service
RequiresMountsFor=/redacted /redacted

[Service]
Type=simple

WorkingDirectory=/**/**/**
ExecStart=/usr/local/bin/einhorn -m manual --reexec-as=/**/**/**/einhorn-redacted -- \
  /usr/local/bin/bundle exec sidekiq -e {{rails_env}} -c {{sidekiq_worker_count}}
ExecReload=/usr/local/bin/einhornsh --execute upgrade

# Use `systemctl kill -s TSTP sidekiq` to quiet the Sidekiq process
# (However, note that `systemctl reload sidekiq` is preferable for rolling restarts)

Environment=RAILS_ENV={{rails_env}}
Environment=RAILS_LOG_TO_STDOUT=1
Environment=BOOM_DB_READ_TIMEOUT=120
Environment=MALLOC_ARENA_MAX=2

Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so

Environment=RAILS_MAX_THREADS=20

User=redacted
Group=redacted
UMask=redacted

RestartSec=1
Restart=on-failure

StandardOutput=journal
StandardError=journal

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=multi-user.target

Possibly due to systemd expecting the notify to come from the Einhorn process rather than Sidekiq. Attemped to set NotifyAccess=all in sidekiq.service, but results in systemd killing it every 10 seconds.