Increase/Decrease number of threads using signals

Hi there!

I'm working on an application that we avoid restarting it as much as possible due to the slow boot process (a majestic monolith).

We also monitor the resources usage such as CPU and MEM to fine-tune the number of processes and threads during the day and we are required to restart the processes/containers when we need to change this.

Similarly to Puma would be nice to have unix signals to increase and decrease the number of threads and/or processes.

In that specific project we don't use sidekiq-enterprise but thinking of OSS and Enterprise we could have both options for swarm and for the individual processes.

Let's say my sidekiq pid is 1:

kill -TTIN 1 # increase one thread
kill -TTOUT 1 # decrease one thread

Sidekiq currently uses TTIN to print the backtraces but we could change that to INFO as Puma does or simple use USR1/USR2 instead of TTIN/TTOUT.

With sidekiq-swarm, we could use those same signals to the swarm process to increase/decrease the number of forks and the individual forks to increase/decrease the number of threads.

Another viable option would be a Ruby Sidekiq API but this would require at least a rails console or an irb instead of simple unix signals, i.e.:

ps = Sidekiq::ProcessSet.new
ps.each do |process|
  process.increment_threads!
  process.decrement_threads!
end

# for swarm
swarm = Sidekiq::Swarm.new
swarm.increment_processes!
swarm.decrement_processes!

That could unlock powerful features such as autoscaling running processes accordingly to the actual machine usage.

We have some apps where it's a mix of IO-bound and CPU-bound jobs and their usage is not predictable.

We could monitor the CPU usage and while it's <25% I could double the number of processes on that container and once it reaches something like >=80% we potentially would double down the number of processes (or threads).

This can make more sense for non-scalable environments, outside the cloud, like 37Signals cloud exit, or in our use-case where the customer has its own dedicated physical servers due to data compliance requirements.

This could potentially be used to monitor foreign metrics such as CPU/memory of the database where I might have enough servers for CPU-bound but once sidekiq is occupied too much with one type or another or a mix, the database might struggle and shut down.

We could monitor the database CPU over 80% and double down the number of processes or threads and when it goes below 20%, double up.

We could scale up/down accordingly to those metrics or any other potential metric instead of booting/shutting down containers.

This could tie to this approach mentioned by Judoscale: https://judoscale.com/blog/introducing-proactive-autoscaling

Sometimes, autoscaling only based on queue time is not enough or adequate.

I'm brain storming here so we can think into new possibilites.