GVL and Health monitoring
The Past
Since forever, Sidekiq has been relatively static in its runtime. You configure the concurrency, Sidekiq creates that many Threads and it executes jobs on those Threads concurrently.
This has the problem that it is quite easy for a user to overload the Ruby process and cause significant GVL contention and slowdown. Recent advances in GVL instrumentation now make it possible to monitor overall system contention and determine if the configured concurrency is leading to poor performance.
The Present
How can we use these new APIs and tools to improve Sidekiq? Thoughts:
- Today GVL monitoring requires Ruby 3.2+ and a native gem to use. Sidekiq has a standing policy to not depend on any native gems.
- The GVL instrumentation will add some amount of overhead, we need to determine that overhead and a usage pattern that is workable in production. Byroot reports 1-5% overhead on a saturated multi-threaded process, which seems like a lot.
The Future
Right now the Launcher runs the heartbeat every 10 seconds. I feel like this is a good place to capture metrics. We can use a ring buffer to store the last 6 items and take action every 60 seconds if the metrics look "bad". This means brief blasts of heavy contention will be allowed but persistent poor performance will trigger action, which should ease any "flapping".
- The feature is code named "Autotune". [insert T-Pain gif]
- Should this functionality be optional or required? Seems like making it optional would be a good thing to ensure the user opts into the feature and understands any limitations.
- Should it scale up or down Processor threads? Should we have a notion like Puma of
threads min:max? - Log a warning? Emit some sort of notification?
- Show some alert or data in the Web UI?
Ideas and feedback welcome.
Resources
Source: sidekiq/sidekiq