#6046·sidekiq

Provide API for creating huge batches

Author: mperhamCreated Sep 21, 2023Updated Oct 26, 2023

If you want to create a batch with 100,000s of jobs today, it can be tricky to do it in a way which is safe and efficient. Essentially we want a transaction:

loop do
  create 1000 jobs
  push 1000 to Redis
end
commit jobs to batch

Ideally this API would mesh well with Enumerators and Enumerables too, e.g. mapping a set of one million CSV rows into 1000 windows of 1000 jobs, seamlessly.

See https://github.com/sidekiq/sidekiq/issues/5929#issuecomment-1593200817