Provide API for creating huge batches
Author: mperhamCreated Sep 21, 2023Updated Oct 26, 2023
If you want to create a batch with 100,000s of jobs today, it can be tricky to do it in a way which is safe and efficient. Essentially we want a transaction:
loop do
create 1000 jobs
push 1000 to Redis
end
commit jobs to batchIdeally this API would mesh well with Enumerators and Enumerables too, e.g. mapping a set of one million CSV rows into 1000 windows of 1000 jobs, seamlessly.
See https://github.com/sidekiq/sidekiq/issues/5929#issuecomment-1593200817
Source: sidekiq/sidekiq