Assign a job_id at enqueue time
Every job resque enqueues is identified only by its payload. Two enqueues of the same class with the same arguments are indistinguishable, and several rough edges follow from that.
Not a replacement for failure_id
Worth stating up front, because the shapes look alike. They answer different questions and compose:
failure_id |
job_id |
|
|---|---|---|
| Assigned at | failure time | enqueue time |
| Identifies | one failure event | one enqueue |
| Correlates across | failure backends (Redis list, Airbrake, …) | a job's lifecycle: queued → working → failed → retried |
| Cardinality | a job that retries produces several | one per enqueue |
| Present when | only on failure | always |
failure_id answers "is the record in my failed list the same failure as the event in Sentry." job_id would answer "are these four failure records across three retries all the same original job," and "which of these five identical queued jobs is the one I selected."
They also compose for free: a failure record stores the full :payload (lib/resque/failure/redis.rb), so an id carried in the payload lands on every failure record without extra plumbing. A failure would then have both its own failure_id and the job_id of the job that produced it.
What it would buy
- Exact single-job operations. #1969 adds a Delete button to the queue page. It removes by payload, which is
LREM key 0 payload— so it takes every identical job with it, and the PR documents that rather than pretending otherwise. With an id, deleting one job means one job. - Retiring payload-byte matching in
Job.destroy.destroy(queue, klass, *args)reconstructsencode(class:, args:)andLREMs that. It matches on exact bytes and silently drops any other keys the stored payload carries. - Tracing one enqueue through its lifecycle, including across retries.
The hard part
This is a wire-format change to every enqueued job, which is why it is meaningfully harder than failure_id was — failures were fresh data resque already owned.
Job.destroybreaks the day this lands. An id in the payload meansencode(class:, args:)no longer matches what is in Redis, so exact-args destroy silently stops removing anything. It has to move to decode-and-compare on class and args, ignoring the id. That is a prerequisite, not a follow-up.- Deduplication plugins (resque-lock, resque-unique-job and similar) key off the payload. A per-job unique id makes every payload unique, which is precisely what they do not want. They need a way to hash the job ignoring the id.
- Jobs already in queues will not have one, so everything reading it must tolerate
nilindefinitely. - Redis memory, one more field per queued job, and resque's whole point is large queues.
Suggested shape, following the failure_id precedent
- Generate at
Job.create/Resque.push, with aResque.generate_job_ids?toggle mirroringResque::Failure.generate_job_ids?, defaulting on. The opt-out matters more here, given the wire-format and memory cost. - Let a caller pass an explicit id, the way
Resque::Failure.createaccepts:failure_id. - Carry the same honesty as
docs/FAILURE_IDS.md: this is not an index and gives no O(1) lookup. A queue is still a list. What the id buys is a check — read the element at a position, confirm itsjob_idis the one that was selected, then act. That is the pattern #1969 would use to delete exactly one job.
Related: #1969 (payload-based deletion and its limits), #1943 (failure_id), docs/FAILURE_IDS.md.
Source: resque/resque