Batch action job ID can silently deduplicate a distinct later operation with a different query or target

Batch action job ID can silently deduplicate a distinct later operation with a different query or target

Summary

Generic batch actions use a BullMQ jobId derived only from:

projectId + tableName + actionId

The ID does not include request-specific inputs such as the batch query or targetId.

As a result, two distinct batch operations for the same project, table, and action type can resolve to the same BullMQ job ID even when they represent different user intent.

If an earlier job with that ID is still retained by BullMQ, submitting the later operation can be deduplicated against the earlier job instead of creating an execution entity containing the new request's payload.

This is particularly problematic after a failed batch action, because failed jobs are retained while the application no longer considers them "in progress."

A later request can therefore appear to submit normally while the retained BullMQ job still contains the earlier operation's query and target.

Affected code

generateBatchActionId() currently generates the batch action identity from only:

export const generateBatchActionId = (
  projectId: string,
  actionId: string,
  tableName: string,
) => {
  return `${projectId}-${tableName}-${actionId}`;
};

For generic batch actions, createBatchActionJob() then stores request-specific values such as query and targetId in the job payload:

payload: {
  projectId,
  actionId,
  tableName,
  cutoffCreatedAt: new Date(),
  query: queryWithSnapshot,
  targetId,
  type: actionType,
}

but submits the job using:

{
  jobId: batchActionId,
}

The code also explicitly describes this ID as being used for deduplication.

Therefore:

same project
+ same table
+ same action type
= same BullMQ jobId

even when:

query A != query B

and/or

target A != target B

Concrete example: Add traces to annotation queue

Consider two separate bulk operations in the same project.

Operation A

Object type: Trace
Query: QA
Target annotation queue: Queue-A

This creates a job similar to:

jobId:
<project>-Traces-trace-add-to-annotation-queue

payload:
query = QA
targetId = Queue-A

Assume this job eventually reaches a failed state and remains retained in BullMQ.

Operation B

The user later performs another valid bulk action:

Object type: Trace
Query: QB
Target annotation queue: Queue-B

where:

QA != QB
Queue-A != Queue-B

The generated job ID is nevertheless identical:

<project>-Traces-trace-add-to-annotation-queue

If the previous job is still retained, BullMQ custom job-ID deduplication can prevent creation of a new job for B.

The retained job still represents:

QA → Queue-A

rather than:

QB → Queue-B

Why the failed-job case matters

The application checks whether a batch action is in progress using states equivalent to:

waiting
delayed
active

A failed job is therefore no longer treated as an in-progress operation.

At the same time, the batch action queue configuration retains failed jobs:

removeOnFail: 10_000

This creates a lifecycle window where:

old job still exists in BullMQ
+
application considers the operation no longer in progress

A user can consequently submit another batch operation of the same action class while the old job ID is still occupied.

Retry interaction

The worker infrastructure also has retry handling for failed batch-action jobs.

This makes the retained payload significant: retrying the retained job operates on the original job entity and its original payload.

For the example above, the problematic lifecycle is:

A: QA → Queue-A
        ↓
job fails
        ↓
failed job remains retained
        ↓
application no longer reports action as in progress
        ↓
user submits B: QB → Queue-B
        ↓
B resolves to the same BullMQ jobId
        ↓
no independent job carrying B's payload
        ↓
retained job still contains QA → Queue-A
        ↓
old job is retried
        ↓
QA → Queue-A is processed

The later operation and the retained execution state therefore no longer necessarily describe the same request.

User-visible impact

For affected generic batch actions, this can potentially result in:

• a newly submitted batch operation not receiving its own execution job;
• the new query or target not being represented in the queue;
• an older failed operation remaining the executable payload;
• a later retry processing the older operation rather than the newly submitted one.

For annotation queues this is particularly visible because the new request can specify a different destination queue.

For example, a user can submit:

QB → Queue-B

while the retained job still represents:

QA → Queue-A

Expected behavior

Two materially different batch requests should either:

1. receive independent execution identities; or
2. be explicitly rejected/merged/replaced according to a defined deduplication policy that preserves the intended payload semantics.

A successful submission of a distinct operation should not silently depend on an older retained job whose payload represents a different query or target.

Actual behavior

Generic batch action identity is based on project, table, and action type rather than the specific operation.

Consequently, materially different operations can collide at the BullMQ job-ID layer while their differing inputs exist only in the payload.

Scope

This report concerns the generic queue-backed batch-action path.

TraceDelete should not be treated as an example of this exact issue because it has additional persistent batchAction state and explicit lifecycle/conflict handling.

Examples worth reviewing include generic operations such as:

• adding traces/sessions/observations to annotation queues;
• dataset batch deletion;
• score batch deletion.

Suggested remediation boundary

The fix does not necessarily require any specific ID format.

The important invariant is that a newly accepted operation with materially different execution inputs must not silently resolve to a retained execution entity representing an older operation.

Possible approaches include:

• assigning an operation-specific job identity;
• explicitly incorporating the relevant operation identity into deduplication;
• removing/replacing terminal retained jobs before admitting a successor;
• or explicitly detecting the collision and returning a conflict instead of acknowledging the later request as submitted.

Whichever approach is used, regression coverage should include two sequential operations with different queries and/or targets where the first operation remains retained after failure.

Evidence status

The identity collision and lifecycle path described above were verified against the Langfuse source, including the current main implementation.

A production report also demonstrates that trace-add-to-annotation-queue jobs can enter stalled/failed lifecycle states after worker failure/restart.

I have not completed a full Redis-backed end-to-end reproduction of the complete A → failure → B → retry sequence, so this report does not claim that such an end-to-end reproduction has already been executed.

The report is based on the source-level execution path and BullMQ's documented custom job-ID behavior.

Method attribution

Identified using ADECAS — Assured Decision, Evidence, Capability and Authority System.