Concurrent idempotent submissions create duplicate jobs for the same owner and key

Summary

Concurrent submissions using the same owner and idempotency key can create multiple QUEUED jobs. The background execution facade forwards request.idempotency_key to JobService.create_job; each successful insert can be scheduled separately. Memory ingestion also uses this dedupe mechanism.

Affected version

Verified against main@df9711c952a8e798e8fbbad8f25fe60be5ff6018 and release-1.13.0@33cedec5b299c7e2e8072627b8ef62460254da5b. The affected service file is identical on both. Python 3.12.14, real SQLite and PostgreSQL 16 databases.

Reproduction

1. Start with no jobs for an owner/key pair.
2. In four independent database sessions, concurrently call JobService().create_job(job_id=uuid4(), flow_id=flow_id, user_id=owner, dedupe_key="batch").
3. Introduce a short delay before INSERT/flush to reproduce the valid interleaving where all four SELECT checks finish before any INSERT commits.
4. Gather results and query the jobs table for that owner/key.

The regression uses real database connections and SQL, with 50 ms insert latency injected through an AsyncSession subclass. It covers regular/empty keys and UUID/NULL owners on SQLite, PostgreSQL READ COMMITTED, and PostgreSQL REPEATABLE READ: all 12 concurrent cases fail before the fix with assert 4 == 1 successful jobs. Thirty existing-behavior cases in the same test module pass on that baseline.

Expected behavior

One request creates a job; the other three receive DuplicateJobError, allowing callers to reuse the existing job. Failed, cancelled and timed-out jobs remain retryable. Different owners remain independent.

Actual behavior

All four requests create different QUEUED job IDs for the same owner/key.

Impact

High: retries or concurrent requests can schedule repeated workflow/tool executions or ingestion work. External side effects depend on the submitted workflow; the directly verified defect is duplicate persisted jobs. No cross-owner access or security boundary bypass is claimed.

Root cause evidence

src/backend/base/langflow/services/jobs/service.py, JobService.create_job, performs an owner-scoped count followed by INSERT in separate concurrent transactions. There is no lock or uniqueness constraint covering the read/insert pair. services/background_execution/service.py passes idempotency_key to this method and handles DuplicateJobError, which is never raised by this interleaving.

Duplicate search

Searched Issues/PRs across all states for dedupe_key, idempotency/race and concurrent dedupe, plus the earlier audit's path history and Discussions searches. #13507/#13633 introduce the background-job paths; #13508's job-service changes concern worker claims and heartbeats, not create-time key reservation. Trigger event-ledger dedupe in #14956/#15234 and Slack triggers in #15333 use a different persistence boundary. No equivalent owner/key atomic-creation fix was found; latest main and release heads were rechecked before publication.

Additional context

A fix should work across connections/processes on both supported database families, retain NULL-owner and retry semantics, and avoid a unique-index migration that could fail on historical duplicate rows. The proposed fix uses transaction-scoped database locks without schema changes.