Startup recovery leaves queued jobs leased when re-enqueue fails
Bug Description
During BackgroundExecutionService.sweep_orphans_on_startup(), each queued workflow is first claimed with claim_queued_lease(). If request reconstruction succeeds but _enqueue() raises, the exception is silently suppressed and the lease is not released.
The row remains QUEUED with a fresh owner/heartbeat even though it was never placed in the executor. In the default in-process backend, the orphan watchdog is not started, and the startup sweep is one-shot. The job can therefore remain invisible to recovery until another process restart (and until its lease becomes stale).
Reproduction
- Return one queued workflow from
_queued_workflow_jobs(). - Make
claim_queued_lease()succeed. - Make
_reconstruct_request()succeed. - Fault-inject
_enqueue()to raise. - Run
sweep_orphans_on_startup(). - Observe that
release_queued_lease()was never called and the exception was swallowed.
Expected behavior
If enqueue fails after a queued lease was claimed, log the failure and release that exact lease using its owner and heartbeat token. This restores the job's recoverability without allowing a stale claimant to clear a newer lease.
Environment
- Branch:
release-1.13.0 - Commit checked:
05b6498e8b54c2d89042afc2e2a96c452dd12002 - Backend: default in-process execution
- Python: 3.12
Source: langflow-ai/langflow