Harden label printing against concurrent delivery and interrupted jobs
Please verify that this bug has NOT been raised before.
- I checked and didn't find a similar issue
Describe the bug*
Related to the #12872 that preserves label-print failures in DataOutput and skips redelivery of failed jobs. The problem is a bit theoretical and ignoring it as 'too unlikely' is reasonable; it is likely that other issues have higher priority.
The #12872 fixes failure reporting and prevents printing again when a later delivery observes a recorded failure. However, label printing still has concurrency and interruption cases that can produce duplicate labels or misleading status.
Failure scenarios
| Scenario | Consequence | Did it exist with delete-on-error? |
|---|---|---|
| Two workers read the same unfinished output before either finishes | Both pass the guards and can print the same labels. | Yes. Deleting the output did not stop an execution already holding it. |
| A worker saves a stale output after another worker records failure | Progress or completion saves can clear the recorded error and replace it with apparent success. | The stale-write race existed. Previously, a stale save could recreate the deleted output; retaining outputs makes overwriting their error another manifestation. |
| Printing partially succeeds, then the worker times out or terminates | The output can remain unfinished, allowing redelivery to print the batch again. | Yes. Django-Q’s TimeoutException inherits from SystemExit, bypassing except Exception. Abrupt termination also bypasses deletion. |
A plugin marks the output complete before after_printing() fails |
The frontend can observe success and stop polling before the failure becomes visible. | Yes. Previously the later failure deleted the output instead of recording an error. |
| An output is deleted while a worker still holds it | A subsequent model save can recreate the deleted row. | Yes, through progress/completion saves, but not through the delete-on-error handler. The related PR extends this risk to failure recording via mark_failure(). |
How overlapping execution can occur
InvenTree supports multiple worker processes: the default is four, restricted to one per cluster for SQLite or without a global cache. Queue leasing normally prevents duplicate delivery, but its timer starts at dequeue, before waiting in the local worker queue.
For example:
| Time | Event |
|---|---|
| 0 s | The job is leased and placed in a local worker queue. |
| 280 s | Worker A starts printing. |
| 300 s | The queue lease expires. |
| 305 s | Worker B receives the same job and starts executing it while A is still printing. |
Worker A’s 90-second execution timeout has not expired. If neither execution has recorded completion or failure, both can pass the unfinished-output guard.
This is a reachable risk, not a reproduced queue-level incident. It was tested by forced overlapping execution. One worker in one cluster prevents this overlap, but does not prevent sequential reprinting after an unrecorded timeout or crash.
Possible fixes
- Claim jobs atomically before printing. Persist execution state and permit only one delivery to transition an unstarted job into execution.
- Treat interrupted printing as uncertain. Record explicit worker timeouts and re-raise them. Preserve the claim across termination or failure to save the result, so recovery does not automatically repeat potentially completed physical printing.
- Restrict status updates. Use conditional, field-specific updates so stale progress writes cannot clear errors, overwrite terminal results, or recreate deleted outputs.
- Publish completion after the full lifecycle succeeds. Include
after_printing(). For plugins that delegate work, distinguish successful dispatch from confirmed printing. - Preserve the original exception. Failure to update a deleted output should not replace the printing error.
Additional state checks or database transactions alone cannot guarantee exactly-once physical printing. That requires printer-side idempotency or reconciliation of uncertain outcomes.
Validation
Four deterministic probes using actual Django task/model code, isolated SQLite, and mocked printing reproduced stale-error loss, timeout redelivery, premature completion, and deleted-row recreation. These were not live multi-worker or hardware tests.
Steps to Reproduce
Overloaded server:
| Time | Event |
|---|---|
| 0 s | The job is leased and placed in a local worker queue. |
| 280 s | Worker A finished previous work and starts printing. |
| 300 s | The queue lease expires. |
| 305 s | Worker B receives the same job and starts executing it while A is still printing. |
Timeout case:
- Submit a label job that prints some labels, then stalls or continues beyond the worker’s execution timeout—90 seconds by default.
- Django-Q raises TimeoutException, which inherits from SystemExit and bypasses the print handler’s except Exception.
- The output remains unfinished, with no recorded failure; delete-on-error never runs.
- When the queue retries the job, it passes the unfinished-output guard and starts printing again, potentially duplicating labels already printed.
Expected behaviour
Print job is started only once; partial print is not restarted automatically
Deployment Method
None
Version Information
Both master and #12872 are affected
Try to reproduce on the demo site
I did not try to reproduce
Is the bug reproducible on the demo site?
Not reproducible
Relevant log output
I have reviewed this report for correctness and filled out all sections.
- I have checked this report. I acknowledge that filing bugs via AI agents without review might lead to being blocked.
Source: inventree/InvenTree