prompt_worker thread dies on any exception escaping `PromptExecutor.execute` — server keeps serving HTTP but never executes prompts again ("zombie" server)

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

Summary

prompt_worker() in main.py calls e.execute(...) without any exception guard. If an exception escapes PromptExecutor.execute_async from framework-level code (anything outside the per-node try/except that turns node errors into ExecutionResult.FAILURE), the exception propagates out of e.execute() and kills the prompt_worker thread.

After that the server is in a zombie state:

• the HTTP server keeps responding normally (/, /queue, /history, /prompt all return 200),
• every newly queued prompt is accepted but never executed,
• there is no error in the UI — jobs just hang forever, and the only recovery is restarting ComfyUI.

This matches the traceback pattern of several "ComfyUI stops executing prompts" reports (e.g. AMD/HIP illegal-memory-access crash logs where the final line is Exception in thread Thread-N (prompt_worker) followed by a silent server).

Real-world trigger (ComfyUI 0.35.1, Windows, 100% reproducible)

We hit a deterministic instance of this with the RAM-pressure cache:

1. A workflow contains a node whose output includes a lazily-materialized Mapping (VideoHelperSuite's LazyAudioMap, which shells out to ffmpeg on first access). The lazy object was never consumed during execution, but it gets stored in the output cache.
2. After the prompt finished, our automation deleted the input file that the lazy object references.
3. When the next prompt starts, execute_async calls ram_release_callback(...) (RAM-pressure cache eviction). The eviction scan iterates cached outputs, materializes the lazy Mapping → ffmpeg fails (No such file or directory) → exception.
4. execute_async's try block has no except clause (only finally), so the exception escapes e.execute() and kills prompt_worker:

Exception in thread Thread-9 (prompt_worker):
Traceback (most recent call last):
  File "main.py", line 394, in prompt_worker
    e.execute(item[2], prompt_id, extra_data, item[4])
  File "execution.py", line 728, in execute
    asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))
  File "execution.py", line 800, in execute_async
    ram_release_callback(ram_inactive_headroom)
  File "comfy_execution/caching.py", line 591, in ram_release
    scan_list_for_ram_usage(cache_entry.outputs)
  File "custom_nodes/ComfyUI-VideoHelperSuite/videohelpersuite/utils.py", line 262, in __iter__
    self._dict = get_audio(self.file, self.start_time, self.duration)
  ...
Exception: VHS failed to extract audio from .../input/lip_shot_004.mp4: ...
Error opening input files: No such file or directory

From this point on every prompt submitted to the server is silently never executed. Deleting the database / cache does not help; only a restart does.

The same class of failure can be triggered by any other framework-level exception escaping execute_async (cache providers, asset enrichment, progress handlers, ...), so the fix is not specific to VHS.

Proposed fixes (two small patches, both against current master)

1. main.py — the worker thread must never die

Wrap e.execute(...) in try/except, log the exception, and record the prompt as failed so q.task_done(...) still completes the queue item with an error status. history_result may not exist if execute_async failed early, hence the getattr fallback.

--- a/main.py
+++ b/main.py
@@ -359,7 +359,19 @@
                 extra_data[k] = sensitive[k]
 
             asset_manager.pause_background_scan()
-            e.execute(item[2], prompt_id, extra_data, item[4])
+            try:
+                e.execute(item[2], prompt_id, extra_data, item[4])
+            except Exception:
+                # Any exception escaping the executor (e.g. from cache
+                # maintenance or other framework-level code outside the
+                # per-node error handling) must not kill the prompt_worker
+                # thread: the HTTP server would keep running but no prompt
+                # would ever execute again ("zombie" server).
+                logging.exception(
+                    "Unhandled exception escaped PromptExecutor.execute; "
+                    "recording the prompt as failed and keeping prompt_worker alive")
+                e.success = False
+                e.history_result = getattr(e, "history_result", None) or {}
 
             need_gc = True

2. comfy_execution/caching.py — ram_release must tolerate scan failures

A cache entry whose outputs fail to scan (e.g. lazy materialization error) should not crash the executor. Treat it as having unknown-but-nonzero size so it still participates in eviction.

--- a/comfy_execution/caching.py
+++ b/comfy_execution/caching.py
@@ -1,6 +1,7 @@
 import asyncio
 import bisect
 import itertools
+import logging
 import time
 import torch
 from typing import Sequence, Mapping, Dict
@@ -588,7 +589,23 @@
                         oom_ram_usage = 1e30
                     elif hasattr(output, "_comfy_cache_tensors"):
                         scan_list_for_ram_usage(output._comfy_cache_tensors())
-            scan_list_for_ram_usage(cache_entry.outputs)
+            try:
+                scan_list_for_ram_usage(cache_entry.outputs)
+            except Exception:
+                # Cached outputs may contain lazily-materialized objects (e.g.
+                # a custom node's lazy Mapping that shells out to ffmpeg on
+                # first access). If materialization fails (e.g. the referenced
+                # file was deleted after the workflow ran), the exception must
+                # not escape ram_release(): it is called from execute_async
+                # outside the per-node error handling, so it would kill the
+                # prompt_worker thread and leave the server "zombie" (HTTP
+                # alive, queue dead). Treat the entry as having unknown size so
+                # it still participates in eviction.
+                logging.exception(
+                    "ram_release: failed to scan cached outputs; "
+                    "treating cache entry as evictable")
+                ram_usage = max(ram_usage, min_entry_size)
+                oom_ram_usage = max(oom_ram_usage, ram_usage)
 
             if ram_usage < min_entry_size:
                 continue

Verification

Both patches applied locally on 0.35.1: the exact trigger above (lipsync automation deleting its uploaded input file between prompts) previously turned the server into a zombie every time; with the patches the scan failure is logged, the entry is treated as evictable, and the server keeps executing prompts.

Environment

• ComfyUI 0.35.1 (vulnerable code paths unchanged on master as of today: main.py prompt_worker bare e.execute(...), caching.py ram_release unguarded scan_list_for_ram_usage(cache_entry.outputs), execution.py execute_async try with only finally)
• Windows 11, Python 3.12, RTX 5090D

Actual Behavior

The prompt_worker thread dies, leaving the server in a "zombie" state:

• The HTTP server keeps responding normally — /, /queue, /history, /prompt all return 200.
• Every newly queued prompt is accepted but never executed. Jobs hang forever in the queue.
• No error is shown in the UI. The only visible sign is the terminal traceback (Exception in thread Thread-N (prompt_worker), see Debug Logs).
• Deleting the database / cache does not help. Only restarting ComfyUI recovers.

In our production automation this made ComfyUI look "alive" (health checks passed) while nothing ever ran again — the failure silently blocked all rendering until we correlated restarts with the terminal traceback.

Steps to Reproduce

1. Start ComfyUI 0.35.1 (Windows) with the RAM-pressure cache enabled (--cache ram).
2. Run a workflow whose output contains a lazily-materialized Mapping — e.g. any VideoHelperSuite audio node (LazyAudioMap shells out to ffmpeg on first access). The lazy object is stored in the output cache even if never consumed.
3. After the prompt finishes, delete the input file that the lazy object references (in our automation: the uploaded input/lip_shot_004.mp4 is removed between prompts).
4. Queue any new prompt. On startup of that prompt, execute_async calls ram_release_callback(...); the eviction scan iterates cached outputs, materializes the lazy Mapping, ffmpeg fails (No such file or directory), and the exception escapes e.execute() because prompt_worker() has no guard — killing the thread.

Expected: the failed cache scan is logged, the prompt runs (or is recorded as failed), and subsequent prompts keep executing.

Actual: the HTTP server keeps responding normally (/, /queue, /history, /prompt all return 200), but every newly queued prompt is accepted and never executed. No error appears in the UI; jobs hang forever. Only a restart of ComfyUI recovers. Deleting the database / cache does not help.

Note: no minimal workflow JSON is attached because the trigger is the automation step (deleting a referenced input file between prompts), not the workflow graph itself — any workflow containing a VHS audio node reproduces it. The failure class is also not VHS-specific: any framework-level exception escaping execute_async (cache providers, asset enrichment, progress handlers, ...) kills prompt_worker the same way.

Debug Logs

Exception in thread Thread-9 (prompt_worker):
Traceback (most recent call last):
  File "main.py", line 394, in prompt_worker
    e.execute(item[2], prompt_id, extra_data, item[4])
  File "execution.py", line 728, in execute
    asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))
  File "execution.py", line 800, in execute_async
    ram_release_callback(ram_inactive_headroom)
  File "comfy_execution/caching.py", line 591, in ram_release
    scan_list_for_ram_usage(cache_entry.outputs)
  File "custom_nodes/ComfyUI-VideoHelperSuite/videohelpersuite/utils.py", line 262, in __iter__
    self._dict = get_audio(self.file, self.start_time, self.duration)
  ...
Exception: VHS failed to extract audio from .../input/lip_shot_004.mp4: ...
Error opening input files: No such file or directory

(After this traceback the HTTP server keeps responding normally, but every
newly queued prompt is accepted and never executed. Only a restart recovers.)

Other

No response