Whole-backend wedge: event loop freezes, all chats stall simultaneously, user messages silently swallowed, no trace left

Whole-backend wedge — event loop freezes, all chats stall simultaneously, user messages silently swallowed, no trace left

Summary

In a dockerized Agent Zero instance, the entire backend can wedge without crashing: every chat stops progressing at the same time, the UI keeps showing spinner state, the browser push channel silently dies (work completes but responses never render), and — worst — user messages typed during the wedge are swallowed with zero trace: they are never persisted to any chat file and never processed. The process remains alive (no OOM, no exception, nothing in exception capture), so only a manual restart recovers it. Post-mortem diagnosis is nearly impossible because the wedge leaves no logs.

Observed 2026-09-08 ~14:17 CEST: all active chats' logs stopped within minutes of each other (14:11–14:17). Agent work that finished at 14:11–14:13 (tool calls, sent emails, generated responses logged in chat JSON) never reached the user's UI. Messages sent after ~14:17 exist nowhere on disk. Manual framework restart at 15:03 was the only recovery.

Measured evidence (second incident, instrumented)

After the first incident, user-space instrumentation was installed (turn trace, periodic asyncio task-state snapshots, loop-stall stack dumps). The wedge recurred the same day and was captured live (2026-09-08 17:20 CEST):

• Turn trace: turn_start logged for a chat agent at 17:20:10, no matching turn_end ever — the wedge onset marker
• Task snapshots: pending asyncio tasks doubled 6 → 12 between 17:18:20 and 17:19:24 — task pileup immediately before the freeze
• Loop-stall monitor: event loop froze at 17:20:09; stall entries with lag climbing 16s → 616s every minute until process replacement — the main event loop was hard-frozen for the duration, not merely slow
• Ruled out again: no OOM (memory.events clean), no captured exceptions, container never restarted

This upgrades the incident from "unknown cause" to a measured signature: task pileup → hard event-loop freeze → all chats + WS push dead, reproducible as a pattern even though the trigger is not yet identified.

Also found and repaired on the same day: 8 empty (0-byte) git object files in a Time Travel workspace repository (/a0/usr/.time_travel/workspaces/<id>/repo.git/objects/) — corrupt objects that cause snapshot/repair-tooling failures and may participate in the pileup (e.g., a repair sweep iterating corrupt repos on the loop). Removed; repository verified functional.

Reproduction

Not yet reliably reproducible — intermittent. Conditions present at the time:

1. Multiple long-running chats active simultaneously (several at ~100k+ token contexts)
2. Heavy concurrent activity (terminal tool sessions, vision loads, multiple chat loops running)
3. UI open in browser; at some point responses stop rendering while chats keep "running"
4. Eventually: ALL chats stop progressing; typed messages show indefinite "waiting" spinner
5. docker container still healthy: process alive, memory peak 10.1/20 GiB, memory.events clean (no oom), no new entries in exception capture

Root cause

Unknown — this is the core of the report. Ruled out so far:

• OOM / cgroup kill: memory.events shows zero oom/oom_kill events; peak 10.1 of 20 GiB
• Crash: process was still running until manually restarted
• Per-chat compression stalls: a separate, already-reported failure mode (#1881) — that bug kills individual chats silently, but does not explain a simultaneous cross-chat freeze, a dead UI push channel, or swallowed input

Suspected directions (unverified):

• Event-loop blockage: a hung tool call / thread pool exhaustion blocking the main asyncio loop would freeze all chat loops simultaneously and stop WS push — matches symptoms
• WebSocket push channel dying silently: responses completed and were logged server-side (14:11–14:13) while the browser showed nothing — the push path had already failed before the full wedge, or the wedge began as a push failure and escalated

The two concrete defects worth fixing regardless of root cause

1. Data loss: user messages are not durable. A message typed while the backend is wedged is lost forever — it is not written to the chat file before processing is attempted. Persisting the user message (and marking it pending) before enqueueing would make every wedge recoverable with zero loss.
2. No observability for process-level wedges. There is no heartbeat log, no event-loop liveness marker, and the exception-capture hook records nothing for this failure class (it only logs raised exceptions, not hangs). A periodic heartbeat (e.g., timestamped touch per active chat + loop-liveness entry) would turn this undiagnosable wedge into a diagnosable one.

Before / after

Scenario	Current	With proposed changes
Backend wedges while user types	Message swallowed, no trace, spinner forever	Message persisted as pending; delivered or replayed after recovery
Post-incident diagnosis	Nothing logged; guesswork	Heartbeat timeline shows exact wedge onset + last live activity
Responses generated during push failure	Never displayed, user resends	Client reconnect/resync replays missed responses

Environment

• Agent Zero, dockerized (single container), current release as of Sep 2026
• Multiple concurrent chats with ~100k+ token contexts, vision embedding, parallel tool sessions
• Incident window: 2026-09-08 14:11–14:17 CEST; recovered only by manual restart 15:03

Related

• #1881 (history compression stalls on utility-model failure) — per-chat silent death of final responses; a different failure mode, but both were active in this instance and their symptoms are easily confused. Distinguishing signal: #1881 kills one chat; this bug stops all chats at once and swallows input.