Half-finalized turn: producer death between the two interrupt writes leaves an unterminated message and a client poll loop

Summary

markGenerationInterrupted finalizes a dead run with two sequential, non-atomic writes. If the process dies between them, the generation is marked interrupted but the message is never flagged — and nothing can ever repair it, because every future reaper sweep filters on status: "running" and the generation is no longer running.

The user-visible result is an assistant message that never terminates. The client treats it as still streaming and re-subscribes to /conversation/:id/stream roughly twice a second, indefinitely, until the page is reloaded.

The window

src/lib/server/generation/finalize.ts

const claim = await collections.generations.updateOne(
  { generationId, status: "running" },
  { $set: { status: "interrupted", endedAt: now, updatedAt: now } }
);
if (claim.matchedCount === 0) return;

// ← process death here leaves the message unflagged, permanently
await collections.conversations.updateOne(
  { _id: run.conversationId, "messages.id": run.messageId, "messages.interrupted": { $ne: true } },
  { $set: { "messages.$.interrupted": true, ... } }
);

The comment above it already names the invariant — "The message flag — not just the generation status — is what every existing reader treats as terminal, so both must move together" — but ordering the writes doesn't make them atomic, and the claim is what closes the door on retry.

Why nothing recovers it

reaper.ts sweeps on:

collections.generations.find({ status: "running", lastHeartbeatAt: { $lt: threshold } })

Once the claim has flipped the generation to interrupted, that row matches no future sweep. The half-finalized turn is invisible to the only thing that would fix it.

finalizeActiveRunsOnExit() runs from onExit, which is where the window is widest: shutdown is already racing, and a Promise.all of two-step finalizations is likely to be cut mid-step.

Observed

Dev server stopped (SIGTERM) during an in-flight generation:

Client behaviour: /api/v2/conversations/:id → /api/v2/conversations → /conversation/:id/stream, repeating every ~450ms indefinitely. Server-side each /stream returns 200 immediately because no producer exists.

Repairing turnStates and appending a FinalAnswer in the database did not stop an already-open tab — the loop is client-side state. A page reload cleared it.

Note that isTurnAlive handles this case correctly and deliberately ("a state doc stuck in 'running' with no running producer is a crashed run, which must read dead so subscribers get closure"), so this is not a liveness-read bug. The damage is the unterminated message.

Impact

Not specific to a dev restart. Any producer death between the two writes does it: pod eviction, OOM kill, deploy, crash. The turn-continuity design covers a parked call (the sweeper resumes it); this is a hard death with no parked call and a partially-applied finalization.

Cost is a wedged conversation for that user plus a ~2 req/s poll loop per affected open tab.

Suggested directions

1. Make the claim recoverable. Rather than flipping straight to interrupted, claim into an intermediate state (e.g. finalizing) with a lease, and let the reaper sweep { status: "finalizing", updatedAt: stale } as well. The claim keeps its single-winner property and stops being a one-shot.
2. Or reconcile from the other side. Give the reaper a second query for terminal generations whose message is not flagged, bounded by a recency window so it isn't a full scan.
3. Also write turnStates in this path. Nothing here moves it off running. isTurnAlive tolerates that today, but the record stays wrong for any other reader.
4. Consider whether a terminated message should always carry a FinalAnswer. The client's resubscribe decision keys off the message looking unfinished, so the interrupted flag alone may not be sufficient.

Worth a regression test that kills the process between the two writes and asserts a later sweep still terminates the message.