#14910 [Feature]: heartbeats wake the coordinator to say nothing changed — make them state, not mail · orca

Problem or use case

Heartbeats are delivered as mail, so a message whose entire content is "nothing has changed" wakes the coordinator.

The wake path. A heartbeat is an ordinary orchestration send, so it goes through notifyMessageArrived. In mailbox-notification-coordinator.ts, waiters are filtered by type; when no waiter matches the arriving type, the code falls into consumers.length === 0 and schedules deliverForHandle, which is the mail-pointer push into the coordinator's terminal. There is no type gate on that branch.

So a heartbeat pushes a pointer in every case except while a check --wait with no type filter is blocking. Notably it still pushes when the coordinator is blocked in check --wait --types worker_done, because the heartbeat matches no waiter and therefore takes the push branch.

The arithmetic. The preamble instructs every worker to heartbeat every 5 minutes (HEARTBEAT_INTERVAL_MIN in preamble.ts). Six lanes is roughly 72 coordinator wakes per hour, each one a turn, to convey that nothing happened. Lanes scale that linearly, which is the opposite of what you want from a fleet.

This was not an oversight in the original design — it was out of scope. The heartbeat was introduced in ea8ea08116 (#1403), which squashed the "preamble rules + heartbeat schema" and "coordinator heartbeat + stale detector" work together. The stated purpose is narrow and still correct: "The coordinator uses this to distinguish 'still thinking' from 'hung / crashed.'" The cadence comment weighs detection latency against inbox spam — volume in a mailbox. What it does not consider is that arriving mail also pushes, because at the time the question was "how often should a worker report" and not "what should wake a coordinator".

The state half already exists and is already correct: dispatch_contexts.last_heartbeat_at, written by recordHeartbeat, read by getStaleDispatches. The message is the part that costs.

Proposed solution

Make the heartbeat state rather than mail. The worker still reports; the runtime still records last_heartbeat_at; the coordinator reads it when it is already talking to Orca instead of being interrupted to hear it.

The read surface already exists in review: #14907 / #14891 return a per-lane block on the orchestration commands a coordinator already runs, carrying delivery state and quiet time. Heartbeat freshness is the natural third field, and no new command or polling loop is needed.

Keep every other type pushing. worker_done, escalation, question and decision_gate are events — something happened and the coordinator should hear it now. Only the heartbeat, whose payload is by definition "no change", stops waking anyone. That preserves the property that matters: the coordinator wakes when something happened, not when nothing did.

Then add back the one wake that is worth it. There is a stall that nothing currently detects: work remains, every lane is idle, and the coordinator is idle too. Nobody is blocked on anything, nobody is working, and no message will ever arrive to break it — precisely because everyone is quiet. Today that Run sits still until a human notices.

That is the wake worth spending, and it is only computable centrally, because no single participant can see it: a lane knows only itself, and the coordinator is asleep. The runtime holds all three facts already — pending or ready tasks, per-lane idle state, and the coordinator's own state.

This has a pleasing symmetry with the first half: remove the wakes that mean nothing, add the one that means everything.

Alternatives or additional context

Coalesced heartbeat at a lower cadence. Instead of removing the message, have the runtime aggregate: one message per Run at a longer interval summarising every lane, rather than one per worker per 5 minutes. This keeps heartbeat-as-mail — so anything consuming those messages keeps working — while cutting wake count by roughly the lane count. Strictly less good than state (it still wakes for "no change"), but a much smaller change and a reasonable landing spot if removing the message is judged too invasive.

Type-gate the push only. Smallest possible version: leave everything as it is and skip the pointer push for heartbeat in the consumers.length === 0 branch. Heartbeats keep arriving as mail and are read on the next check, but stop interrupting. This is close to a one-place change and would capture most of the benefit; the downside is that the mailbox still fills, which is what the original cadence comment was trying to avoid.

Risks worth checking before implementing any of these:

check --wait waits for messages. Heartbeats are not what it waits for, so removing them should not strand a waiter — but that deserves a test rather than an argument.
Heartbeats cross to federated peers as messages today. Turning them into state is a behaviour change that reaches older clients even with no wire change, which is the case docs/reference/remote-wire-compatibility.md warns about.
#14897 (nudging a silent dispatch) needs to know a worker is alive. If heartbeats become state, that supervisor reads state — which is actually better than depending on a message having arrived, but the two designs should be settled together rather than in sequence.

Related: #14907 and #14891 (the read surface this would use), #14897 (acting on the signal), #14832 (the mail-pointer push path, and its interaction with a user typing), #9228 (event-driven coordinator lifecycle — this issue is effectively an argument about which events deserve to be one).

Update (2026-08-26)

The read-surface PRs referenced in the proposal (#14891/#14892) closed unmerged on 2026-08-22, so the paragraphs that lean on them no longer describe anything in review. The problem statement is unchanged on current main: the mailbox push path has no type gate — when no waiter matches, mailbox-notification-coordinator.ts schedules delivery regardless of message type, so a heartbeat still wakes a coordinator that was explicitly waiting for worker_done. The state half also still exists (last_heartbeat_at on dispatch_contexts), so the proposal's substance — stop pushing heartbeats, read them as state — is intact; only the "read surface already in review" framing is stale.