Deploy the new block exporter: rollout risks, tests and instrumentation

(AI-generated)

Context

#6639 adds in-worker block export: a per-process queue task on every validator that pushes each executed block to the rest of the committee, relayed through the proxy so shards need no route to the internet. This issue tracks what we need to know, measure, and add before turning it on in production.

Export is gated behind --export-blocks-to-committee and is off by default, and the flag only changes the sending validator. That makes a single-validator enable a genuine A/B, and it bounds the blast radius of everything below.

Export cannot corrupt a peer. Certificates are validated on receipt, so a bug here is a resource or availability problem on the sending validator or on shared storage — not a safety problem. Every risk below is in that class, which is also the class that has caused our previous validator incidents.

Code references are against 9c407d1e (PR head at the time of writing) and will drift.

Risks

1. A down peer pins per-chain state for every active chain — OOM risk

By design, and documented in the module header: "a destination that is down keeps every chain that produced a block while it was gone, because that set is the work-list its catch-up needs." There is no cap on chains and no shedding policy.

Rough sizing: per tracked chain we hold a ChainRecord (~96 B including the map entry) plus one ChainDest per destination (~64 B in the dests vec), and each destination's lagging: BTreeSet<ChainId> holds a 32-byte id per lagging chain. At 1M chains and a 20-member committee that is on the order of 2–3 GB; at 100 members it is well past what the pod has.

block_export_tracked_chains counts chains, not chain–destination pairs, so the metric grows sublinearly with the memory it is meant to expose.

2. Catch-up storage reads have no global budget — Scylla saturation risk

drain_ready spawns up to window (default 8) sends per destination, each reading up to max_catch_up_blocks (default 200) certificates from storage, across every destination, retried every idle_catch_up_interval (default 200 ms). Nothing bounds aggregate read concurrency across destinations.

Worse, the AIMD control only responds to transport failures. ViewError — our own storage failing — is classified chain-scoped (is_chain_scoped, linera-core/src/chain_worker/export.rs), so storage pressure backs off individual pairs at 1 s while every other pair keeps reading. The control loop cannot see the bottleneck it is creating.

This is the most likely way the PR takes a validator down, and it is the same shape as the 2026-05-15 reader-concurrency cascade.

3. Progress-mutex stalls on the block-execution path

export_block takes progress.lock() on every executed block. The convergence sweep takes the same std::sync::Mutex and may call shrink_to_fit on the heights map. MAX_FORGET_PER_SWEEP bounds the removals but not the reallocation — shrinking a large table under that lock stalls every chain worker in the process.

4. The exported_heights write throttle is defeated by worker eviction

last_exported_heights_fold lives on ChainWorkerState and is initialized to None. Chain workers are TTL-evicted (chain_worker/handle.rs). So for any chain whose inter-block gap exceeds the worker TTL, every block gets a fresh worker, the throttle does not apply, and the whole exported_heights RegisterView is re-serialized and written.

The throttle therefore helps busy chains — which were already the cheaper case — and does nothing for the many-idle-chains workload we actually run.

5. Catch-up resurrects idle chain workers on the receiving side

The send side is carefully kept off chain workers. The receive side cannot be: handle_lite_certificate / handle_confirmed_certificate go through the worker and reset its TTL. Client broadcast already does this for live blocks, so the increment there is small — but catch-up pushes blocks for chains that are otherwise idle, resurrecting exactly the workers the TTL exists to evict.

6. Enabled-but-dead is invisible

If scan_committees cannot read the network description or load any committee, destinations stays empty and export silently does nothing, with a debug! line as the only trace. There is no destinations gauge, so on a dashboard "healthy" and "doing nothing" are identical.

7. One poison chain degrades a destination for all chains

Any error outside the is_chain_scoped list halves that destination's AIMD window and backs the destination off globally. A single chain reliably producing such an error oscillates a healthy peer's window for every other chain.

8. Deploy ordering: proxies must precede shards

New shards against an old proxy get an unimplemented relay RPC, which classifies as destination-scoped, so they back off. Self-limiting, and it will not stall block execution, but export is dead and the logs are noisy until the proxies land.

Recommended tests

Highest value first. Items marked fails today are expected to fail against the current implementation — that is the point: they force a decision rather than assert existing behaviour.

• Memory ceiling under a dead peer. Drive N chains × M blocks with one destination permanently failing; assert a bound on tracked pairs (chains × destinations, plus the summed lagging sizes), not just chain count. Turns the documented "bounded by active chains" into an enforced number and gives a shedding policy somewhere to land. (risk 1)
• Storage read budget. Wrap Storage with a call counter; simulate a validator joining with a K-block backlog across C chains; assert total read_certificates_by_heights / read_blobs calls scale with max_catch_up_blocks × window × destinations and not with backlog × chains. (risk 2)
• Backpressure under storage pressure. Inject ViewError or latency into storage; assert aggregate in-flight sends fall. Fails today — forces the decision on whether storage errors should shrink a global budget rather than a per-pair backoff. (risk 2)
• Tick fires under sustained load. Saturate the block stream with a destination that never completes; assert tick-only work still happens (a backoff expires and retries, or a dropped tip is repaired) within a few intervals. The timer fix in this PR currently has no regression test, and its absence previously disabled drop repair, backoff expiry, the committee scan and the convergence sweep — all silently. (risk: regression of a fixed bug)
• Progress-mutex hold time. Instrument the lock; sweep MAX_FORGET_PER_SWEEP chains out of a large map; assert maximum hold stays under ~1 ms. (risk 3)
• Register writes across worker eviction. Chain worker with a TTL below the fold interval, one block per TTL period; count exported_heights register writes. Either move the last-fold timestamp into the chain state, or assert the current behaviour deliberately. (risk 4)
• Poison-chain isolation. One chain always returning a non-chain-scoped error; assert other chains' throughput to that destination is unaffected. (risk 7)
• Version skew. New shard against an old proxy; assert export degrades to backoff and that block-execution latency is unchanged. (risk 8)

Recommended instrumentation (before enabling anywhere)

• block_export_destinations gauge — the only way to distinguish "enabled and working" from "enabled and resolving nothing". (risk 6)
• A gauge for total lagging pairs, not chains — this is what tracks the memory. (risk 1)
• Alert on the block_export_dropped_blocks rate.
• Alert on block_export_chain_scoped_backoffs staying flat and non-zero. The module doc already identifies this as the "a destination is stuck on some chain and nothing is repairing it" signal, but nothing alerts on it.

Recommended rollout

1. Deploy proxies before shards (risk 8).
2. Deploy with the flag off everywhere; confirm no behaviour change.
3. Enable on one validator. Measure against an unchanged peer: p99 block-execution latency delta, Scylla read IOPS delta, resident chain-worker count, RSS. Hold for at least a day.
4. Kill-a-peer soak: with export on, take one validator down for hours at production block rate and watch block_export_tracked_chains and RSS on the survivors. This measures risk 1 directly, and the number it produces decides whether a cap is required before going further.
5. Restart catch-up storm: restart a node holding a large backlog; measure peers' storage read rate and the restarted node's convergence time. This is risk 2's worst case.
6. Enable on a minority of validators, then fleet-wide.

Steps 4 and 5 are the two that produce numbers we do not currently have and cannot get from a unit test.