An idle sending chain never retries a failed cross-chain delivery

A cross-chain message that a validator fails to deliver internally can stay undelivered indefinitely, because the only thing that re-attempts delivery is another write request for the sending chain. If that chain goes idle, nothing retries.

How a delivery is lost

Two independent drop points, both silent apart from a log line.

Queue overflow. Outgoing cross-chain requests are handed to a bounded channel with try_send, and a full queue drops the request:

if let Err(error) = routing_sender.clone().try_send((request, shard_id)) {
    error!(%error, "dropping cross-chain request");
}

(linera-rpc/src/grpc/server.rs:404, and again at :610.) The bound is --cross-chain-queue-size, default 1000.

Retry exhaustion. forward_cross_chain_queries retries a failed send with backoff, then abandons it:

Action::Retry => self.retries >= max_retries,

(linera-rpc/src/cross_chain_message_queue.rs:292; when is_finished holds, the job state is remove()d at :165.) Defaults are --cross-chain-max-retries 10, --cross-chain-retry-delay-ms 2000, backing off to --cross-chain-max-backoff-ms 30000 — so roughly ten attempts over a few minutes, after which the in-memory attempt is gone.

Why the outbox does not save it

The persistent outbox still holds the entry, and ChainWorkerState::create_network_actions re-derives all pending cross-chain requests from it rather than only new ones — so any call to it re-emits the lost delivery.

But every call site is a request handler for the sending chain:

There is no timer, no background sweep, and no periodic reconciliation. Notably handle_chain_info_query is not among them, so a plain read does not trigger redelivery.

So the message is re-emitted only if the sending chain produces another block, or someone re-submits a certificate for it — and the already-processed path at :978 means re-submitting a known certificate is enough. If the sending chain is idle and nobody touches it, the bundle sits in the outbox.

Why it is not fatal today

Recovery is client-driven and lazy. A client proposing a block that consumes the bundle gets MissingCrossChainUpdate from the lagging validator and pushes the sending chain's certificates to it (send_block_proposal's retry arm in linera-core/src/updater.rs), and CrossChainMessageDelivery::Blocking lets a client wait for delivery explicitly. So a recipient that wants the message can obtain it.

What is missing is delivery for a recipient that is not actively asking — and, more importantly, any validator-side guarantee at all. The failure is also asymmetric across a committee: a validator that dropped the request has an inbox other validators do not, until someone repairs it.

Why it matters for the specification

linera_core::proof::storage::StorageConvergesAtEqualHeights (#6796) is conditional on cross-chain delivery having quiesced, and this is exactly the condition it cannot discharge. Nothing in the specification states that an outbox is ever drained, and as a validator-side property it is currently false rather than merely unproved.

Possible directions

• A periodic reconciliation sweep per chain worker, re-deriving pending requests for chains with a non-empty outbox — nonempty_outboxes already indexes exactly those.
• Retry with unbounded attempts and backoff, dropping only on shutdown, so exhaustion stops being a silent terminal state.
• Surface the drops: neither path increments a metric that distinguishes "delivery abandoned" from ordinary errors, so a validator in this state looks healthy.

The last is worth doing regardless of which of the first two is chosen.