RFC: Delivery receipts for outbound messages

Problem

Nothing in ZeroClaw can tell whether a message an agent sent to a human actually arrived.

An outbound message carries no identifier. SendMessage (crates/zeroclaw-api/src/channel.rs) has content, recipient, subject, thread, attachment and voice fields, and no id of any kind. Channel::send returns anyhow::Result<()>. Nothing records what was sent, and there is no outbound dedup ledger anywhere in the tree: greps for outbox, dead_letter, redelivery and already_sent return only inbound broker redelivery handling in the AMQP channel and a Matrix parameter bundle that happens to share the name. The nearest precedent proves the point rather than contradicting it — PENDING_DELIVERY_ACKS in model_picker_delivery.rs is a process-global Mutex<HashMap> of one-shot senders, scoped to model-picker selections and ephemeral by construction.

That single absence is upstream of a family of failures that look unrelated:

• A dropped send is indistinguishable from a delivered one. The pacing wrapper discards the newest message on queue overflow and returns Ok(()). #10600 is open and proposes returning an error there, which makes the in-process case honest — but only that case. Once a message leaves for the transport, nothing tracks it again.
• Duplicate delivery cannot be detected or suppressed. escalate_to_human sends to the active channel and then fans the same text out to [escalation] alert_channels. #10600 proposes excluding the origin, but any other overlap between configured routes still double-delivers, and there is no key to deduplicate on after the fact.
• A failed send leaves little an operator can find. Cron is the exception and shows what good looks like: the delivery error is appended to the stored run output and is readable through cron_runs and GET /api/cron/{id}/runs. Everywhere else a failure is a WARN in state/runtime-trace.jsonl, whose default rolling mode keeps default_log_persistence_max_entries = 200 entries, so on a busy agent the evidence is gone within minutes.
• There is no way to ask "what did you try to tell me that I missed?" The question has no data behind it.

The operational cost is not theoretical. Operator replies and agent notifications have repeatedly gone missing on this project's own fleet while both the sender and the tooling believed they had been delivered, because "never arrived" and "arrived and was answered" are byte-identical from outside.

Proposal

Give every outbound message an identity, and give every send an outcome worth recording.

1. A stable id on SendMessage, minted at construction, carried unchanged through wrappers such as PacedChannel, and never regenerated on retry.
2. A richer send outcome. Channel::send returns a DeliveryOutcome instead of (), carrying at minimum whether the transport accepted the message and, where the transport supplies one, its own reference: Discord message id, Telegram message_id, Slack ts, Matrix event id. A default implementation keeps existing channels compiling and simply reports acceptance.
3. A delivery ledger recording (message_id, channel_alias, recipient_digest, kind, sent_at, outcome, provider_ref), durable alongside the other state stores, with a bounded retention policy.
4. Dedup at the send boundary, keyed on the message id, so a message already recorded as delivered is not sent twice by a second route.
5. A query surface so an operator, and an agent, can ask what was attempted and what failed.

Design sketch

Affected surfaces: zeroclaw-api (SendMessage, Channel), every channel adapter, PacedChannel, escalate_to_human, ask_user, cron and heartbeat delivery, the gateway.

Proposed phases:

• Phase 1, additive and non-breaking. Add the id field and DeliveryOutcome with a default trait implementation. Introduce the ledger behind a config flag, written at the wrapper layer rather than by each adapter, so one code path covers every channel. No behavior changes.
• Phase 2, per-transport truth. Channels that can report a provider reference do so, upgrading "the API accepted it" to "the platform assigned it this id". This is what makes a later read or reaction receipt possible at all.
• Phase 3, dedup and query. Enable id-keyed suppression at the send boundary and expose the ledger through the existing admin surface, so "what did you try to tell me" becomes answerable.

Recipient values are personal data. The ledger stores a digest using the same redaction shape PacedChannel::redact_recipient already applies to logs, never a raw JID, phone number, or user id.

Alternatives considered

• Do nothing. The status quo: every duplicate and every silent drop stays invisible, and each is fixed one call site at a time, as #10600 does for two of them.
• Log-only. Keep the WARN lines and raise the trace retention default. Cheaper, but a rolling text log is not queryable and not dedupable, and the retention default is exactly what makes it useless today.
• Per-channel ids only. Let each adapter track its own sends. This is roughly where the codebase already is, and it yields several bespoke half-solutions with no common contract.
• A full store-and-forward outbox with retry. Strictly more capable and a much larger change. A receipt is its prerequisite, so this proposal deliberately stops short of it.

Non-goals

• Human read receipts. Knowing a person opened a message is a separate, transport-dependent problem.
• A retry queue or store-and-forward outbox. Worth doing, and it should build on this.
• Changing which sends happen, or any pacing or routing policy.
• Guaranteed delivery. This makes delivery observable, not reliable.

Risks and mitigations

• Trait churn across every channel. Mitigated by a default DeliveryOutcome implementation so adapters opt in as they gain the ability to report a real reference.
• Ledger growth. Bounded retention with a documented default, following the existing cron run-history cap rather than inventing a new policy.
• Personal data at rest. Recipients are digested, never stored raw, consistent with the privacy contract and the existing log redaction.
• False confidence. A transport acknowledgement is not a human reading the message. Field names and docs must say "accepted by the transport", never "delivered to the operator".
• Rollback. Phases 1 and 2 are additive and revert cleanly. Phase 3 sits behind the flag that gates dedup.

Breaking change?

No

Decision and revisit surface

This issue, then a proposal PR for phase 1. Revisit after phase 1 ships, since the value of dedup and of the query surface should be judged against real ledger data rather than argued in advance.

Data hygiene checks

• I removed personal/sensitive data from examples, payloads, and logs.