Gateway reply delivery is not durable end-to-end: inbound is marked complete before the reply is sent, and effectively-once delivery is opt-in (Slack-only)

Summary

The gateway/bot layer guarantees "the agent ran", not "the user received the reply". On a clean turn, the durable inbound journal entry is marked complete before the adapter delivers the reply, and the persist-before-send / effectively-once machinery that would close the gap is opt-in and adopted by only one shipped adapter (Slack). A crash (or process kill / OOM / redeploy) in the window between "agent finished" and "reply delivered" therefore loses the user's reply permanently, with no replay — the worst failure mode for a chat gateway, because the user is left waiting for an answer that was computed, paid for, and then dropped.

This is the single highest-value robustness gap for making the gateway world-class: a production messaging gateway's core promise is that an accepted message is eventually answered exactly once (or at-least-once with an honest duplicate marker), across restarts.

Current behaviour

1. Inbound journal is completed before the reply is delivered. In src/praisonai-bot/praisonai_bot/bots/_session.py, BotSessionManager.chat() marks the inbound journal entry complete on the clean-exit path and then returns the reply string; the adapter delivers it afterwards:

# src/praisonai-bot/praisonai_bot/bots/_session.py:1644-1650
else:
    # Clean exit - mark journal complete before releasing claim
    if journal_key is not None and self._ingress_journal is not None:
        try:
            self._ingress_journal.complete(journal_key)   # <-- reply NOT yet sent
            self._last_journal_key = None

The adapter's finalize/send happens after chat() returns (e.g. Telegram at telegram.py ~:768). Because the journal row is already complete, the boot-time replay (InboundJournal.replay(), _ingress.py:529) will not re-drive it. A deferred-completion seam exists — complete_last_journal_entry() (_session.py:2009) — but no adapter calls it.

2. Persist-before-send / effectively-once is opt-in and largely unused. The durable primitives already exist and are good:

• OutboundQueue (_outbox.py:113) with a pending → sending → recovered → sent/failed state machine, a crash reconciler, and an honest RECOVERED_PREFIX "possible duplicate after restart" label (_delivery.py:36).
• DurableDelivery (_delivery.py:456) and DurableAdapterMixin (_durable_adapter.py:19).

But shipped reply paths do not use them. Built-in adapters use live-send + retry + DLQ-park (OutboundResilienceMixin.deliver_outbound(), _outbound_resilience.py:147), which is best-effort at-least-once at most. Crucially, effectively-once reconciliation via was_delivered() is implemented by Slack only:

$ grep -rln 'def was_delivered' src/praisonai-bot/praisonai_bot/bots/*.py
slack.py

Every other adapter falls back to at-least-once with no crash-time reconciliation.

3. The flagship adapter drifts from the safe path. TelegramBot inlines its own retry/DLQ instead of the shared mixin, and its outbound DLQ is off unless dlq_path is configured (telegram.py:232-236), so permanently-failed Telegram replies are silently dropped where the mixin default would park them. LocalBot has neither.

Desired behaviour

An accepted inbound message is not considered settled until its reply is durably accounted for. Concretely:

• Record a durable delivery obligation before the first send attempt; transition pending → attempting → delivered and only mark the inbound journal complete once the obligation is delivered (or definitively failed).
• On boot, sweep undelivered obligations owned by dead processes and re-drive them; a send that may have landed (crash mid-await) is re-delivered with a visible recovered/duplicate marker rather than silently or blindly.
• Make this the default for every shipped adapter, not an opt-in mixin, with a typed send outcome so "delivered", "failed", and "ambiguous (may have landed)" are distinguishable instead of inferred from None/False/raise (delivery.py:820-833).

Layer placement

• Primary layer: wrapper (praisonai-bot — the gateway/bot runtime that owns adapters, OutboundQueue, DurableDelivery, the ingress journal).
• Why not core: core already holds the right protocol seams (praisonaiagents/gateway/protocols.py, bots/protocols.py, and the MESSAGE_SENT / MESSAGE_UNDELIVERED hook events); the missing work is wiring and safe-by-default adoption in the runtime, which must not put heavy SQLite/queue logic in core.
• Why not tools: this is framework-owned lifecycle reliability, not an agent-callable capability.
• Why not plugins: delivery integrity is a first-class runtime guarantee the framework must provide by default, not an optional lifecycle add-on a user opts into.
• Secondary touch: core — formalise a typed SendResult (delivered/failed/ambiguous) and an obligation contract on bots/protocols.py so every adapter reports outcomes uniformly.
• 3-way surface (CLI + YAML + Python): yes — a delivery/reliability block in gateway.yaml, the existing --reliability {production|default|off} start flag, and the Python adapter contract.

Proposed approach

• Extension point: adapter delivery contract (bots/protocols.py) + the existing OutboundQueue/DurableDelivery primitives, promoted to the default reply path via OutboundResilienceMixin.
• Defer inbound journal.complete() until the outbound obligation resolves (use the existing complete_last_journal_entry() seam).

Minimal API sketch:

class SendResult(TypedDict):
    status: Literal["delivered", "failed", "ambiguous"]
    provider_message_id: str | None
    recovered: bool  # True -> prepend the honest "possible duplicate" marker

class BotAdapter(Protocol):
    async def send_message(self, target, content, *, idempotency_key: str) -> SendResult: ...
    async def was_delivered(self, idempotency_key: str) -> bool | None: ...  # None = unknown

Resolution sketch

# Before (today): agent runs, journal completed, THEN reply sent — crash loses the reply
reply = await agent.chat(...)
self._ingress_journal.complete(journal_key)   # _session.py:1647
await adapter.send_message(target, reply)      # crash here => lost, no replay

# After (proposed): obligation recorded before send; inbound settled only on delivery
obl = outbox.record_obligation(session_key, message_ref, reply)   # pending
reply = await agent.chat(...)
outbox.mark_attempting(obl)
result = await adapter.send_message(target, reply, idempotency_key=obl.id)
if result["status"] == "delivered":
    outbox.mark_delivered(obl)
    self._ingress_journal.complete(journal_key)   # settle inbound ONLY now
# on boot: outbox.sweep_recoverable() re-drives 'attempting'/'pending' with a recovered marker

Severity

Critical — an accepted, fully-computed reply can be lost with no replay on any crash/redeploy in the send window; this is silent data loss of the gateway's primary output.

Validation

• src/praisonai-bot/praisonai_bot/bots/_session.py:1644-1650 — journal.complete() fires on clean exit, before adapter delivery; complete_last_journal_entry() seam at :2009 is unused.
• grep -rln 'def was_delivered' .../bots/*.py → slack.py only (effectively-once reconciliation is Slack-only).
• telegram.py:232-236 — outbound DLQ only initialised when dlq_path set (off by default); Telegram inlines its own retry instead of OutboundResilienceMixin (_outbound_resilience.py:147).
• Durable primitives confirmed present but opt-in: _outbox.py:113 (OutboundQueue, recovered state), _delivery.py:36 (RECOVERED_PREFIX), _delivery.py:456 (DurableDelivery), _durable_adapter.py:19 (DurableAdapterMixin).
• Implicit send-return contract: delivery.py:820-833 (None=success / False=failure / raise=failure).

Generated with Claude Code

https://claude.ai/code/session_01YXMLKVbg32jxwo829ZEApT