Gateway horizontal-scale story is asymmetric: turn-lock has a Redis backend but inbound idempotency and the session store do not, so multiple replicas double-process the same message

Summary

The gateway can be run behind multiple replicas (there is a cluster-wide RedisTurnLock), but two of the three pieces of shared state needed for safe horizontal scaling are missing: inbound idempotency has no shared/Redis backend (it silently runs per-replica), and the gateway session store is sqlite/file only (no shared or fail-over store). The result is that the same inbound webhook delivered to two replicas that do not share a state file can be de-duplicated on each replica independently and processed twice — the turn-lock does not help because each replica computes a different lock only after it has already admitted the duplicate, and a client reconnecting to a different replica cannot resume its live session. Per-process edge rate-limiters compound this (an attacker gets N× the budget across N replicas).

For a world-class gateway, "just add a replica" must be safe by construction. Today it is a correctness footgun that a green health surface hides.

Current behaviour

Turn-lock is cluster-ready (good): RedisTurnLock (_redis_turn_lock.py) uses SET NX PX leases with owner-token compare-and-delete and fails open to a local lock with a DegradedCapabilityRegistry entry.

Inbound idempotency is not — the Redis backend is explicitly unimplemented and downgrades to per-replica SQLite:

# src/praisonai-bot/praisonai_bot/bots/_idempotency.py:275-311
if backend == "redis":
    # A cross-replica Redis idempotency backend is not yet implemented in
    # the wrapper (unlike ``RedisTurnLock`` for the turn lock). ...
    logger.warning(
        "Idempotency store_backend='redis' is not implemented; inbound "
        "dedup runs per-replica (durable SQLite fallback). A message "
        "delivered to multiple replicas that do not share the state file "
        "may be processed more than once. ..."
    )

So multi-replica inbound dedupe only holds if every replica shares one SQLite file on a shared volume; otherwise the same webhook runs twice. (The degradation is honestly reported — this is not a silent-failure bug — but the capability is absent.)

The gateway session store is not shareable — only sqlite and file backends exist:

# src/praisonai-bot/praisonai_bot/gateway/server.py:863,889
store = DefaultSessionStore(session_dir=persist_path)   # sqlite|file only

There is no Redis/shared session store, so live session objects (WebSocketGateway._sessions) have per-replica affinity: a client that reconnects to a different replica finds only the persisted transcript, not its live in-flight session, and there is no ownership/fail-over so a second replica can safely take over a session from a dead one.

Edge rate-limiters are per-process (rate_limiter.py: AuthRateLimiter, PreauthConnectionBudget, UnauthorizedFloodGuard), so budgets multiply by replica count.

Desired behaviour

Running N replicas is safe by construction:

• Shared inbound idempotency so a webhook fanned to multiple replicas is admitted exactly once, keyed on (platform, account, channel_id, message_id).
• A shared / ownership-fenced session store so a session has a single owner at a time and a live replica can take over a session from a crashed one without split-brain (durable state stamped with an owner identity + a monotonic generation, claimed by fenced compare-and-swap).
• Shared edge rate-limiting so budgets are cluster-wide, not per-process.

Layer placement

• Primary layer: wrapper (praisonai-bot — it owns the idempotency store, the gateway session store selection, and the edge limiters; the Redis client and cross-replica coordination are heavy integrations that belong here).
• Why not core: the contracts already exist in core as protocols (session/protocols.py:SessionStoreProtocol, the idempotency store shape, gateway/protocols.py); a Redis-backed implementation is a heavy optional dependency and must not enter the protocol-only core.
• Why not tools: this is gateway infrastructure, not an agent-callable capability.
• Why not plugins: exactly-once admission and single-owner sessions are core runtime-correctness guarantees, not optional lifecycle add-ons.
• Secondary touch: core — add an owner-stamp + generation-fence method to SessionStoreProtocol (claim/renew/release) so any backend can implement safe takeover uniformly.
• 3-way surface (CLI + YAML + Python): yes — gateway.session_store: redis, gateway.idempotency.store_backend: redis, and matching start flags; Python selects the backend via config.

Proposed approach

• Extension point: implement the already-declared store_backend='redis' idempotency path, add a RedisSessionStore implementing SessionStoreProtocol, and add a shared limiter behind the existing edge-limiter interfaces. Reuse the ownership pattern the durable delivery path already relies on (owner id + start-time stamp + stamp-guarded atomic claim, with PID-reuse detection) so session takeover is split-brain-safe.

Minimal API sketch:

class SessionStoreProtocol(Protocol):
    def claim(self, session_id: str, owner: OwnerStamp, generation: int) -> bool: ...  # fenced CAS
    def renew(self, session_id: str, owner: OwnerStamp) -> None: ...
    def release(self, session_id: str, owner: OwnerStamp) -> None: ...

# config
gateway:
  session_store: redis          # sqlite | file | redis
  idempotency: { store_backend: redis }

Resolution sketch

# Before (today): idempotency Redis backend missing -> per-replica dedup
if backend == "redis":
    ... # not implemented; downgrade to per-replica SQLite -> same webhook runs twice

# After (proposed): shared dedup + fenced session ownership
if backend == "redis":
    return RedisIdempotencyStore(client)          # cluster-wide exactly-once admission

# reconnect to any replica: claim (or take over) the session under an ownership fence
if store.claim(session_id, owner=self.owner, generation=gen):
    session = store.load(session_id)              # resume live state on this replica

Severity

High — the advertised multi-replica deployment can double-process messages (duplicate side effects: duplicate replies, duplicate tool actions) and cannot fail a session over between replicas; correctness silently depends on an undocumented shared-volume requirement.

Validation

• Turn-lock has a Redis backend: src/praisonai-bot/praisonai_bot/bots/_redis_turn_lock.py (RedisTurnLock).
• Idempotency Redis backend explicitly unimplemented, downgrades per-replica: src/praisonai-bot/praisonai_bot/bots/_idempotency.py:275-311.
• Gateway session store is sqlite/file only: src/praisonai-bot/praisonai_bot/gateway/server.py:863,889 (DefaultSessionStore); SessionStoreProtocol defined in src/praisonai-agents/praisonaiagents/session/protocols.py.
• Per-process edge limiters: src/praisonai-bot/praisonai_bot/gateway/rate_limiter.py (AuthRateLimiter, PreauthConnectionBudget, UnauthorizedFloodGuard).

Generated with Claude Code

https://claude.ai/code/session_01YXMLKVbg32jxwo829ZEApT