A brief `Unavailable` blip resets History queue backoff, causing a sustained retry storm

Summary

When a History cluster is running at its system persistence-QPS limit, the fleet stays stable only because ResourceExhausted errors put queue readers/task executables onto their long reschedule backoff, which spreads persistence load out over time. If persistence briefly returns a different transient error (Unavailable, or context deadline exceeded) - e.g. during a database failover/switchover - readers/executables fall back to their fast retry path, the fleet's long-backoff state is effectively reset, and on recovery every shard re-fires in a synchronized burst. The aggregate read QPS then exceeds the system persistence limiter, ack-level advancement is starved, and the queues settle into a saturated equilibrium instead of draining.

Environment

• Temporal Server version: v1.30.2 (behavior also present, byte-identical in the relevant code, on v1.31.1 and main - see code pointers)
• Persistence: Aurora PostgreSQL
• History service running at/near its configured system persistence-QPS limit

Current behavior

1. Under chronic system-QPS pressure, a fraction of persistence ops trip ResourceExhausted. Readers/executables detect this specific class via IsResourceExhausted(err) and reschedule on the long ResourceExhausted backoff curve (CreateTaskResourceExhaustedReschedulePolicy: initial ~3s, coefficient 1.5, max ~5m), which naturally de-synchronizes the fleet. RangeCompleteHistoryTasks slips through often enough to keep ack levels advancing. No customer impact, but zero headroom.
2. A brief persistence interruption (observed ~60s during a DB switchover) returns Unavailable / context-deadline for calls - the incident logs show this window as Unavailable/deadline, not ResourceExhausted.
3. The long backoff is engaged only when the error is ResourceExhausted:
   • Readers: on ResourceExhausted they wait a fixed throttle delay; for any other transient error they use the fast exponential retrier (~50ms initial → ~1s max), which is then reset once a call succeeds.
   • Executables: they increment a per-task resourceExhaustedCount and take the long reschedule only while that error is ResourceExhausted; any other error path resets resourceExhaustedCount = 0 and falls back to the default reschedule policy (~1s initial, 1.1 coefficient, ~3m max). For the duration of the blip, every reader/executable on every shard retries on its fast/default path and the accumulated long-backoff state is discarded fleet-wide.
4. On recovery, each shard's first successful read returns a full batch with MoreTasks=true, which triggers an immediate re-notification (no throttle between that batch and the next read). Thousands of shards across multiple scheduled queue categories become eligible at once. Each individual reader is still gated by its own rate limiter, so this is not a single reader spinning with zero gap - it is an aggregate, fleet-wide synchronized burst whose combined read QPS exceeds the system persistence cap.
5. RangeCompleteHistoryTasks (advances ack levels by deleting completed task ranges) goes through the same persistence rate limiter as the reads: GetHistoryTasks, CompleteHistoryTask, and RangeCompleteHistoryTasks all funnel through the same allow() gate (system + namespace + shard limiters). Under saturation it is throttled/failed alongside the reads, so ack levels don't advance, readers re-read the same ranges every cycle, and the limiter stays saturated. Archival tasks (which read history from persistence before writing the blob) fail the same way and reschedule, compounding read load.
6. Result: a stable "saturated" equilibrium. Individual task executions can succeed, but queues do not logically drain. Recovery only happens by scaling persistence and/or raising the QPS limit; the backlog then drains slowly.

Expected behavior

A short-lived Unavailable/deadline condition on a cluster that is otherwise persistence-rate-limited should not collapse the fleet's backoff and cause a persistent retry storm. Specifically:

• Transient non-ResourceExhausted persistence errors should not discard readers'/executables' long-backoff state in a way that synchronizes the whole fleet on recovery.
• Ack-level advancement (RangeCompleteHistoryTasks) should not have to compete on equal footing with the same reader load it is trying to relieve - otherwise there is no escape path from saturation.

Impact

Extended (multi-hour) stall of workflow progress across all shards on the affected cluster: workflow starts, signals, workflow-task completions, timer fires, and archival all intermittently fail or make no progress, even though the database itself is healthy after the blip. Frontend availability metrics understate the impact because affected shards make zero forward progress regardless of client retries.

Suggested directions (for discussion)

1. Apply the congestion-style long/jittered backoff to a broader set of transient persistence errors - at minimum Unavailable and context-deadline - not just ResourceExhausted, so a brief outage doesn't reset the fleet to the fast curve.
2. Add jitter / de-synchronization to reader re-fire after recovery so shards don't dispatch in lockstep when persistence returns.
3. Give ack-level advancement (RangeCompleteHistoryTasks) priority or a separate budget from read traffic through the persistence limiter, so the queue always has an escape path from saturation.
4. Gate immediate MoreTasks=true re-fires (e.g. small jittered delay) so a synchronized fleet of recovery batches can't collectively produce a read burst that exceeds the persistence cap.

Relevant code pointers

• common/util.go: CreateTaskResourceExhaustedReschedulePolicy (the long backoff, applied specifically for resource-exhausted); IsResourceExhausted (the actual decision point that selects the long curve); IsPersistenceTransientError (treats both Unavailable and ResourceExhausted as transient/retryable, but only the latter gets the long reschedule).
• service/history/queues/reader.go: fixed throttle delay on ResourceExhausted vs. the fast exponential retrier for other errors; MoreTasks() immediate re-notification.
• service/history/queues/executable.go: resourceExhaustedCount (incremented only on ResourceExhausted, reset to 0 on any other outcome) selecting the long reschedule.
• common/persistence/persistence_rate_limited_clients.go: GetHistoryTasks, CompleteHistoryTask, and RangeCompleteHistoryTasks all pass through the same allow() gate (shared system/namespace/shard rate limiters) - so ack-advancement competes with reads for the same budget.

(Code paths above verified identical at tags v1.30.2, v1.31.1, and main.)