#4718·valkey

[NEW] Exponential backoff for full sync attempts

Author: murphyjacob4Created Sep 16, 2026Updated Sep 16, 2026
Labelsenhancement

The problem/use-case that the feature addresses

Today, when a replica fails during a full resync (for example due to a transient network timeout during RDB transfer, client output buffer overflow on the primary, or OOM/load issues while creating/reading the snapshot), it tears down the connection and resets server.repl_state = REPL_STATE_CONNECT.

In replicationCron(), the replica immediately attempts to reconnect on a fixed 1-second interval. Once reconnected, if it cannot do a partial resync (which is usually the case since the cached offset is either invalid or fallen behind the backlog), it immediately re-initiates a full PSYNC.

If the root cause that triggered the failure hasn't cleared up (e.g., high write load continuing to bust the replication buffer, or congested network), this creates a tight retry loop:

  1. The replica hammers the primary for a new full sync every ~1 second.
  2. The primary repeatedly forks/spawns BGSAVE or initiates diskless transfers, chewing up significant CPU, memory, and bandwidth.
  3. If multiple replicas disconnect at the same time (e.g., rack switch flap or transient AZ packet loss), they all hammer the primary in lockstep on the same 1-second cadence, creating a persistent thundering herd that makes it nearly impossible for the primary or replicas to recover.

Description of the feature

Introduce exponential backoff with randomized jitter on the replica side for repeated failed full sync attempts. Rough idea of how this could work:

  1. Track consecutive full sync failures on the replica (e.g. a counter or last-failure timestamp incremented when an in-progress full sync disconnects or fails to reach steady state).
  2. When scheduling the next reconnect/PSYNC attempt in replicationCron(), compute a delay with exponential backoff and jitter:
    • delay = min(max_backoff, base_delay * (2 ^ failures)) + jitter
    • Normal transient drops where a partial resync (PSYNC) succeeds can stay on the fast path (1s). The backoff would kick in once a full resync fails.
  3. Reset the failure count / backoff timer back to baseline as soon as the replica successfully finishes loading the RDB and transitions into steady-state replication (REPL_STATE_CONNECTED).
  4. Config knobs:
    • repl-sync-backoff-base-time (default: 1s)
    • repl-sync-backoff-max-time (default: 30s or 60s, or 0 to disable for legacy behavior)

Alternatives you've considered

  1. Primary-side gating (#3893): PR #3893 introduces repl-disable-full-resync-until on the primary to reject full resyncs with -NOMASTERLINK. This is helpful for control-plane mitigation during massive DC events, but it requires operator/orchestrator intervention and doesn't provide client-side / replica-side self-healing.
  2. Fixed retry delay: We could just bump the retry delay from 1s to something static like 5s or 10s, but without exponential growth and jitter, synchronized thundering herds will still hit in waves when multiple replicas are involved.
  3. Write throttling (#1649 / #1672): Throttling writes to avoid buffer overflow addresses keeping replicas from falling out of sync during steady state, but doesn't address how to safely recover once a replica has already failed a full sync and needs to retry.

Additional information

Exponential backoff is the standard, but it could possibly be worse here. Something to consider is that the primary would wait and try to coalesce multiple replica syncs into a single snapshot. There is a tradeoff between separating the replication request timing and increasing the number of overall full syncs/forks. I think it is probably fine since this is only on the error path.