S3 backend: backup stalls silently for the duration of a stalled connection - 5-min watchdog timeout is absorbed by minio-go's silent internal retries

Output of restic version

restic 0.18.1 compiled with go1.25.1 on darwin/arm64

(reproduced with the official release binary; also relevant to any version using minio-go v7 with the backend-error-redesign watchdog, which is beta = enabled by default in 0.18.1)

What happened?

Backing up to S3/MinIO, the upload froze for the entire duration of a stalled connection without printing a single error or retry line. When the connection eventually started working again, the backup instantly resumed at full speed.

The interesting part: restic 0.18.1 does have protection for this — the backend-error-redesign watchdog cancels requests that make no progress for 5 minutes (StuckRequestTimeout in internal/backend/http_transport.go). I can show that it fires — but its cancellation is silently absorbed by minio-go's internal retry loop, so each watchdog timeout just starts another 5-minute stall with no user-visible output.

Steps to reproduce

Deterministic, self-contained, single command (downloads and sha256-pins restic 0.18.1 and a MinIO server binary, then SIGSTOPs the server mid-PUT — the kernel keeps ACKing TCP for the frozen process, so from restic's side the connection stays ESTABLISHED while the peer never responds):

./repro.sh            # freezes the server for 12 minutes (2.4x the watchdog timeout)

Repo: https://github.com/schickling-repros/2026-08-restic-s3-stall (includes the full recorded timeline)

In short: 1 GiB of random data → restic backup --limit-upload 4096 against s3:http://127.0.0.1:39093/repro → once bytes are flowing, kill -STOP the MinIO process → wait → kill -CONT.

Expected behavior

A stalled connection should surface to the user within a bounded, visible time. Since the watchdog cancels stuck requests after 5 minutes, at the latest ~5 minutes into the freeze the user should see retry lines like Save(<data/…>) returned error, retrying after ….

Actual behavior

Measured during a 723-second freeze (full 10s-resolution timeline in the repro repo):

09:38:19 | >>> FREEZING minio (SIGSTOP) mid-upload -- 40232 KiB already stored, error/retry lines so far: 0
09:43:21 | frozen 302s: stored=40232KiB error/retry_lines_total=0 est_conns=6 NEW_SILENT_RECONNECT(client_port(s): 64662 64663 64664 64665 )
09:48:22 | frozen 603s: stored=40232KiB error/retry_lines_total=0 est_conns=7 NEW_SILENT_RECONNECT(client_port(s): 49684 49685 49686 49687 49701 )
09:50:22 | >>> UNFREEZING minio (SIGCONT) after 723s -- bytes stored during entire freeze: 40232 KiB -> 40232 KiB, error/retry lines surfaced: 0
09:54:38 | restic exit status: 0

• 12 minutes (= 2.4× the watchdog timeout): zero new bytes stored, zero error/retry lines printed.
• The watchdog does fire under the hood — visible only as silent TCP reconnect storms at exactly ~300s and ~600s (27 distinct client ports used during the freeze) — but each minio-go retry just stalls for another 5 minutes, invisible to the user.
• After SIGCONT the backup resumed instantly and completed with exit 0.

Why this compounds to multi-hour freezes

From source (restic 0.18.1, minio-go v7.0.95):

• minio-go retries internally up to MaxRetry = 10 times (retry.go), silently, before an error reaches restic,
• each attempt stalls for the full 5-minute watchdog timeout before cancellation → one restic-level Save needs ≈ 11 × 5 min ≈ 55 minutes before the first user-visible retry line,
• restic's own retry layer then repeats the cycle (retry.New(be, 15*time.Minute, …)), so a permanently dead path means roughly 2¾ hours of silence before the backup finally aborts (extrapolation from source, not measured).

This matches what we saw in production against a MinIO endpoint over a WAN link: recurring freezes of minutes-to-hours (longest observed: 2h37m) with no errors, NAS receive counters flat, and sudden full-speed recovery.

Do you have an idea how to fix it?

Making the watchdog timeout (or a shorter stall budget) propagate as a visible, logged condition — e.g. surfacing minio-go's internal retries to restic's retry/report layer, or capping the total time a single Save may spend inside minio-go's internal retry loop — would turn silent multi-hour freezes into visible retry behavior.

Did restic help you today?

Yes, massively — restic has been quietly excellent except for this failure mode being hard to diagnose precisely because it's silent.