Scanner never runs a cycle: leader stuck with pause backlog `persistence_state: unavailable`

Supported storage confirmation

• I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.

Describe the bug

[!NOTE] Disclosure: I made heavy use of LLMs when debugging and writing this issue. I do not grasp the RustFS internals as I am but a humble YAML-engineer. I have rephrased parts of this a little, but most of it is LLM output.

On a 4-pool distributed deployment running RustFS 1.0.0, the data scanner has never started a cycle. One node holds leader.lock and reports execution_role: leader, but never starts a cycle. It reports an empty cycles_completed_at and no completed cycle recorded (so do all other nodes, but scanner status is node-local, see the side note). Its pause backlog reports persistence_state: unavailable with scanner pause backlog has neither a surviving membership commit nor a stable rollback point. There is no decommission or rebalance, and data_movement_pause.paused is false.

Impact:

• Buckets with lifecycle rules exist (lifecycle_active: true), so expiry never runs.
• No usage snapshot has ever been published, so the console shows 0 buckets and 0 objects (the cluster has about a dozen buckets and millions of objects).

Leader status (rc admin scanner status, trimmed). Two snapshots taken 10 minutes apart are identical apart from collected_at:

{
  "metrics": {
    "current_cycle": 0,
    "current_cycle_active": false,
    "cycles_completed_at": [],
    "leader_lock_state": "acquired",
    "leader_lock_last_update_unix_secs": 1790693391
  },
  "cycle_schedule": { "execution_role": "leader", "effective_interval_seconds": 300 },
  "cycle_recovery": { "state": "healthy", "last_attempt_at_unix_secs": 1790693391 },
  "data_movement_pause": { "paused": false, "reasons": [] },
  "pause_backlog": {
    "persistence_state": "unavailable",
    "replica_count": 0,
    "path": "buckets/.scanner-pause-backlog.json",
    "error": "scanner pause backlog has neither a surviving membership commit nor a stable rollback point; <same message again>"
  }
}

• 1790693391 = 2026-09-29T14:49:51Z, when the leader acquired leader.lock and loaded the cycle state. Nothing in the status has changed since.

• The effective interval is 300 seconds instead of the configured 60 seconds, with clean-idle/superseded/deferred backoff all off. That matches SCANNER_PAUSE_REFRESH_INTERVAL_SECONDS, the retry delay used while the pause backlog is persistence_disabled.

• With RUST_LOG=info,rustfs::scanner=debug, the leader logs the wait on a 300-second grid (the one under "Relevant logs" is exactly 6 × 300 seconds after 14:49:51), but never Scanner cycle started.

• The only scanner ERROR is Scanner pause backlog persistence is unavailable, logged once at startup (see "Relevant logs").

• All pools report decommission status none, and no rebalance is running.

Steps to reproduce

We don't have a minimal reproduction. This is how the cluster got into this state:

1. Deploy RustFS 1.0.0 with the official Helm chart and 4 pools. It would not start with all pools at once (we no longer have the errors; possibly the multi-pool bootstrap issue fixed in #8216).
2. Deploy with 1 pool instead, then add the other 3 pools once those pods are running.
3. Upload data (backups, about a dozen buckets, millions of objects, totalling around 7TiB of used usable storage) after all 4 pools are up.
4. Run rc admin scanner status against each node: the leader has never started a cycle, and its pause backlog stays unavailable across restarts.

Expected behavior

A cluster whose pause backlog has never been committed should be able to bootstrap a fresh ledger (or at least run scanner cycles without persistence) instead of skipping every cycle indefinitely. If manual intervention is needed, the error should say what to do.

RustFS version

1.0.0

Deployment method

Kubernetes / Helm

Data directory filesystems and mounts

RustFS data paths: /data in each pod (1 drive per pod, 22 pods) Filesystem and mount source: XFS, each on its own local 4 TB SATA SSD, all on one host Kubernetes: local PersistentVolumes from sig-storage-local-static-provisioner 2.9.0, one StorageClass per pool (backup-rustfs-pool-{0,1,2,3}), host disks mounted under /mnt/disks/by-uuid/backup-rustfs-pool-$N/

Mount command output:

TARGET                                                                       SOURCE   FSTYPE OPTIONS
/mnt/disks/by-uuid/backup-rustfs-pool-0/2af4d638-0d66-42ba-b4c3-24e08bad219f /dev/sdf xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-0/51af677a-1188-452f-8b7c-9e8150f07d8f /dev/sdd xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-0/5ac8d993-0a42-41da-a3ad-91114778c22b /dev/sda xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-0/80201dd6-01d9-41fc-9aa6-3002f153ffa8 /dev/sde xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-0/cb8d1781-73ed-418e-9001-cfa1fd2db34e /dev/sdb xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-0/e1dd1c96-fd38-4c52-b020-ef58f9df82ab /dev/sdc xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-1/37b7e56b-74a3-4f76-ac6a-33bb382a2265 /dev/sdk xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-1/69d5ee4b-ec5f-47a2-aa9b-b59a5e7ef857 /dev/sdl xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-1/8459890b-0998-424e-a121-0af6c9d64bfe /dev/sdg xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-1/91267a45-36c5-42c5-934e-03c0b4ce4145 /dev/sdh xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-1/9f1ab97b-bd5c-4f51-aacb-5daa8a963578 /dev/sdj xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-1/cbbee780-ecb0-40b1-a3ff-fb0339bc6d6d /dev/sdi xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-2/2844f556-714e-4fc0-b23b-b1045896ce85 /dev/sdp xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-2/489f943b-9b36-41b3-aac7-2208fae740a0 /dev/sdm xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-2/5e527359-cbfd-44ae-a590-fe8bd328d7b4 /dev/sdr xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-2/9f3a9cf7-ee46-489a-b59f-e56143a7205e /dev/sdn xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-2/ed1cd0c5-8518-4498-bfec-e0ef40e88650 /dev/sdo xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-2/ef84c5be-7e5b-40ba-83dd-09b7c79cf0e8 /dev/sdq xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-3/2f487cd8-2e1a-4c26-b1f5-cabb0f759773 /dev/sds xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-3/76b7ecd0-c085-4c39-a400-7332795ff2f4 /dev/sdu xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-3/d2521d11-fc41-47e5-811f-0820e8457007 /dev/sdv xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
/mnt/disks/by-uuid/backup-rustfs-pool-3/eaeb312b-8f11-4cc8-9c8e-130000681071 /dev/sdt xfs    rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota

Server operating system and architecture

Ubuntu 24.04.5 LTS (Linux 6.8.0-142-generic), x86_64

Client environment

Not client-specific: the scanner is a background task. Status was read with the rc CLI (rc admin scanner status) and the web console.

But for completeness:

$ rc --version
rc 0.1.36

Accessed the console using Zen browser 1.22.2b (Firefox 156.0) (64-bit) from Fedora Linux

Relevant logs

# Current leader, with RUST_LOG=info,rustfs::scanner=debug: waits on a 300-second grid, never starts a cycle
{"timestamp":"2026-09-29T15:19:51.449569503Z","level":"DEBUG","message":"Scanner cycle wait scheduled","event":"scanner_cycle_state","component":"scanner","subsystem":"runtime","configured_interval":"60s","effective_interval":"300s","clean_idle_max_interval":"2531.25s","scheduled_delay":"300s","usage_bootstrap_rebuild_pending":false,"interval_multiplier":1,"clean_idle_backoff_enabled":false,"superseded_retry_backoff_enabled":false,"superseded_cycles":0,"deferred_retry_backoff_enabled":false,"deferred_cycles":0,"lifecycle_active":true,"replication_active":false,"feature_inspection_failed":false,"state":"wait_scheduled","target":"rustfs::scanner","filename":"crates/scanner/src/scanner.rs","line_number":3143,"threadName":"rustfs-worker","threadId":"ThreadId(70)"}

# Logged once at startup on every restart (this one is from an earlier restart, with a different leader)
{"timestamp":"2026-09-29T12:16:53.703933404Z","level":"ERROR","message":"Scanner pause backlog persistence is unavailable","event":"scanner_persist_state","component":"scanner","subsystem":"runtime","state":"pause_backlog_claim_failed","error":"scanner pause backlog has neither a surviving membership commit nor a stable rollback point","target":"rustfs::scanner","filename":"crates/scanner/src/scanner.rs","line_number":2737,"threadName":"rustfs-worker","threadId":"ThreadId(68)"}

# Before RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30: leadership claim failing on lock timeouts after a restart of all pods
{"timestamp":"2026-09-29T09:47:08.815637035Z","level":"ERROR","message":"Scanner leadership claim failed","event":"scanner_persist_state","component":"scanner","subsystem":"runtime","path":"buckets/.bloomcycle.bin","state":"leader_claim_failed","error":"Lock error: Lock acquisition timeout for resource '.rustfs.sys/pool.bin@latest' after 5s","target":"rustfs::scanner","filename":"crates/scanner/src/scanner/leadership.rs","line_number":551,"threadName":"rustfs-worker","threadId":"ThreadId(74)"}
{"timestamp":"2026-09-29T09:47:31.668791217Z","level":"ERROR","message":"Scanner leadership usage epoch fencing failed","event":"scanner_persist_state","component":"scanner","subsystem":"runtime","path":"buckets/.usage.v2.json","state":"usage_epoch_fence_failed","claimed_epoch":53,"error":"Scanner error: scanner usage epoch fence save failed: Lock error: Lock acquisition timeout for resource '.rustfs.sys/buckets/.usage.v2.json@latest' after 5s","target":"rustfs::scanner","filename":"crates/scanner/src/scanner/leadership.rs","line_number":331,"threadName":"rustfs-worker","threadId":"ThreadId(69)"}

# Inbound ReadVersion requests in flight for more than 5 seconds (leader at the time, earlier restart)
{"timestamp":"2026-09-29T13:29:52.194959746Z","level":"WARN","message":"HTTP request remains in flight","event":"http_request_inflight_slow","component":"server","subsystem":"http","request_id":"req-4bdfb9c5","trace_id":"unknown","span_id":"unknown","peer_addr":"::ffff:<censored pod IPv4>","method":"POST","uri":"http://rustfs-pool2-4.rustfs-headless.rustfs-backup.svc.cluster.local:9000/node_service.NodeService/ReadVersion","duration_ms":5001,"active_requests":0,"threshold_ms":5000,"state":"response_pending","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":462,"threadName":"rustfs-worker","threadId":"ThreadId(79)"}

Screenshots

Additional context

Topology and configuration:

• 4 pools: 6 + 6 + 6 + 4 pods, 1 drive per pod, storage_class_standard: EC:2 (22 drives, split into pools so we can take parts of the cluster offline pool by pool)
• Scanner config: all defaults (cycle_interval_seconds: 60)
• RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30 (see below)

Where we think it gets stuck

Links are to the 1.0.0 tag (d47f54b):

1. ScannerPauseBacklogController::claim fails in select_scanner_pause_backlog_replicas, because there is no authoritative commit and there is no stable consensus: https://github.com/rustfs/rustfs/blob/d47f54bfb2f39f48bd1adda334bd27e151fe85b8/crates/scanner/src/scanner/backlog.rs#L1100-L1108

   scanner_pause_backlog_consensus requires every replica's stable record to be identical, so a mix of present and missing replicas returns Err(()): https://github.com/rustfs/rustfs/blob/d47f54bfb2f39f48bd1adda334bd27e151fe85b8/crates/scanner/src/scanner/backlog.rs#L981-L991

2. The scanner logs the error and falls back to ScannerPauseBacklogController::unavailable: https://github.com/rustfs/rustfs/blob/d47f54bfb2f39f48bd1adda334bd27e151fe85b8/crates/scanner/src/scanner.rs#L2734-L2748

   Which sets persistence_disabled: true: https://github.com/rustfs/rustfs/blob/d47f54bfb2f39f48bd1adda334bd27e151fe85b8/crates/scanner/src/scanner/backlog.rs#L1609-L1631

3. begin_attempt_with then always returns PersistenceUnavailable: https://github.com/rustfs/rustfs/blob/d47f54bfb2f39f48bd1adda334bd27e151fe85b8/crates/scanner/src/scanner/backlog.rs#L1674-L1676

   And the leader loop continues without starting a cycle: https://github.com/rustfs/rustfs/blob/d47f54bfb2f39f48bd1adda334bd27e151fe85b8/crates/scanner/src/scanner.rs#L3274-L3279

4. Every 300 seconds, try_recover_persistence re-runs claim on the same replicas, which fails the same way, so the scanner never recovers on its own: https://github.com/rustfs/rustfs/blob/d47f54bfb2f39f48bd1adda334bd27e151fe85b8/crates/scanner/src/scanner/backlog.rs#L1755-L1775

The same code is in 1.0.1-preview.11: the PersistenceUnavailable → continue branch is unchanged, and the consensus failure is only reclassified as AuthorityConflict.

Possible causes for the replicas getting out of sync:

• Bootstrap: the cluster has never completed a cycle, so the backlog file may never have been fully written on every set.
• Pool expansion: our pools were added shortly after the first one started (see "Steps to reproduce"). #7594 fixed a writer bug where pool expansion could leave missing or stale pause backlog replicas, and its regression test failed with the same error.

For comparison, a second cluster of ours with a single pool (also 1.0.0) has a healthy pause backlog (replica_count: 1, durable: true) and completes scanner cycles normally.

Related pause backlog consensus failures: #7967 / #7979 (decommission restart). Older reports with the same symptom but a different cause (usage floor): #6907, #6969.

#8141 also involves a 5-second lock timeout on a scanner state file, but is a different failure: there cycles keep running and remote scans are rejected with scanner_cycle_mismatch until the leader restarts. Here no cycle ever starts, restarts don't help, and we see no scanner_cycle_mismatch.

Lock timeouts

Before we set RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30, the leadership claim itself also failed after a restart of all pods, on lock timeouts for .rustfs.sys/pool.bin and .rustfs.sys/buckets/.usage.v2.json (see "Relevant logs"), which looks related to #7962. The pause backlog error was already present then. With the 30-second timeout the lock timeouts are gone, but the scanner still doesn't run because of the pause backlog. #8227 describes a similar pattern in a multi-pool E2E run: a 5-second lock timeout while publishing .scanner-pause-backlog.json, recovery waiting the 5-minute refresh interval, and pool.bin lock timeouts. We also see inbound NodeService/ReadVersion requests staying in flight for more than 5 seconds.

Questions

• Is there a supported way to reset the pause backlog, similar to rc admin scanner usage-state reset / cycle-state reset mentioned in #7953?
• If not, is it safe to delete .rustfs.sys/buckets/.scanner-pause-backlog.json (all replicas) so the next claim starts from an empty ledger? If so, what is the recommended way to do that?
• Is there a way to list the per-set replica states, to see which sets disagree? The status reports replica_count: 0 while unavailable.

Side note: scanner status is node-local

While debugging this, the console's "Running Status" page and rc admin scanner status were confusing, because they only show the scanner state of the node that answers the request. On a follower, the status reads "Never run" / no completed cycle recorded even on our healthy single-pool cluster, where the leader completes cycles normally. We had to query every node to find the leader.

Preferably, the status would show the cluster-wide scanner state regardless of which node answers. At minimum, a follower should say that it is a follower and which node currently holds leader.lock.