Remote disks remain `Suspect` under continuous successful gRPC traffic after a transient probe failure

Supported storage confirmation

• I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.

Describe the bug

After a transient connectivity-probe failure moves a remote disk from Online to Suspect, continuous successful disk RPCs keep it in Suspect indefinitely. Three behaviours combine:

1. Success does not change state. The RPC success path calls log_success(), which only refreshes a last_success timestamp.
2. Success actively skips the probe. The periodic health monitor skips its connectivity check whenever a successful operation occurred within the last five seconds.
3. The first failure never starts recovery. mark_failure() returns true only when the drive transitions to Offline, so entering Suspect does not schedule recovery monitoring.

The result is recovery starvation: the disk is not stuck because probing fails, but because successful I/O prevents probing from ever being attempted. Node readiness counts only reachable drives whose runtime state is Online, so affected remote drives can be dropped from the quorum calculation even while their RPCs succeed.

Trigger: any single transient probe failure on a remote disk, followed by continued successful traffic to that disk. No real hardware failure or sustained network outage is required.

Scope: the defect is specific to RemoteDisk. At this commit, record_operation_success() is already used by the connectivity-probe success paths in the same file (remote_disk.rs L1323 and L1394) and by the local-disk operation paths in disk_store.rs (L1924, L1938, L2212, L3418). Only the remote RPC success path (L1694 and L1708) is left on the timestamp-only log_success(). Local disks are unaffected.

Reproduced on the official commit below with real, authenticated gRPC ReadAll requests. Over seven seconds all 68 requests succeeded while the runtime state stayed Suspect; once traffic stopped, the state recovered to Online. Replacing the two success-path log_success() calls with record_operation_success() made the same regression pass.

Steps to reproduce

1. Check out upstream commit 8741bb77eaa1a3cbe56d017934e17a0ce9e17aa3 (rustfs/rustfs).
2. Add the attached suspect_repro.rs inside the existing #[cfg(test)] mod tests of crates/ecstore/src/cluster/rpc/remote_disk.rs via include!("suspect_repro.rs");. It reuses that module's AuthenticatedReadPeer / AuthenticatedReadPeerService.
3. Run with Rust 1.98.1: cargo test --locked -p rustfs-ecstore --lib regression_successful_grpc_must_clear_suspect -- --nocapture --test-threads=1.
4. Observe that every real gRPC call succeeds, yet the assertion requiring Online during successful traffic fails. The idle control in the same test restores Online before the final assertion.

The test reserves and releases a loopback port to force a real TCP refusal. Once Suspect is observed, it starts the gRPC server on that same port and sends a request roughly every 100 ms. The monitor interval is shortened to one second for the test; the production five-second recent-success threshold is unchanged.

Expected behavior

Once the remote endpoint is reachable and authenticated disk RPCs succeed repeatedly, the runtime state must be able to leave Suspect and advance to Online under the recovery policy without waiting for traffic to go idle. Readiness must not keep excluding a recovered drive merely because active I/O prevents its cached state from advancing.

RustFS version

• Release associated with the original deployment: 1.0.1-preview.12. - Official source commit used for the regression: 8741bb77eaa1a3cbe56d017934e17a0ce9e17aa3. - The unpatched test uses an archive of this commit with only the regression test added. The production contents of remote_disk.rs, disk_store.rs, and readiness.rs were compared byte-for-byte with GitHub's files at this commit and matched. - Rust toolchain: rustc/Cargo 1.98.1.

Deployment method

Kubernetes / Helm

Data directory filesystems and mounts

Minimal regression: no RustFS data directories are created or opened; the gRPC fixture returns an in-memory payload. The source/build directory and the Docker/BuildKit overlay2 store both sit on local ext4 (data=ordered). Original deployment: three nodes, four locally attached data disks each. All 12 backing filesystems are local ext4 mounted at /home/sdb…/home/sde (rw,noatime,nodiratime,seclabel). Container paths /data/rustfs0…/data/rustfs3 map to those hostpaths through OpenEBS LocalPV (openebs.io/local, StorageType=hostpath, WaitForFirstConsumer), each PV bound to the corresponding local directory with node affinity. No remote/shared filesystem underneath.

Server operating system and architecture

• Reproduction container: Ubuntu 22.04, Linux x86_64. - Docker host: CentOS Linux 7, kernel 3.10.0-957.el7.x86_64, x86_64. - Docker Engine 25.0.3; storage driver overlay2.

Client environment

The regression uses RustFS's generated tonic NodeService client and its existing authenticated gRPC test service (tonic 0.14.6) over a loopback TCP connection with the repository's test RPC-signature setup.

This is not a mocked successful future: RemoteDisk::read_all() sends real gRPC requests, and the server's read_all_calls counter independently confirms all 68 calls. The test service returns a fixed payload rather than accessing object shards. No S3 SDK or browser is involved.

Relevant logs

Unpatched official source:


REPRO initial_state=Suspect recovery_monitor=false
REPRO grpc_successes=68 server_read_all_calls=68 busy_state=Suspect strict_online=false
REPRO idle_state=Online

assertion `left == right` failed: successful gRPC traffic must restore Online rather than strand Suspect
  left: Suspect
 right: Online

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5721 filtered out; finished in 12.06s


After the fix below:


REPRO initial_state=Suspect recovery_monitor=false
REPRO grpc_successes=68 server_read_all_calls=68 busy_state=Online strict_online=true
REPRO idle_state=Online

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5721 filtered out; finished in 7.09s

Screenshots

Not applicable; the reproduction is a source-level network regression test.

Additional context

Relevant code paths at the affected commit:

• DiskHealthTracker::mark_failure() — first failure moves Online to Suspect; the boolean return indicates entering Offline.
• monitor_remote_disk_health() — the initial probe failure only schedules recovery monitoring when that boolean is true.
• Recent-success skip — skips connectivity probes without requiring an Online runtime state.
• RPC success handling — success calls log_success(), which only updates a timestamp. This is the inconsistent call site.
• Node readiness — only reachable Online runtime observations count as available drives.

Inconsistency inside the same file: record_operation_success() already has exactly the required semantics — it calls log_success() when the drive is Online and mark_recovery_success() otherwise — and the connectivity-probe success paths in the very same remote_disk.rs already call it. The RPC success path is the only caller left on the timestamp-only helper.

Tested fix (attached as suspect-success.patch): replace self.health.log_success() with self.health.record_operation_success(&self.endpoint, "operation_success") at the two RPC success sites in remote_disk.rs (L1694 and L1708) — the only two occurrences of that call in the file, matching the probe paths above. This reuses existing recovery handling rather than changing quorum thresholds, and makes the regression pass.

The original deployment is a single 12-drive erasure set with Standard storage class EC 8+4. After member replacement and pod restarts, one node continuously reported /health/ready HTTP 503 with storage_quorum_unavailable while its cluster read/write health endpoints returned HTTP 200. This motivated the investigation. The regression above confirms the health-state defect; the exact runtime state of each drive handle on that original node was not captured, so attribution of that entire production symptom remains unconfirmed.