Durable MRF partial-write intents for healthy legacy objects do not discharge

Durable MRF partial-write intents for healthy legacy objects do not discharge

Summary

On RustFS de484b93e (one node, four local disks, EC 2+2), a durable MRF partial-write backlog persists with no foreground writes or active scanner. The background-heal manager queue stays near 9,000 normal-priority MRF tasks. A scoped deep heal of a sampled target reports all four drives ok but returns an unknown outcome because the object uses the default legacy shard format. The durable MRF journal is unchanged after repeated attempts and a service restart.

Hostnames, volume paths, bucket names, and object keys below are omitted or generalized.

Observations

• After a successful full scanner cycle (7 h 26 min), the heal queue remained near 9,000. The scanner was then put on a 24-hour cadence.
• Restarting RustFS dropped healQueueLength from about 9,000 to 5,137 briefly. It refilled to about 9,000 within one minute while the scanner was inactive.
• The signed POST /rustfs/admin/v3/background-heal/status response showed queuedBySource.mrf = 8,999, all at normal priority; all other queued sources were zero. One MRF task was active. admission.full and admission.merged increased rapidly without foreground writes.
• The identical on-disk buckets/.heal/mrf/journal-scoped.bin snapshots on all four disks contain 16,409 CRC-valid records. All records are scoped, unversioned PartialWrite intents; 16,406 refer to application objects and three to internal metadata. The snapshot was written at restart and remained byte-for-byte unchanged for more than 25 minutes while the manager cycled through target objects.
• Three sampled application targets are visible by S3 HEAD. A sampled 568-byte target has object metadata and a data directory on all four disks. A scoped admin deep heal with recreate=false, remove=false completed, but its raw outcome had disposition.state = "unknown", bucketIncarnationId = null, poolIndex = null, and setIndex = null. Its item detail was: Legacy object uses standard repair; independent object identity remains unverified. Before and after drive states were all ok; no bytes were healed.
• No new heal warnings or errors appeared after the restart. The queue remains full silently.

Source-level explanation to investigate

The default RUSTFS_SHARD_INTEGRITY_WRITE and RUSTFS_SHARD_INTEGRITY_FLEET_CONFIRMED values are both false (crates/config/src/constants/object.rs). In crates/ecstore/src/set_disk/ops/heal.rs, integrity_verified requires a protected object; metadata_verified also requires protection or a deleted object. For a healthy legacy object, the deep heal therefore produces no VerifiedHealthy receipt. crates/heal/src/heal/storage.rs returns no object receipt when neither integrity nor metadata was verified. crates/heal/src/heal/manager/scheduler.rs only publishes an MRF verified-repair event for a Repaired, VerifiedHealthy, or authoritatively absent object result. crates/heal/src/heal/mrf_queue/partial_write.rs retains durable responsibilities until an exact verified proof is recorded.

This appears to leave healthy legacy partial-write intents permanently retryable. The files above are unchanged between installed de484b93e and local main 5f4d5e8fe.

Expected behavior

A completed deep check should either produce a storage-verified receipt sufficient to discharge a healthy legacy partial-write obligation, or provide a bounded, explicit operational path to resolve such obligations. Presence alone should not be treated as proof. The background-heal status should make this non-converging state visible rather than presenting only a near-full queue.

Related but distinct: #7951 covers priority starvation; #7997 covers authoritatively absent deleted objects; #8132 covers an absent internal scanner object.