Lock server: every acquire write-locks all 64 guard shards and scans the whole guard table, so a node with leftover guards falls into a lock convoy it cannot leave

Describe the bug

While chasing #7962 (one node's lock endpoint answering slower than the 3 s lock-RPC deadline for hours) we found the cause on the server side. It is not the disk, and it is not one erasure set: /mnt/rustfs11 is just the name of each node's lock endpoint. init_lock_clients de-duplicates endpoints by host_port() with a plain insert, so each host keeps its last endpoint, that URL names the host's single RemoteClient, and each node serves all of it from one LocalClient — one guard table per node. The "slow endpoint" in #7962 (and in #7363 before it) is node3's whole lock server. The cause is that server's acquire path.

Every LocalClient::acquire_lock first calls reclaim_expired_guards_for_resource, and extract_expired_guards implements that by taking the write lock of all 64 guard shards in turn and scanning every entry (guards are sharded by the whole LockId across 64 shards, so one resource's guards can be anywhere). The contended-acquire retry path sweeps again. So every acquire on a node is serialized behind 64 exclusive async locks, and its cost grows with the size of the guard table.

To Reproduce

Normally the table is tiny and nobody notices. Live top/locks?count=10000 during the storm, three samples 10 s apart:

node	entries	of which ttl_secs = 0
node1	48 / 23 / 36	0
node2	35 / 17 / 30	0
node4	13 / 7 / 11	0
node3 (the slow one)	2,840 / 3,537 / 4,181 (4,709 a few minutes later)	761 / 1,390 / 2,060

A second capture an hour later shows the same picture — node3 at 3,475 / 4,230 / 4,879 entries (1,513 / 2,075 / 2,660 expired), the other three at 14–59 with none expired. On node3 the entries are owned almost entirely by the three other nodes (2,092 / 1,889 / 892 in one sample, 6 by node3 itself), spread evenly over ages 0–60 s: they are the grants that arrived after the caller's 3 s RPC deadline (Could not release every remote lock granted after its caller timed out), whose releases then time out the same way. Each one stays for its 30 s TTL and then, expired, until the 60 s reaper — about 70 new ones per second, a sawtooth between ~800 and ~4,700.

That closes the loop we could only guess at in the report: slow acquire → caller times out → late grant nobody releases → bigger table → every acquire slower. Once a node tips over it cannot recover by itself while the peers keep retrying at full rate, which matches what we see (rustfs CPU on node3 is ~15 of 80 cores — it is serialized, not busy).

Microbenchmark on the crate (8 workers, 4,000 resident guards of unrelated resources, 256 tasks × 20 acquire/release pairs of distinct resources): 6,655 pairs/s on main, 351,979 pairs/s with the change below. 6.6 k/s on an idle laptop is the same order as the ~4 k lock RPCs/s the three peers send this node, before counting releases and refreshes.

Expected behavior

An acquire should cost the same whatever else the node's guard table holds, and should not exclude acquires of unrelated resources. Proposed fix (happy to open the PR): shard guards by lock_id.resource instead of the whole LockId, so a resource's guards share one shard; on the acquire path look only at that shard, under the read lock, and take the write lock only if an expired guard was found; let the reaper skip shards with nothing expired. Includes a deterministic test (acquire must succeed while every other shard is write-locked). cargo test -p rustfs-lock, fmt and clippy are clean.

Environment

• RustFS 1.0.0, git d47f54bfb2f39f48bd1adda334bd27e151fe85b8 (code identical on main as of d9c3eeea0); 4 nodes × 12 NVMe, 3 erasure sets, EC 12+4; lock RPC timeout 3000 ms, lock TTL 30 s, RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30.
• Workload: five JuiceFS filesystems, continuous small-object write + delete (~4 k lock RPCs/s into each node).

Additional context

The same per-acquire sweep is already in 1.0.0-rc.5 (identical code at 40a2470f), so it also fits the two storms we reported in #7363. We also suspect — without proof — that it is the mechanism behind an earlier incident on rc.5 in which one node accumulated ~3,100 in-flight inbound NodeService/Lock requests and was OOM-killed: a serialized acquire path with a growing table is exactly how inbound lock RPCs would pile up faster than they drain. This does not replace the asks in #7962 (request-level backoff, not orphaning late grants) — those are what create the residents in the first place — but it removes the amplifier.