`RemoveLease` hogging `cm.mu`

What is the issue?

tldr RemoveLease scans every snapshot under cm.mu for each pruned result, so a routine GC pass on a large cache takes ~17 minutes. Meanwhile, disk-pressure GC waits behind it on gcmu (observed: 21 minutes) while free space falls below minFreeSpace. A ~20-line reverse index (same sort of fix as #13888) makes it O(chain depth)

Summary

snapshotManager.RemoveLease removes one lease. To do this, it examines each entry of snapshotOwnerLeases. It holds the global cm.mu during the examination.

GC calls RemoveLease one time for each pruned result. Thus one prune pass does O(pruned × all-snapshots) work under cm.mu. Live sessions also need cm.mu for snapshot metadata. Examples are AttachLease and the read operations during a pull.

On an engine with a large cache, one usual GC pass can continue for more than ten minutes. The functions gc() and gcIfDiskPressure() use one lock, Server.gcmu, for the full pass. The lock has no preemption. Thus the disk-pressure GC must wait behind the slow usual pass. During this wait, the free space decreases below minFreeSpace.

We found the cause of this problem on a production CI cluster with v0.21.8. The applicable code is byte-identical in v0.21.9, v1.0.0-beta.11, and current main. Thus the defect is in the 1.0 RC. The defect is the same type as the defect that PR #13888 repaired in dagql. A derived reverse index seems like it would solve the problem here.

Production evidence

Environment: a production CI cluster with 4 engine cells. Each cell has a 2 TiB local cache. The engine is v0.21.8 with no local changes. The configuration sets minFreeSpace = "400GB". During the busy hours, 15–20 clients operate at the same time. The dagql cache of each cell holds 250k–390k entries.

We made one goroutine dump during a minFreeSpace breach.

• The disk-pressure GC goroutine was blocked for 21 minutes. It waited for Server.gcmu:

  复制

  goroutine 73372299 [sync.Mutex.Lock, 21 minutes]:
    ...
    dagger/engine/server.(*Server).gcIfDiskPressure  (engine/server/gc.go:197 @ v0.21.8)

• The usual GC pass held gcmu. It was [runnable], not deadlocked. It was in the lease cleanup for one result:

  复制

  goroutine 72298129 [runnable]:
    dagger/engine/snapshots.(*snapshotManager).RemoveLease
    dagger/dagql.(*Cache).resultSnapshotLeaseCleanup   (dagql/cache.go:989 @ v0.21.8)
    dagger/dagql.removePersistedEdge                   (dagql/cache_prune.go:97)
    dagger/dagql.(*Cache).Prune
    dagger/engine/server.(*Server).gcLocked
    dagger/engine/server.(*Server).gc

We measured that pass with the dagql pruned result log-line timestamps. The pass pruned 708 results. It released 54.2 GiB. It ran for 17 minutes. That is approximately 53 MB/s while the pass ran. In the 53 minutes before the pass, the engine released zero bytes.

The engine releases space only while a pass runs. The duty cycle was 17 minutes in a 70-minute window. Thus the average release rate was approximately 13 MB/s. The clients wrote approximately 45 MB/s without stop. The average release rate is less than the write rate. Thus the free space decreases below minFreeSpace during each busy period. The free space increases again only when the sessions close. In a 3-day window, each cell had a breach each day. The free space decreased to 0–60 GiB in the worst periods. The engines had a restart approximately 19 hours before the dump. Thus this is the steady-state behavior, not a blocked process.

We counted the lock-wait sites in the same dumps. The top wait sites on the snapshot manager mutex are RemoveLease, GetBySnapshotID, and SnapshotRecordMetadata. Thus the GC and the live sessions compete for the same lock.

The pass is slow because of serialization, not because of the CPU. The node CPU idle value stayed at 80–96%. The iowait value stayed below 5% during the breach periods.

Mechanism

RemoveLease (engine/snapshots/persistent_metadata.go:198–221 @ v1.0.0-beta.11; byte-identical in v0.21.8, v0.21.9, and main @ b2d75d552):

cm.ownerLeaseLocker.Lock(leaseID)
defer cm.ownerLeaseLocker.Unlock(leaseID)

err := cm.LeaseManager.Delete(ctx, leases.Lease{ID: leaseID})  // containerd bolt txn
if err != nil && !cerrdefs.IsNotFound(err) {
        return pkgerrors.Wrapf(err, "delete owner lease %s", leaseID)
}

cm.mu.Lock()
for snapshotID, leaseIDs := range cm.snapshotOwnerLeases {   // scans all snapshots
        delete(leaseIDs, leaseID)
        if len(leaseIDs) == 0 {
                delete(cm.snapshotOwnerLeases, snapshotID)
        }
}
cm.mu.Unlock()

Three defects have a combined effect:

1. Each removal scans all snapshots under cm.mu. snapshotOwnerLeases is map[snapshotID]map[leaseID]struct{}. The key direction is not correct for this lookup. Thus the removal of one lease examines the lease set of each snapshot. GC calls RemoveLease one time for each pruned result (708 in the measured pass). DeleteStaleDaggerOwnerLeases calls it one time for each stale lease. Thus that function is O(stale × all-snapshots). The live sessions need the same cm.mu for the snapshot traffic (AttachLease; the read at engine/snapshots/pull.go:377–378).

2. Each pruned result causes one containerd bolt transaction (fsync). Each RemoveLease does one LeaseManager.Delete. Thus a pass pays one disk-transaction cost for each result, in addition to the scans.

3. gcmu is held for the full pass, with no preemption. gc() and gcIfDiskPressure() get srv.gcmu with defer Unlock (engine/server/gc.go). In v0.21.9 and subsequent versions, the metadata prune is a third user of the lock. The emergency path has no priority over the routine path. This caused the 21-minute wait while the disk became full.

Precedent

• #13888 (merged 2026-08-17) repaired the same defect type in dagql. The result removal "repeatedly scanned … while holding egraphMu". The work "grew superlinearly". The repair added derived inverse indexes. This issue requests the #13888 repair for engine/snapshots.
• The v0.20.8 cache.Size() close-walk was the same defect type: an O(everything) walk under a busy lock on a per-item path. This is the third occurrence. Thus a general search for this pattern is possibly useful.

Proposed repair

Add the inverse index to snapshotManager:

// ownerLeaseSnapshots is the inverse of snapshotOwnerLeases:
// leaseID -> set of snapshotIDs that lease owns. Derived state,
// maintained under cm.mu wherever snapshotOwnerLeases is mutated.
ownerLeaseSnapshots map[string]map[string]struct{}

With this index, RemoveLease touches only the snapshots of the applicable lease. The cost becomes O(chain depth), not O(all snapshots):

cm.mu.Lock()
for snapshotID := range cm.ownerLeaseSnapshots[leaseID] {
        leaseIDs := cm.snapshotOwnerLeases[snapshotID]
        delete(leaseIDs, leaseID)
        if len(leaseIDs) == 0 {
                delete(cm.snapshotOwnerLeases, snapshotID)
        }
}
delete(cm.ownerLeaseSnapshots, leaseID)
cm.mu.Unlock()

• All four write sites of snapshotOwnerLeases (manager.go:116, manager_test.go:323, persistent_metadata.go:87–89, 189–192) get the mirror write, in the same cm.mu critical section.
• Invariant: leaseID ∈ snapshotOwnerLeases[s] ⟺ s ∈ ownerLeaseSnapshots[leaseID]. Readers of the forward map are unchanged.
• Not persisted: no schema change, no rebuild at start. Lock order unchanged.
• DeleteStaleDaggerOwnerLeases gets the same speedup with no extra code.
• Size: 17 non-test lines in 2 files (manager.go, persistent_metadata.go). The patch is written and applies to current main with no conflict. We can send the PR, plus a 0.21 backport (precedent: #13884 / #13886).
• Tests: TestSnapshotManagerOwnerLeaseIndexConsistency (invariant check after attach and remove), TestSnapshotManagerOwnerLeaseIndexConcurrent (concurrent attach/remove under go test -race ./engine/snapshots).

Measured

BenchmarkRemoveLease on the patch, one lease removal, Go 1.26, ns/op:

The "before" numbers understate the cost by up to ~2×, because the benchmark map shrinks while the benchmark runs. The production baseline is the pass above: 708 results in 17 minutes.

Environment

• Engine: v0.21.8. The applicable code is the same at v0.21.9, v1.0.0-beta.11, and main @ b2d75d552 (2026-09-10).
• Deployment: a production CI cluster; 4 engine cells; each cell has a 2 TiB local cache; minFreeSpace = "400GB"; 15–20 clients at the same time at peak; a dagql cache of 250k–390k entries in each cell.

Dagger version

Engine: v0.21.8