Multi-level background writes and backfills are untracked and can be lost at shutdown

Summary

In the multi-level backend, backfills and writes to levels above L0 are tokio::spawned without retaining a handle. At shutdown the server waits only for client connections to drain, then drops the runtime — cancelling any storage task still in flight. As a result sccache --stop-server does not guarantee that accepted cache writes reached the remote backend.

Code

v0.17.0 (c037e11). Detached writes and backfills, no JoinHandle retained — src/cache/multilevel.rs L609 and L678.

Shutdown — src/server.rs L733-L747:

const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(10);
// ...
// This `wait` future will resolve once all instances of `SccacheService` have been dropped.
runtime.block_on(async { time::timeout(SHUTDOWN_TIMEOUT, wait).await })?;

wait tracks SccacheService references (client requests), not spawned storage tasks, so a backfill or L1+ write that has been accepted but not yet completed is simply dropped when the runtime goes away.

Measured behaviour

From a CI benchmark using SCCACHE_MULTILEVEL_CHAIN=disk,s3 (with SCCACHE_CLIENT_SIDE=1, ephemeral runner):

• Population run: 5,447 local writes, 5,197 completed S3 writes — 95.41% — when the job ended.
• The next run against the same namespace reported exactly 250 misses, matching the 250 writes that had not completed.

This is a single pair of runs from an internal benchmark rather than a controlled repeat, but the exact match between unfinished writes and subsequent misses is hard to explain another way. Happy to gather more detail if it would help.

Related but distinct

#204 covers idle shutdown terminating in-progress compile jobs. This issue is about detached storage tasks, which are not tracked at all.

Suggested fix

Track spawned storage tasks (e.g. JoinSet / TaskTracker) and await them during shutdown under an explicit, configurable drain deadline, separate from the client-connection timeout. Reporting the number of outstanding or abandoned writes in --show-stats would also let CI detect the situation instead of discovering it as misses on the next run.