Backup cancellation can leave SST objects without returned file metadata

What's wrong?

The ordinary Backup gRPC service can continue saving an already queued in-memory backup file after the client response stream has been cancelled or disconnected. The save worker writes the backup SST object to external storage first, then builds the corresponding brpb::File descriptor and sends it back through BackupResponse.files.

If the response channel is already disconnected when the save worker reaches tx.unbounded_send(response), the send fails and the worker returns. At that point the external SST object can already exist, but the metadata that names the object and records its checksum, key range, CF, IV, size, and byte counts has not been delivered to the backup coordinator.

This is a backup-attempt external-state consistency issue. It is not a claim that live TiKV key-value data is corrupted, and it is not a claim that successfully returned backup files are corrupt.

What version of TiKV are you using?

TiKV 8.5.x / current Backup service implementation, around:

• components/backup/src/service.rs
• components/backup/src/endpoint.rs
• components/backup/src/writer.rs
• components/external_storage/src/local.rs

What operating system and CPU are you using?

Linux x86_64.

Steps to reproduce

A minimal source-level scenario is:

1. Start a normal Backup request through the gRPC service.
2. Let a scan worker create an InMemBackupFiles value and hand it to the save-worker queue.
3. Let the client response stream fail or disconnect so the service-side response forwarding path treats the request as cancelled.
4. Let the queued save worker continue processing the already queued InMemBackupFiles value.
5. Let the writer save the backup SST to the configured external storage.
6. Observe the save worker reaching the response-send step after the external write has completed.

The important source-level transition is:

Backup response stream fails
  -> Service::backup response forwarding returns an error
  -> request cancel flag is set
  -> an InMemBackupFiles item is already queued for a save worker
  -> save_backup_file_worker() calls msg.files.save(&storage).await
  -> Writer::save_and_build_file() writes the SST object
  -> brpb::File metadata is created after the write
  -> tx.unbounded_send(response) reports a disconnected response channel
  -> the save worker returns without deleting or otherwise publishing the file metadata

What did you expect?

Cancelling a Backup request should not leave externally visible backup SST objects that are outside the returned backup file set and outside a documented cleanup or recovery ledger.

Expected safe outcomes include one of the following:

• the save worker observes request cancellation before writing queued in-memory backup files;
• already queued save work is drained to a safe point while the service still owns the response or cleanup decision;
• if an SST has already been written, its brpb::File descriptor is persisted somewhere that survives client stream cancellation;
• if response delivery is disconnected after a write, the just-written object is removed before the worker returns;
• backup files are written under an attempt-scoped staging prefix that is promoted only after metadata publication succeeds, and removed on cancellation.

What did happened?

The current control flow appears to let the external SST write and the metadata publication step be cancelled independently.

In components/backup/src/service.rs, Service::backup creates an internal response channel, schedules the backup task, and forwards responses to the gRPC sink. The forwarding future calls sink.send_all(&mut s).await and then sink.close().await. If that forwarding future returns an error, Service::backup stores true in the request cancellation flag.

In components/backup/src/endpoint.rs, the endpoint creates a bounded save-worker queue and spawns scan workers separately from save_backup_file_worker(). Scan workers check the request cancellation flag between backup ranges, but already queued InMemBackupFiles values are owned by the save-worker stage.

The save worker receives each queued item and, when it contains data to flush, calls:

msg.files.save(&storage).await

Only after that save completes does the worker construct a BackupResponse, store the returned brpb::File descriptors into response.files, and call:

tx.unbounded_send(response)

The writer save path in components/backup/src/writer.rs writes the external SST before returning the brpb::File descriptor. Writer::save_and_build_file chooses the final SST name, wraps the reader for encryption/checksum calculation, calls storage.write(...).await, then computes the checksum and builds the brpb::File.

This means the external object can become visible before the only normal metadata publication path has succeeded. If the response channel is disconnected at that point, save_backup_file_worker() logs the send failure and returns when e.is_disconnected() is true. It does not delete the just-written SST object, write a side manifest, retry publication through a different owner, or hand the file descriptor to a cancellation-surviving cleanup path.

The backup file naming path also appears to make this worse for cancelled attempts. backup_file_name() uses a timestamped generated name so a repeated request after connection reset does not overwrite the previous attempt's objects. That protects against overwrite, but it also means a retry naturally writes a different object name rather than reusing or discovering the object from the cancelled attempt.

The visible contract for consumers is that backup responses carry the file descriptors used to locate and validate the generated SSTs. If a file is written but its descriptor is never returned or durably recorded elsewhere, the backup attempt can leave an orphaned or unindexed backup artifact in external storage.

This is a cancellation-correctness issue rather than a storage write bug:

• the save operation has an externally visible side effect;
• client cancellation can close the only response metadata path;
• queued save work can still publish an SST after that response path is gone;
• the worker has no commit/cleanup step that reconnects the written object to a surviving metadata ledger.

The bounded impact is external-state consistency, cleanup, storage accounting, and backup-attempt atomicity. It does not imply corruption of TiKV's live data or corruption of backup SSTs whose metadata was successfully returned.

Possible fix direction

The backup path should make the attempt commit boundary explicit instead of relying on a cancellable response stream as the only owner of file metadata. Possible repair directions include:

• pass the request cancellation state into save_backup_file_worker() and check it before turning queued in-memory writers into external SST objects;
• on response-stream failure, stop the scan stage and then drain or join the save stage to a point where each written file is either published or cleaned;
• if tx.unbounded_send(response) fails after a successful save, delete the just-written SST object before returning;
• write a durable per-attempt manifest before or atomically with external SST publication, so file descriptors survive client stream cancellation;
• use an attempt-scoped staging namespace and promote files into the final backup namespace only after the coordinator has received or persisted their descriptors;
• document and implement a backup-coordinator cleanup rule for failed attempt prefixes if the intended contract is that cancellation may leave temporary objects.