borg2 check: overview / tracking ticket

This ticket collects all cmd: check tickets (especially the ones in the 2.0.0b23 milestone) into one place: a summary of how borg2 check is supposed to work, plus a checkbox list of the concrete work items. Ticked items are already in master (commit linked); unticked items are still to do (open PR linked where one exists).

Tracked tickets: #9696 #9825 #9898 #9901 #9218 #9443 #8009 #7893 #2397 #8466 #8476 (b23), #8478 (b24), #6584 (2.1), plus the already-closed #9897 #9411 #8518 #8517 #8559 #8297 #1931 #7062.

How borg2 check is supposed to work

What changed vs. borg 1.x

borg 1.x check had a repository part (crc32-check all segment entries, always rebuild the repository index, run server-side for ssh repos) and an archives part (read all archives, check chunk presence, optionally --verify-data). Much of that design no longer applies:

• A borg2 repository is a borgstore store. There is no borg process on the server side (except for rest: served by borg serve --rest), so "run the check on the server" is replaced by something better: content-addressed verification.
• Pack files (packs/<sha256>) and chunk index fragments (index/<sha256>) are named by the sha256 of their content (fd177535). So "is this object intact?" is simply Store.hash(name) == name. The REST backend computes the hash server-side, so a full repository check of a remote repo transfers no pack data.
• The chunks index (chunk id → pack id, offset, size) lives in the repository and is the authoritative record of which chunks exist and where.
• There is no chunks_healthy anymore (#8559): an item's chunks list always holds the correct chunk ids, even when chunks are missing. Missing chunks are handled dynamically at read time (zeros or error), and if a later borg create re-produces a missing chunk, all old archives referencing it are healed instantly — no repeated check --repair runs.

Phase 1: repository check (cheap, read-only, no key needed)

1. Verify all index/ fragments via Store.hash (small, always checked in full).
2. If (and only if) the index is intact, verify all packs/ the same way.
3. Partial checks (--max-duration) resume from a persisted set of checked packs (cache/checked-packs, #9897), with periodic checkpoints. Packs recorded as corrupt are always re-verified, never skipped.
4. The read-only check never rebuilds the index: reading every pack is far too slow and expensive for a routine check. A corrupt index is reported and left for repair.

This phase is designed to be a routine, cron-able job: no key, no prompt, bounded time, meaningful exit code. Its findings (corrupt pack ids) should be persisted so a later repair can consume them instead of re-scanning everything (PR #9925).

Phase 2: archives check (client-side, needs the key)

1. Ask for the passphrase up front, not hours into the run (#1931).
2. Check the manifest (rebuild if missing/corrupt), the archives directory, each selected archive's metadata object and item metadata stream (robust resync on damage), and each item's file-content chunk presence against the (verified) chunks index.
3. --verify-data: additionally read, decrypt (AEAD-authenticate) and decompress every object — full cryptographic verification, detects malicious tampering too (for encrypted repos, against attackers without the key).
4. --find-lost-archives: full scan of all objects for archive metadata without an archives directory entry; with --repair, recreate the entry (possible until borg compact removes the data).

A normal (non-repair) archives check trusts the hash-verified index and does not rebuild it (#8476); only repair rebuilds from the actual packs.

Repair philosophy

• Only repair what is known to be broken. A repair should be able to consume the corrupt pack list a previous cheap check produced, instead of re-scanning all packs (#9696).
• Index repair rebuilds the index by scanning pack object headers only (seek over payloads, no full read) — but must not blindly trust the unauthenticated headers (#9901).
• Pack repair salvages objects out of a corrupt pack: every object that still decrypts (AEAD-authenticates) goes into a new pack, the corrupt pack is dropped, the index updated (#8476, #9696, refs #8572).
• Defect chunks are only deleted after a second read attempt also fails (transient errors — network, RAM — must not destroy data).
• Command split is open for discussion: borg check strictly read-only, all repairing/rebuilding in borg repair (with e.g. --index, --packs PACKID...) (#9696).

Design goals

• Data safety: never delete data on a single failed read; never let a rebuilt index claim objects that do not exist (fatal for borg create dedup, #8476); verify before trusting (#9901); a interrupted check/repair must leave a consistent state (Ctrl-C at safe boundaries, #7893).
• Usability for admins: routine checks need no key and no prompts; progress indication everywhere (#9443, #8009); a summary of findings and repairs at the end (#2397); reports organized around the question the admin actually has — "which files/archives are affected?" (#9218); a documented "what do I do now?" workflow after corruption (#9825); customizable output (#9411).
• Performance: server-side hashing (no transfer); no unnecessary index rebuilds (#8476); partial/time-boxed checks; reuse of still-fresh check results (--max-age, PR #9925); headers-only scans where possible; combine full-repo passes instead of reading everything twice (#8478, ex-#8518).

Work items

Repository check (read-only)

• Content-addressed read-only check: verify index/ and packs/ objects via Store.hash, server-side hashing for rest: repos; index first, packs only if the index is intact; no index rebuild in the read-only path (#9696, #8476, #8518) — 27d693c2 (PR #9794, with PRs #9783/#9789 for sha256 pack naming)
• Progress indicators (percent) for the index and pack verification loops (#9443, #8009) — 27d693c2 (PR #9794)
• Partial checks: persist a set of checked packs instead of a "last pack" marker, so packs added mid-cycle cannot be skipped (#9897) — bc8b3c07
• Re-verify packs recorded as corrupt instead of skipping them on resume (#9897) — 9543c141
• Checkpoint the checked-packs set periodically (30 min) during long runs (#9897) — a1b88e2d
• Keep pack check results after a completed check and add --max-age to skip packs with a fresh intact result — makes cheap periodic checks feed later repairs and avoids re-hashing everything on slow/cloud storage (#9696, #9898) — PR #9925 open
• Detect vanished packs: cross-check that every pack referenced from the chunks index is present in packs/ and has the expected size (#9898). Currently a completely missing pack is only noticed at read time (cbc94ae8 PackNotFound), by --verify-data, or by the --repair index rebuild.
• Cheap check mode for cloud repos: make full pack-sha256 verification optional; default (or an option) could be listing packs and checking presence + size against the index, avoiding the full data transfer (#9898)

Archives check

• Ask for the key passphrase early, before the (potentially very long) repository phase (#1931) — 2340e605
• Archive selection filters --first/--last/-a/--sort-by (#1663) and --older/--newer/--oldest/--newest (#7062) — b2654bc1
• --format support for the "Analyzing archive ..." output, validated up front, BORG_CHECK_FORMAT env var (#9411) — d7ffe748
• Non-repair archives check trusts the hash-verified in-repo chunks index — no slow index rebuild for a routine check (#8476, #8466) — 27d693c2 (earlier step: reuse the repo-part ChunkIndex, 7288f4f9, #8468)
• Remove chunks_healthy; self-healing archives, defined read-time behaviour for missing chunks, no all-zero patch chunks (#8559) — 6357f2eb
• --find-lost-archives (né --undelete-archives): scan for archive metadata lacking an archives directory entry, recreate entries with --repair — 682aedba, renamed in a48a8d2b
• Speed up --verify-data combined with --find-lost-archives: remember archive-metadata object ids during the verify pass and reuse them instead of a second full repo scan (#8478, ex-#8517 — milestone b24)
• --verify-data reads chunks in index (hash) order, i.e. randomly across packs; read pack-by-pack instead so each pack is fetched once, sequentially ("read data only once", idea from #8518, folded into #9696)

Repair

• Repository-level repair is not implemented yet — Repository.check(repair=True) currently just logs "repository repair not implemented" (#9696 "to be continued", #9825). This is the main open block of work:
  • Index repair: when index/ objects are corrupt (or the index provably lies), rebuild the index by scanning pack object headers only (range reads, skip payloads), not by reading full packs (#9696, #8476; code TODO at repository.py check(), refs #8572)
  • Pack repair (--repair-pack PACKID... or similar): salvage all AEAD-valid objects of a corrupt pack into a new pack, drop the corrupt pack, update the index — without a full scan of all other packs (#8476, #9696)
  • Let repair consume the persisted corrupt-pack list from previous read-only checks, so an admin's --repair doesn't redo the cron job's scanning work (#9696; groundwork in PR #9925)
  • Decide the command split: keep everything under borg check --repair vs. a separate read-only borg check + borg repair --index/--packs/... (#9696)
• Index rebuild must not blindly trust unauthenticated pack headers: a tampered header (wrong chunk id / sizes) currently goes straight into the rebuilt index (#9901). Mitigation ideas: authenticate on first real read (AEAD binds the id), or verify object ids cryptographically during rebuild for suspicious/mismatching entries.
• After check --repair, write the freshly built chunks index back to the repository instead of deleting it and forcing the next command into a slow rebuild (#8466 — the read-only path no longer rebuilds at all, but ArchiveChecker.finish() still deletes the index after repair)
• Repair progress indication: the --repair paths (verify_data delete pass, archive rebuild, future index/pack repair) need progress output like the read-only check has (#9443, #8009)
• Repair using redundant related repos: fetch chunks that are missing/corrupt here from a related repo (same key material / same chunk ids), e.g. borg check --repair --other-repo=...; borgstore cache vs. primary self-healing is a variant of the same idea (#6584 — milestone 2.1)

Reporting & UX

• Summarize findings at the end of check, and repairs actually done with --repair ("N problems found, M repaired") (#2397, dup #8192) — PR #9891 open
• Report missing chunks grouped as chunk → files → archives instead of one log line per (archive, file, chunk); avoids the hardlink-caused flood of repeated lines and answers "which files are affected?" directly (#9218) — PR #9965 open
• Ctrl-C handling: first Ctrl-C stops soon but cleanly at a safe boundary (saving checkpoint state, running finish()), second Ctrl-C aborts immediately (#7893) — PR #9966 open
• Document the post-corruption workflow (#9825): what --repair does and does not do, that re-running borg create heals reappearing chunks automatically (no second check --repair needed), how to get rid of permanently lost data (borg recreate --exclude ... the affected files), "considerations to fix your repo corruption" guide
• A way to acknowledge known-lost chunks so subsequent check/compact runs stop returning exit code 2 / error mails for damage the admin has already accepted (#9825)

Done / resolved by the borg2 design (no action needed)

• No more borg check requirement before prune/compact, so the --repair YES prompt no longer haunts cron jobs (#8297; BORG_CHECK_I_KNOW_WHAT_I_AM_DOING=YES still skips the prompt for actual repair runs)
• Segment-era tickets are obsolete: there are no segments, transactions or commits in a borgstore repo — #2454 (check with no commit), #8535/#8580 (segment refcount / index object count mismatch), #8070 (check a single segment; nearest borg2 equivalent is the planned per-pack repair above)
• "check --repair commit prompt" (#2398) — obsolete: borg2 has no repo transaction to commit; the equivalent admin need (see what check finds, then repair exactly that) is covered by the persisted check-results + repair-consumes-them items above

Milestone view (2.0.0b23)

Ticket	Topic	State
#9696	check redesign umbrella	read-only part done; repair part is the open core work
#8466	check --repair improvements	open (index caching after repair)
#8476	avoid unnecessary index rebuild	read-only part done; repair-index/-pack open
#9898	vanished packs	open (PR #9925 related)
#9901	index rebuild trusts pack headers	open
#9825	dealing with repo corruption	open (docs + acknowledge mechanism)
#9218	missing-chunk reporting	PR #9965 open
#2397	summarize findings/repairs	PR #9891 open
#7893	Ctrl-C	PR #9966 open
#9443	repo check progress	done for read-only (27d693c2); repair progress open
#8009	progress counters	borg1 wontfix; borg2 covered by #9443 items
#9897	set-of-checked-packs	done (bc8b3c07)
#9411	check --format	done (d7ffe748)

---

made by Claude Fable 5