Log pages have no checksum: a power loss can leave a transaction partially applied
Problem
Recovery of the log file (WalIndexService.RestoreIndex) trusts what it finds: a page whose header says IsConfirmed commits its transaction, all pages read before it with the same TransactionID are applied, and blank pages are skipped silently. Log pages carry no checksum, and nothing records how many pages a transaction has.
That is safe only while every log page of a transaction is on the device before its confirm page is. When pages are still in the operating system's cache, a power loss or OS crash can persist them in any order, so a confirm page can land while an earlier page of the same transaction did not (it reads back blank or stale) or landed torn. Recovery then applies part of a transaction and reports nothing.
Who is exposed:
DurableCommits = false(the opt-out added with #2818 in #2918) and every release before 6.0: all recent commits are in that state until the next checkpoint.- The durable default: only the one transaction in flight during the power loss, because its pages and its confirm page share a single sync. The commit was never acknowledged, but a partial application is still worse than losing it.
- Storage that ignores or rejects a sync (#2242) behaves like the first case.
SQLite closes exactly this gap with salted, cumulative checksums on WAL frames: recovery stops at the first frame that does not verify, so a torn tail costs whole commits, never half of one. That is why its synchronous=NORMAL is "may lose recent commits" while our equivalent is "may lose recent commits or, rarely, leave one half applied".
What a valid fix looks like
- Recovery applies a transaction only if all of its log pages are present and intact; otherwise it discards that transaction and everything after it in the log (later transactions may depend on it), then truncates the log there.
- "Intact" is verifiable: a checksum per log page (covering the page content and header, salted per log generation so stale pages from an earlier use of the same log region cannot verify), and the confirm page records how many pages the transaction wrote, so a missing page is detected even when the missing position reads as a valid older page.
- The outcome is visible: recovery that drops a tail says so (e.g. a
$databasefield or a log message), instead of failing silently or succeeding silently. - Data pages written by a checkpoint are out of scope here; they are synced and redoable from the log. A data-page checksum would be a separate issue.
- Files written by older versions still open. Whatever marks "this log has checksums" must let an old log (no checksums) recover with today's rules, and must make an older LiteDB refuse or safely ignore a new log rather than misread it. Encrypted files work the same way (checksum over the plaintext page or over the ciphertext — pick one and test both).
- Cost stays small next to a device sync: a CRC32C/xxHash over 8 KB per page; measure single inserts and
InsertBulkbefore and after.
Design notes (not decisions)
- The 32-byte page header has one unused byte (31); a 4-byte checksum does not fit without moving a field or using the page tail, so this is a file-format change and belongs with a format revision (the vector index already bumped the version once).
- Alternative that leaves data pages untouched: keep the checksum outside the page, in a small per-transaction commit record appended to the log (page count + checksum of each page or one cumulative checksum). Only the log format changes, and the log is empty after every clean close, which makes the compatibility story much easier.
- Tests need a storage double that drops/reorders/tears unsynced pages at a simulated power loss;
ScriptedFlushFileand friends from #2918 are a starting point.
Acceptance
- A red test first: write N transactions, simulate a power loss that drops one middle page of the last confirmed transaction, reopen → today half the transaction is visible; after the fix none of it is, and all earlier transactions are.
- Same for a torn (partially written) page and for a stale page from a previous log generation at that position.
- Once this lands, the
DurableCommitsdocs can drop "or, rarely, leave them partially applied".
Refs #2818, #2918, #2242.
Source: litedb-org/LiteDB