Restored DB fails integrity_check when the initial snapshot build is slow relative to write activity (0.5.12, GCS replica)

Summary

After upgrading a production deployment from v0.3.13 to v0.5.12, the LTX backup chain written by 0.5.12 restores into a database that fails PRAGMA integrity_check with ~100 errors. The source database is healthy, and the pre-upgrade 0.3.13 generation chain for the same database restores clean (verified independently twice). Three sibling deployments with small (MB-scale) databases migrated identically and their restores verify clean.

The database is 22 GB on a throughput-capped disk (GCE pd-standard, ~12 MB/s), so the initial L0 snapshot takes ~25 minutes to build while the application writes continuously (~24 MB/min of WAL). The relevant variable appears to be that long snapshot window racing checkpoint activity, not database size. To reproduce on fast hardware, throttle I/O (e.g. cgroup io.max) while writing during the initial sync.

No ERROR-level log line was emitted at any point; replication looked entirely healthy from the logs.

Environment

• Litestream v0.5.12 (litestream-0.5.12-linux-x86_64.tar.gz), upgraded from v0.3.13
• Debian 12, GCE e2-medium (2 shared vCPU, 4 GB RAM)
• Boot disk: pd-standard (throughput-capped ~12 MB/s at 100 GB)
• Database: SQLite, WAL mode, 22 GB (~5.74 M pages), continuous writes (~24 MB/min WAL growth; workload rewrites ~150 KB blobs per request)
• Replica: GCS

Config

snapshot:
  interval: 1h
  retention: 24h

dbs:
  - path: "/app/<database>.db"
    replica:
      type: gs
      bucket: "<bucket>"
      path: "<prefix>"
      sync-interval: "10s"

Timeline (UTC, single run from a clean state dir)

(The journal is silent between startup and the first replica sync; the build/read phases above were observed via /proc/<pid>/fd read offsets and the growth of the ltx/0/*.ltx.tmp staging files.)

Restore result

On a separate VM (read-only storage scope), the restore exits successfully, but the result fails integrity_check:

$ litestream restore -o restored.db "gs://<bucket>/<prefix>"   # 0.5.12, exit 0
$ sqlite3 restored.db "PRAGMA integrity_check;"
*** in database main ***
On tree page 7940 cell 0: overflow list length is 9 but should be 7
On tree page 5398900 cell 0: invalid page number 5742718
On tree page 4909460 cell 0: invalid page number 5742735
On tree page 12581 cell 0: 2nd reference to page 5731965
On tree page 483950 cell 0: Rowid 8969123161 out of order
On tree page 1490325 cell 0: invalid page number 218103808
... (~100 errors total)

Notably, many of the invalid/duplicated page references cluster near the end of the page range (pages 57xxxxx of ~5.74 M) — consistent with pages that were being appended/relocated while the database grew during the snapshot window.

What was ruled out

• Source corruption: the pre-upgrade 0.3.13 generation chain (final state written minutes before the upgrade) restores with integrity_check = ok. Verified twice, independently: 0.3.13 binary restoring from gcs:// on a separate VM, and locally from a file:// replica of the downloaded bucket contents. The application also operated normally on the source DB throughout.
• Small-DB regression: three sibling deployments (databases of a few MB) upgraded 0.3.13 → 0.5.12 the same day, same role/config; their restores pass integrity_check.
• Dirty upgrade state: the run above started from a deleted state directory (fresh snapshot from scratch).

Hypothesis

The initial L0 snapshot appears to read the main database file over a long window (~25 min here: 22 GB at the disk's ~12 MB/s cap) while checkpoints move pages into it — the forced TRUNCATE checkpoint at snapshot finalization, and passive checkpoints during the second build. If the snapshot read is not isolated against checkpoint writes, the resulting page set is torn (mixed old/new pages), matching the observed signature. Possibly related: #1083 (silent replication failure on WAL reuse), #800 (GCS EOF on 0.3→0.5 upgrade).

Happy to provide full journald logs, the corrupt LTX chain (retained in the bucket), or run instrumented builds — this is reproducible infrastructure for us.