InfluxDB 3 Enterprise: unclean shutdown leaves zero-byte V2 compaction checkpoints, causing a permanent unrecoverable startup loop (and silent orphaning of all history if deleted)

Summary

After an unclean host shutdown, InfluxDB 3 Enterprise can be left with zero-length V2 compaction checkpoint files (*.ptv2chk.lz4). On every subsequent start the server tries to lz4-decompress one, fails, retries with backoff, and exits 1 — permanently.

There is no recovery path that preserves data. The only way to make the server boot is to delete the checkpoints, and doing that resets the compactor to loaded_checkpoint_sequence=0, which orphans every compacted parquet file on disk. The server then comes up perfectly healthy and returns 0 rows for every measurement while hundreds of MB of parquet sit unreferenced.

I think there are two separate defects here:

1. Durability — checkpoints do not appear to be written atomically (temp file → fsync → rename) before being treated as live. A zero-length checkpoint should never be observable on disk.
2. Recovery — a truncated/undecodable checkpoint is a hard, infinitely-retried error. There is no fallback to an older valid checkpoint, and no way to rebuild compaction state from the manifest/parquet set.

Environment

• InfluxDB 3 Enterprise (At-Home license), via the erik73/hassio-addons InfluxDB3 app v1.1.3, image ghcr.io/erik73/influxdb3/amd64
• Home Assistant OS 18.2, amd64 / generic-x86-64
• Data dir on ext4 mounted with commit=30 (Home Assistant OS default for /mnt/data), so an unclean shutdown can lose up to 30s of un-fsync'd writes.

What happened

The host rebooted uncleanly at 2026-09-10 22:02 UTC. That left 12 zero-byte files, every one stamped 22:02:xx:

influxdb3/cluster0/cv2/checkpoints/00000000000000219446.ptv2chk.lz4   <- fatal
influxdb3/cluster0/cv2/checkpoints/00000000000000219447.ptv2chk.lz4   <- fatal
influxdb3/cluster0/cv2/checkpoints/00000000000000219448.ptv2chk.lz4   <- fatal
influxdb3/<id>/cluster0/cv2/windows/2026-09-10/p0/run_sets/*.idx       (2)
influxdb3/<id>/cluster0/cv2/windows/2026-09-10/p0/files/*.pt           (2)
influxdb3/<id>/0/pt_wal/*.pt                                           (4)
influxdb3/<id>/0/gen0/*.pt                                             (2)
influxdb3/0/pt_snapshots/*.ptsnap                                      (2)

Critically, all three checkpoints were zero-length — there was no older intact one to fall back to.

Startup then loops forever (1/2/4/8/10s backoff, then exit 1):

INFO  influxdb3::commands::serve: temp catalog loaded (484 ms)
INFO  influxdb3::commands::serve: catalog initialized with uuid: '<redacted>'
INFO  influxdb3::commands::serve: licensing initialized (724 ms)
WARN  influxdb3_lib::commands::serve: server initialization failed with retryable error, retrying
      error=failed to initialize write buffer: Failed to create V2 compactor:
      internal combined mode error: failed to start standby compaction consumer:
      Internal error: Failed to decompress V2 checkpoint: expected another byte, found none
      retry_in_secs=1.0
...
Serve command failed: failed to initialize write buffer: Failed to create V2 compactor:
internal combined mode error: failed to start standby compaction consumer:
Internal error: Failed to decompress V2 checkpoint: expected another byte, found none

Note the catalog itself loads fine — only the compactor is wedged.

The data-loss trap

Deleting the empty checkpoints makes the server start, which looks like a fix. It is not:

WARN influxdb3_pacha_tree::enterprise::compactor::compactor_loop::checkpoint:
     compactor delete queue is newer than the loaded checkpoint (likely a restore);
     discarding it to avoid deleting files the restored checkpoint references
     queue_checkpoint_sequence=219448 loaded_checkpoint_sequence=0

loaded_checkpoint_sequence=0. The server reports healthy, /health returns OK, 738MB of parquet is still on disk — and SELECT count(*) returns 0 for every measurement. The data is intact but unreachable.

Removing the checkpoints also exposed a second truncation fault behind them, from the zero-byte WAL/snapshot/gen0 files:

WARN influxdb3_lib::commands::serve: server initialization failed with retryable error, retrying
     error=failed to initialize write buffer: PachaTree error: Internal error: Manifest too small for magic

So the fragility isn't limited to checkpoints — zero-length segments elsewhere in the tree are also treated as hard errors rather than being skipped.

What actually recovered the data

A restore of the whole data directory from a backup predating the reboot. History came back in full (384,996 rows in a single measurement, Aug 24 → present). Everything written between the backup and the crash was unrecoverable.

Suggested fixes

• Write checkpoints atomically: temp file → fsync → rename. A zero-length checkpoint should never be observable.
• On a truncated/empty/undecodable checkpoint, fall back to the newest valid older checkpoint rather than failing.
• If no valid checkpoint exists, fail loudly and distinctly, and do not leave "delete the checkpoint files" as the only apparent remedy — it silently orphans all compacted history. An explicit rebuild-compaction-state-from-manifest path would turn this from data loss into an inconvenience.
• Skip or truncate zero-length WAL / snapshot / gen0 segments on replay instead of erroring.