Running out of disk space resulted in unrecoverable WAL error

What did you do?

Ran Prometheus

What did you expect to see?

Out-of-disk errors should cause temporary loss of data but not a fatal condition I have to log in and delete the WAL to fix.

What did you see instead? Under which circumstances?

The disk filled up, and the first report of this was from the WAL:

ts=2023-10-23T23:11:06.646Z caller=scrape.go:1359 level=error component="scrape manager" scrape_pool=kube-system/cadvisor target=https://kubernetes.default.svc.cluster.local.:443/api/v1/nodes/gke-dev-redacted/proxy/metrics/cadvisor msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/data/wal/00006644: no space left on device"
ts=2023-10-23T23:11:06.704Z caller=scrape.go:1359 level=error component="scrape manager" scrape_pool=kubernetes-pods target=http://10.132.15.32:9150/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/data/wal/00006644: no space left on device"
...
ts=2023-10-23T23:11:08.474Z caller=scrape.go:1359 level=error component="scrape manager" scrape_pool=kube-system/kubelet target=https://kubernetes.default.svc.cluster.local.:443/api/v1/nodes/gke-dev-redacted/proxy/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/data/wal/00006644: no space left on device"

After this point apparently something was cleaned up on the disk (I didn't do it), because those errors stop. Then remote-write reports it can't read that segment:

ts=2023-10-23T23:13:40.683Z caller=dedupe.go:112 component=remote level=error remote_name=some_remote_write url=https://example.com:1234/prometheus msg="error tailing WAL" err="segment 6644: unexpected full record: unexpected full record"

Now we get a flood of these, various ref numbers, presumably because those series were in the lost WAL segment:

ts=2023-10-23T23:14:06.048Z caller=dedupe.go:112 component=remote level=info remote_name=some_remote_write url=https://example.com:1234/prometheus msg="Dropped sample for series that was not explicitly dropped via relabelling" ref=56218783

Subsequently, the WAL cannot be truncated:

ts=2023-10-24T01:02:11.908Z caller=checkpoint.go:101 level=info component=tsdb msg="Creating checkpoint" from_segment=6625 to_segment=6656 mint=1698105600000
ts=2023-10-24T01:03:02.735Z caller=db.go:984 level=error component=tsdb msg="compaction failed" err="WAL truncation in Compact: create checkpoint: read segments: corruption in segment /prometheus/data/wal/00006644 at 61899498: unexpected full record"

This error repeats every 2 hours. The WAL continues to grow.

System information

Linux 5.15.107+ x86_64

Prometheus version

I built this Prometheus myself; it's a few commits on from 624b973eb.

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

See log excerpts above.