Compactions cause the configured storage.tsdb.retention.size to be exceeded (and risk of running out of disk space)

Excerpt / TL;DR

The set limit of storage to be used (storage.tsdb.retention.size) is violated during compaction which poses a the risk of running out of disk space. The reason is that Prometheus does not take into account the required space for the temporary blocks. See Proposal section for two suggested approaches to fix this and allow Prometheus to reliably avoid exceeding it's allowance.

Issue

The storage.tsdb.retention.size option (https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects) was a great addition to allow keeping metrics in relation to the available storage. While this can be a provisioned volume from some cloud provider or a LVM volume or just the amount of space you can give to Prometheus and then have it make the most of the available space (https://www.robustperception.io/configuring-prometheus-storage-retention/).

While there were some issues with WAL files and head_chunks not counted against the configured limit, those are now all taken into account when determining the current storage size thus reducing confusion for users and the risk of running out of disk space. Since checkpoints and first level of compactions happen over relatively small time intervals of two hours the peak is relatively small and usually just about keeping some sensible head room for - just as written in the documentation:

So the minimum requirement for the disk is the peak space taken by the wal (the WAL and Checkpoint) and chunks_head (m-mapped Head chunks) directory combined (peaks every 2 hours)

There also is good documentation on how to calculate the space requirements regarding the amount of samples ingested per time interval and their average size via built-in metrics:

• https://www.robustperception.io/how-much-disk-space-do-prometheus-blocks-use/
• https://www.robustperception.io/how-much-space-does-the-wal-take-up/

Even though this all helps to anticipate how man samples and time series can be stored and over which retention times using the available space, more critical to the risk of running out of space are the compactions over longer periods covering up to 31 days of samples by default (https://prometheus.io/docs/prometheus/latest/storage/#compaction). The leveled compaction time ranges and the compaction is nicely explained at leveled compaction: https://ganeshvernekar.com/blog/prometheus-tsdb-compaction-and-retention/#condition-2-preset-time-ranges.

And there also is documentation at https://www.robustperception.io/configuring-prometheus-storage-retention/ explaining the requirement to keep enough space for such compactions:

As of Prometheus 2.7 and 2.8 there's new flags and options. 2.7 introduced an option for size-based retention with --storage.tsdb.retention.size, that is specifying a maximum amount of disk space used by blocks. This is still a bit best effort though as it does not (yet) include the space taken by the WAL or blocks being populated by compaction. To be safe allow for space for the WAL and one maximum size block (which is the smaller of 10% of the retention time and a month).

This is simply because the compaction only marks blocks as Deletable (https://github.com/prometheus/prometheus/blob/e51a17b50159d397544d7f62775ee9ceaf48cfc8/tsdb/compact.go#L71), but requires the additional storage to first write the merged blocks.

Coming to the real issue for me, this is not easy to convert into an actual amount of space to subtract from the total available space when setting storage.tsdb.retention.size aiming to maximize the storage retention time and making the most use of the space there is, see https://github.com/prometheus/prometheus/issues/10287. And even if I would use the referenced metrics to calculate the size a directory holding 31d worth of samples has now, this clearly is changing over time with time series churn or just more time series being ingested. And while the metrics are there in Prometheus, it's still not something that works without manual interventions and adjustments. This might be the cause for the issues like https://github.com/prometheus/prometheus/issues/10890.

To me, the current complexity in setting up retention by storage size beats the purpose of having a set it and forget it setting which translates to: Only use X GB of storage and make the most of it. If I have to calculate the ingest rate and bytes per month and continuously monitor and adjust for that ... I could just end up converting this to a suitable retention TIME instead beating the purpose of having a retention size setting in the first place.

Proposal

I therefore propose to add functionality to not exceed storage.tsdb.retention.size during compactions.

1. Maybe this could be done by extending on the existing compaction planning to calculate a worse-case of storage required to write the new merged blocks. This value could be added to the "used storage" just like the current size of the WAL was added via https://github.com/prometheus/prometheus/pull/5886.
2. Another approach would be to make the compaction have a smaller footprint altogether by continuously removing the blocks from the previous compaction level and not having all block of two compaction levels exist in parallel. This would then define an upper bound for the amount of additional storage required - something similar to the documented space requirements for WAL and their checkpoints.

By tackling this issue, it would be come MUCH less likely to run Prometheus into out of space ... which still happens when currently using retention.size feature (https://github.com/prometheus/prometheus/issues/8884).