tsdb: OOO blocks contain dead symbols

What did you do?

An out-of-order (OOO) block can accumulate symbols, that aren't referenced by any entry in the block's postings offset table.

That is, TSDB is configured with non-zero OutOfOrderTimeWindow. When it cuts its OOO head, the OOOCompactionHeadIndexReader.Symbols() produces a string iterator, that walks over symbols from the entire head, without applying the OOO-series (or any time-range) filtering. This results in OOO block, that contains dead symbols.

These symbols later survive LeveledCompactor.Compact(), because the DefaultBlockPopulator.PopulateBlock() merges the complete symbol iterators from all input indices and writes their union into the output index (ref tsdb/compact.go).

What did you expect to see?

Either production of OOO block should filter out symbols, that aren't referenced by postings offset table. Or, the compaction should clear those out.

What did you see instead? Under which circumstances?

In grafafan/mimir, we observed OOO blocks, that over many rounds of non-split compaction cycles can accumulate gigabytes of such dead symbols.

System information

No response

Prometheus version

I've reproduced this with a simple unit-test in the current main (ref test).

Prometheus configuration file

n/a

Alertmanager version

n/a

Alertmanager configuration file

n/a

Logs

n/a