tsdb: no metric for how many series stale series compaction actually evicts

Proposal

I was reading through the stale series compaction code and I think there's a gap in what it reports.

Right now there are six metrics for stale and selected series compaction:

• prometheus_tsdb_stale_series_compactions_triggered_total
• prometheus_tsdb_stale_series_compactions_failed_total
• prometheus_tsdb_stale_series_compaction_duration_seconds
• and the same three for selected series compaction

All of them tell you the compaction ran. None of them tell you whether it actually freed anything.

That matters because eviction is deliberately conservative. truncateStaleSeries skips a series if it picked up out of order data, got a new sample, or its fingerprint changed while the block was being written. So a run can go through 50k candidates and evict zero of them, and from the outside it looks identical to a run that evicted all of them. Triggered counter goes up, failed counter stays flat, duration gets recorded, done.

The log line doesn't really help either:

db.logger.Info("Ending stale series compaction", "num_series", len(staleSeriesRefs.sortedByRef), "duration", elapsed)

num_series there is the candidate count, not the number that got evicted. So a run that freed nothing still logs num_series=50000, which reads like it worked.

The slightly odd part is the number already exists. truncateSeries returns len(deleted) and both callers throw it away:

func (h *Head) truncateStaleSeries(...) error {
	_, err := h.truncateSeries(seriesRefs, maxt, func(s *memSeries) bool {

Same thing in truncateSelectedSeries.

I checked whether prometheus_tsdb_head_series_removed_total covers it and it doesn't. That one gets incremented from four different places (WAL replay, normal head gc, this path, and deleteSeriesByID), so you can't pull stale compaction out of it.

Why I think it's worth adding: stale_series_compaction_threshold is opt in and still experimental, so the people running it are exactly the people who need to know if it's helping. In #18379 someone turned it on, hit problems, and was debugging off a memory graph because there wasn't anything better to look at.

I hit a smaller version of this myself while writing a test against this code. I spent a while assuming eviction was happening when it actually wasn't, and only worked it out by reading head.metrics.seriesRemoved directly. An operator can't do that.

What I'd suggest:

• counters for candidates and evicted per compaction type, so something like prometheus_tsdb_stale_series_compaction_series_candidates_total and prometheus_tsdb_stale_series_compaction_series_evicted_total, plus the selected series versions
• add num_evicted to both log lines, and leave num_series alone so nothing parsing logs breaks

Both numbers are useful since evicted over candidates is the thing you'd actually alert on, and you can't work one out from the other with what's there today. If four new metrics feels like too many, just the evicted ones would still fix the main problem.

The change itself is small. The count already gets computed, it just needs returning instead of dropping, and the callers can capture it in the closure they already pass so headSeriesEvictor doesn't need to change.