Deep metrics buffer backlog drains slowly (`MAX_SCRAPE_FETCH_COUNT = 4`), so backlog alerts persist long after the export pipeline recovers

Summary

A deep metrics-buffer backlog drains very slowly because scrape_endpoints fetches at most MAX_SCRAPE_FETCH_COUNT = 4 files per export cycle. After any export interruption, the on-disk done/ buffer can fill to max_file_retention (~120 files), and once the pipeline recovers the monitor only clears ~4 files per cycle (net of new arrivals). Backlog-based alerting then keeps firing/re-escalating for many minutes after exports are already succeeding.

Details

lib/metrics_target_methods.rb:

MAX_SCRAPE_FETCH_COUNT = 4
...
def scrape_endpoints(session)
  scrape_files = session[:ssh_session].exec!(
    "ls :metrics_dir/done | sort | head -n :fetch_count",
    metrics_dir:, fetch_count: MAX_SCRAPE_FETCH_COUNT
  ).split("\n")
  ...
end

The collector buffers up to max_file_retention (default 120) files and trims the oldest beyond that (rhizome/common/bin/metrics-collector). With a 15s collection interval, 120 files ≈ 30 minutes of buffer.

Failure mode: if exports stall for a while (e.g. a metrics-backend outage), done/ fills toward retention. After recovery, draining 4 files/cycle while ~1 new file/cycle arrives gives a net drain of ~3/cycle, so a full buffer takes ~10+ minutes to clear. Any monitoring that pages on the buffered-file count stays above threshold for that entire tail and re-escalates, long after the underlying pipeline is healthy — which is misleading during incident response (it reads as "still broken" when it's actually just catching up).

Observed impact

After a metrics-backend outage was fixed and exports returned to ~0% failure, per-target backlog pages continued firing for many minutes while the buffers drained at the 4-files-per-cycle rate. Direct inspection of the affected hosts showed the done/ buffers steadily emptying to 0 — i.e. the pages were a lagging artifact of the slow drain, not a live failure.

Suggested directions

• Make the per-cycle fetch count adaptive: drain more aggressively when a backlog is detected (e.g. scale the batch to the current done/ count up to a cap), then fall back to the small steady-state batch.
• Or simply raise MAX_SCRAPE_FETCH_COUNT / batch-read more files per cycle.

Affected files

• lib/metrics_target_methods.rb (MAX_SCRAPE_FETCH_COUNT, scrape_endpoints)
• rhizome/common/bin/metrics-collector (max_file_retention trimming)