prometheus: scrape-path perf — histogram labels, text batching, protobuf reuse (tracking)
Motivation
src/core/prometheus.cc's scrape path (write_body() and friends) does more per-scrape work than it needs to: histogram/summary text output re-walks the label map once per bucket, text-path series writes go through the output stream one at a time, and the protobuf path allocates a fresh message/string/label-string per family instead of reusing one across the request. Three independent, minimal-diff PRs fix these, verified individually with tests/perf/metrics_perf.cc.
PRs
- PR1: render histogram/summary labels once per metric, not per bucket (branch
prom-perf/1-histogram-labels) - PR2: batch text-path series writes, avoid per-family name concat (branch
prom-perf/2-text-batching) - PR3: reuse protobuf message, buffer and label storage across families (branch
prom-perf/3-protobuf-reuse)
Each is independently mergeable and individually benchmarked in its own PR description. This issue tracks the combined effect of stacking all three together with #3672 (prometheus: cache per-series aggregation key instead of rebuilding it every scrape).
Combined stack results (PR1+PR2+PR3+#3672 vs master)
Median of 3 runs, tests/perf/metrics_perf, -c1:
| Case | Master inst/op | Stack inst/op (%Δ) | Master allocs/op | Stack allocs/op (%Δ) |
|---|---|---|---|---|
| test_few_metrics | 8577.95 | 8709.07 (+1.53%) | 22.125 | 22.125 (+0.00%) |
| test_large_families | 1481.73 | 1436.61 (-3.05%) | 0.165 | 0.165 (+0.00%) |
| test_large_families_int | 991.52 | 943.87 (-4.81%) | 0.165 | 0.165 (+0.00%) |
| test_large_families_int_aggr | 857.72 | 312.49 (-63.57%) | 0.158 | 0.268 (+69.62%) |
| test_many_families_int | 2994.58 | 3106.81 (+3.75%) | 7.273 | 7.273 (+0.00%) |
| test_many_families_int_aggr | 4668.88 | 4445.79 (-4.78%) | 12.289 | 12.399 (+0.90%) |
| test_middle_ground | 1678.29 | 1648.84 (-1.75%) | 0.846 | 0.846 (+0.00%) |
| test_middle_ground_int | 1188.18 | 1156.67 (-2.65%) | 0.845 | 0.846 (+0.12%) |
| test_middle_ground_protobuf | 5773.06 | 4259.27 (-26.22%) | 19.843 | 4.861 (-75.50%) |
| test_histogram | 1715.90 | 1320.12 (-23.07%) | 0.092 | 0.092 (+0.00%) |
| test_histogram_protobuf | 416.19 | 344.57 (-17.21%) | 1.226 | 0.548 (-55.30%) |
| test_histogram_aggr | 245.03 | 203.33 (-17.02%) | 0.113 | 0.118 (+4.42%) |
| test_name_filter_exact_match | 170.18 | 170.29 (+0.06%) | 0.538 | 0.538 (+0.00%) |
| test_name_filter_many_no_match | 310.21 | 310.32 (+0.04%) | 0.548 | 0.548 (+0.00%) |
The two test_name_filter_* cases are flat — the fourth optimization in the original set (a hash-based name-filter lookup, targeting exactly those two cases) was dropped after investigation found no evidence of the many-exact-filter scrape pattern it optimizes (federation-style match[]=) in ScyllaDB's own monitoring config.
The aggregation allocs/op rises (_int_aggr/histogram_aggr) are #3672's cache, a one-time per-metadata-rebuild cost, not per-scrape — see #3672 for detail.
Per-PR benchmark tables (PR alone vs master) are in each PR's own description.
Generated with Claude Code
Source: scylladb/seastar