#3683·seastar

prometheus: scrape-path perf — histogram labels, text batching, protobuf reuse (tracking)

Author: mykaulCreated Sep 11, 2026Updated Sep 12, 2026
Labelsenhancementmetricsai-assisted

Motivation

src/core/prometheus.cc's scrape path (write_body() and friends) does more per-scrape work than it needs to: histogram/summary text output re-walks the label map once per bucket, text-path series writes go through the output stream one at a time, and the protobuf path allocates a fresh message/string/label-string per family instead of reusing one across the request. Three independent, minimal-diff PRs fix these, verified individually with tests/perf/metrics_perf.cc.

PRs

  • PR1: render histogram/summary labels once per metric, not per bucket (branch prom-perf/1-histogram-labels)
  • PR2: batch text-path series writes, avoid per-family name concat (branch prom-perf/2-text-batching)
  • PR3: reuse protobuf message, buffer and label storage across families (branch prom-perf/3-protobuf-reuse)

Each is independently mergeable and individually benchmarked in its own PR description. This issue tracks the combined effect of stacking all three together with #3672 (prometheus: cache per-series aggregation key instead of rebuilding it every scrape).

Combined stack results (PR1+PR2+PR3+#3672 vs master)

Median of 3 runs, tests/perf/metrics_perf, -c1:

Case Master inst/op Stack inst/op (%Δ) Master allocs/op Stack allocs/op (%Δ)
test_few_metrics 8577.95 8709.07 (+1.53%) 22.125 22.125 (+0.00%)
test_large_families 1481.73 1436.61 (-3.05%) 0.165 0.165 (+0.00%)
test_large_families_int 991.52 943.87 (-4.81%) 0.165 0.165 (+0.00%)
test_large_families_int_aggr 857.72 312.49 (-63.57%) 0.158 0.268 (+69.62%)
test_many_families_int 2994.58 3106.81 (+3.75%) 7.273 7.273 (+0.00%)
test_many_families_int_aggr 4668.88 4445.79 (-4.78%) 12.289 12.399 (+0.90%)
test_middle_ground 1678.29 1648.84 (-1.75%) 0.846 0.846 (+0.00%)
test_middle_ground_int 1188.18 1156.67 (-2.65%) 0.845 0.846 (+0.12%)
test_middle_ground_protobuf 5773.06 4259.27 (-26.22%) 19.843 4.861 (-75.50%)
test_histogram 1715.90 1320.12 (-23.07%) 0.092 0.092 (+0.00%)
test_histogram_protobuf 416.19 344.57 (-17.21%) 1.226 0.548 (-55.30%)
test_histogram_aggr 245.03 203.33 (-17.02%) 0.113 0.118 (+4.42%)
test_name_filter_exact_match 170.18 170.29 (+0.06%) 0.538 0.538 (+0.00%)
test_name_filter_many_no_match 310.21 310.32 (+0.04%) 0.548 0.548 (+0.00%)

The two test_name_filter_* cases are flat — the fourth optimization in the original set (a hash-based name-filter lookup, targeting exactly those two cases) was dropped after investigation found no evidence of the many-exact-filter scrape pattern it optimizes (federation-style match[]=) in ScyllaDB's own monitoring config.

The aggregation allocs/op rises (_int_aggr/histogram_aggr) are #3672's cache, a one-time per-metadata-rebuild cost, not per-scrape — see #3672 for detail.

Per-PR benchmark tables (PR alone vs master) are in each PR's own description.

Generated with Claude Code