web/api/v1: Search API panics when Querier is nil during TSDB reload
What did you do?
the experimental search API (--web.enable-feature=memory-snapshot-on-shutdown is not needed, just --web.search / search APIs enabled on main) and queried the new search endpoints while Prometheus was reloading.
Steps:
- Build Prometheus from
main(tested on commita1b2c3d~ 2026-09-01) with search endpoints enabled - Start with a standard config (TSDB, single node, no special storage settings)
- In one terminal, loop reloads:
while true; do kill -HUP $(pidof prometheus); sleep 2; done
- In another terminal, loop search queries:
while true; do curl -s "http://localhost:9090/api/v1/search/metric_names?search[]=up" > /dev/null; echo -n .; done
- After a few reload cycles the search request panics. It's a race window, not 100% reproducible every reload, but happens within 1-2 minutes of looping. Easiest to reproduce with a mock
Queryablethat returns(nil, nil).
What did you expect to see?
A proper API error, e.g. 503 TSDB not ready or 500 search not supported by storage, with a JSON error body. No panic, no 500 with stack trace. Other API endpoints correctly return 503 when TSDB is not ready.
What did you see instead? Under which circumstances?
Panic in the HTTP handler for any of /api/v1/search/metric_names, /label_names, /label_values when the query hits the narrow window during SIGHUP / TSDB reload. The panic is recovered by the web layer but the request still fails with 500 and leaves a stack trace in logs.
Under normal operation without reloads I never see it. Only during reload / restart, and only on the search endpoints.
System information
Linux 6.8.0-52-generic x86_64
Prometheus version
prometheus, version 3.5.0-rc.0 (branch: main, revision: a1b2c3d4e5f6)
build user: aman@dev
build date: 20260903-10:00:00
go version: go1.23.4
platform: linux/amd64
tags: netgo static_build
Prometheus configuration file
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
Alertmanager version
Alertmanager configuration file
Not relevant (no Alertmanager in this setup).
Logs
ts=2026-09-03T10:12:14.123Z caller=main.go:987 level=info msg="Reloading
Source: prometheus/prometheus