#19650·prometheus

web/api/v1: Search API panics when Querier is nil during TSDB reload

Author: SinghAman21Created Sep 7, 2026Updated Sep 16, 2026

What did you do?

the experimental search API (--web.enable-feature=memory-snapshot-on-shutdown is not needed, just --web.search / search APIs enabled on main) and queried the new search endpoints while Prometheus was reloading.

Steps:

  1. Build Prometheus from main (tested on commit a1b2c3d ~ 2026-09-01) with search endpoints enabled
  2. Start with a standard config (TSDB, single node, no special storage settings)
  3. In one terminal, loop reloads:
while true; do kill -HUP $(pidof prometheus); sleep 2; done
  1. In another terminal, loop search queries:
while true; do curl -s "http://localhost:9090/api/v1/search/metric_names?search[]=up" > /dev/null; echo -n .; done
  1. After a few reload cycles the search request panics. It's a race window, not 100% reproducible every reload, but happens within 1-2 minutes of looping. Easiest to reproduce with a mock Queryable that returns (nil, nil).

What did you expect to see?

A proper API error, e.g. 503 TSDB not ready or 500 search not supported by storage, with a JSON error body. No panic, no 500 with stack trace. Other API endpoints correctly return 503 when TSDB is not ready.

What did you see instead? Under which circumstances?

Panic in the HTTP handler for any of /api/v1/search/metric_names, /label_names, /label_values when the query hits the narrow window during SIGHUP / TSDB reload. The panic is recovered by the web layer but the request still fails with 500 and leaves a stack trace in logs.

Under normal operation without reloads I never see it. Only during reload / restart, and only on the search endpoints.

System information

Linux 6.8.0-52-generic x86_64

Prometheus version

prometheus, version 3.5.0-rc.0 (branch: main, revision: a1b2c3d4e5f6)
  build user:       aman@dev
  build date:       20260903-10:00:00
  go version:       go1.23.4
  platform:         linux/amd64
  tags:             netgo static_build

Prometheus configuration file

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Alertmanager version

Alertmanager configuration file

Not relevant (no Alertmanager in this setup).

Logs

ts=2026-09-03T10:12:14.123Z caller=main.go:987 level=info msg="Reloading