benchmark serving tool does not control or record radix cache state; CI silently runs a different cache protocol than users copying the same command

benchmark serving tool does not control or record radix cache state; CI silently runs a different cache protocol than users copying the same command

Summary

python/sglang/benchmark/serving.py (invoked as python -m sglang.bench_serving, deprecated alias retained) measures serving performance against a server whose radix cache is on by default and shared across runs, but by default the harness neither flushes that state nor records it, while the project's own CI silently enables flushing. Consequences, all verified from source at commit b510881157b8f12b98387340c7fbfc266dbbb35e:

1. The same benchmark command invoked twice against one server measures two different cache states. With the default seed 42 the random dataset is identical across invocations, so the second run's prompts hit the radix residue of the first run and skip nearly all prefill work, while the records written for both runs have the same schema and no field that distinguishes warm from cold.

2. The project's CI workflows set SGLANG_IS_IN_CI: true (for example .github/workflows/_pr-test-stage.yml, workflow-level env), and the flush gate is (verbatim from serving.py:1467-1469):

   python复制

   should_flush_cache = (
       "sglang" in backend and _get_bool_env_var("SGLANG_IS_IN_CI")
   ) or flush_cache

   so CI numbers are produced cold-flushed after warmup while a user copying the same command locally gets warm-residue numbers from the second invocation onward. The comment above the gate (serving.py:1464-1465, "Flush cache after warmup so the measured run does not benefit from request-local prefix reuse.") documents the intent, but the gate keeps that benefit on for every non-CI run: --flush-cache is store_true, default off (serving.py:2596-2601).

3. The result dictionary (serving.py:1796-1854) contains throughput and latency keys only; cache state appears solely behind the opt-in --cache-report (serving.py:2448-2453), and runs are appended to one JSONL file (serving.py:1899), so records from different states sit side by side, indistinguishable.

4. The state is third-party flippable between runs: POST /flush_cache (python/sglang/srt/entrypoints/http_server.py:983-988) wipes the global radix cache, is callable by any peer when no api key is configured and by any tenant key when only an api key is configured (python/sglang/srt/utils/auth.py, ADMIN_OPTIONAL semantics: the inference api_key is accepted when admin_api_key is unset), and is refused only while requests are running or waiting. A third party can therefore reset, or conversely pre-warm (by sending the victim's known benchmark prefixes), the state a benchmark will measure, with no trace in the output record.

5. Accuracy-oriented benches under benchmark/ (for example benchmark/gsm8k/bench_sglang.py, shared few-shot prefix across all requests, default temperature 0.0) do not flush or record cache state at all.

Measured demonstration (real RadixCache, tool decision order)

A stdlib simulation of the tool's exact decision order (warmup of 1 request, the CI-or-flag flush gate, measured run, append-mode record) against the real RadixCache from the tree, seeded dataset as with the tool default seed 42, one command on one persistent server:

run	state	mean uncached prefill tokens of 512
1	fresh server, user default flags	490.7
2	same command again (radix residue of run 1)	0.0
3	same command under SGLANG_IS_IN_CI=1 (CI flush)	512.0
4	after a third party POSTs /flush_cache	490.7

All four records are schema-identical and contain no state key; no key in the tool's own result schema contains cache/flush/warm state.

Related but distinct

Cache-state effects on throughput are long-known (#268 asks about periodic flushing; #1268 shows --disable-radix-cache changing results). Those discuss the cache's performance behavior. This report is about the harness protocol: the default does not control the dominant state variable, the record does not stamp it, and CI and users run different protocols for the same command.

Suggested fix

Small and data-schema-level; a patch is packaged with this report (applies at the cited commit, plus a doc note):

• Record "flushed_cache": should_flush_cache unconditionally in the result dictionary, so appended JSONL records are comparable and warm-residue reruns are distinguishable from cold-flushed runs.
• Document in the bench_serving guide that SGLANG_IS_IN_CI flips the flush on in CI, that --flush-cache is off by default, and that the second invocation of the same command measures warm residue unless flushed.

A stronger follow-up (not in the patch, changes default behavior): default the sglang backend to flushing after warmup, matching what CI already measures.

Scope notes

The latency effect of prefix caching is the feature working as designed; the defect is that the harness's default protocol leaves the dominant state variable uncontrolled and undisclosed, and that CI and users run different protocols for the same command. Whether warm-vs-cold also flips sampled tokens at temperature 0 (extend-length and batch-composition numerics) was not verified on hardware and is not claimed. The /flush_cache authorization posture (ADMIN_OPTIONAL accepting the inference key) is mentioned as the flippability primitive, not filed as a separate authorization issue here.