KV-cache / long-context: smallest canonical repro boundary + metric (7-day receipts eval)

Feature request

Request: add an official “smallest long-context / KV-cache perf boundary” recipe for TGI.

What I mean by “boundary”:

• a single runnable command/config (model, context length, batch/concurrency, backend),
• the metric to report (tokens/s and p99 latency at minimum; OOM threshold optional),
• a short note on the expected bottleneck (KV-cache memory traffic) so users don’t mis-measure.

Why this matters: KV-cache / long-context discussions often stall without a canonical minimal repro. A maintained boundary makes perf work comparable, and lets contributors return receipts instead of opinions.

Reference: vLLM boundary request already posted here: https://github.com/vllm-project/vllm/issues/32038

Motivation

I’m frustrated that long-context / KV-cache perf threads are hard to reproduce across machines because every report uses a different model/context/concurrency/backend.

A canonical minimal boundary would:

• make regressions obvious,
• make improvements measurable,
• reduce triage time for maintainers.

Your contribution

If maintainers confirm the canonical boundary (command/config + acceptance metric), I can run it and return within 7 days:

• receipts-backed before/after on the same boundary (p95/p99/p99.9 deltas),
• optional energy/cost proxy,
• filename-only SHA256 index for internal forwarding.

I’m not asking anyone to believe a claim; I’m asking for the smallest boundary to measure.