[FEATURE]: ThunderAgent shared-prefix accounting: GPU A/B/C results with explicit headroom

Feature request

Investigate shared-prefix-aware program accounting with explicit growth headroom in ThunderAgent. I ran a single-replica GPU experiment comparing unchanged accounting, an idealized deduplication prototype, and threshold-only tuning.

At 90% initial sharing, the prototype improved program completion throughput by 30.2% versus unchanged accounting (95% paired bootstrap interval: 23.6–37.3%), and reduced program P95 by 23.2%, but increased P50 by 16.9%. This is a scoped throughput/tail improvement with a median-latency tradeoff, not a general latency win or a production-ready implementation.

The motivation was a Slack discussion in which the repeated accounting was described as implicit buffer/headroom and investigating deduplication was encouraged. That discussion was encouragement to investigate, not approval of this implementation or its performance claims.

Describe the problem you're encountering

In the tested scheduler source, _replica_usage_locked() adds each ACTIVE program's token total plus a per-program buffer. An exact prefix shared by multiple programs is therefore charged repeatedly, even when the backend reuses it. Capacity discovery and the backend's physical occupancy signals have different semantics; this report does not infer waste merely from apparently unused GPU memory.

A preceding unchanged-scheduler pilot covered 0/50/90% sharing and 2/4/8/16/32 programs: no pauses at the tested 2–16 levels, but the 32-program level initially admitted 28 at every sharing level. Exact token-prefix checks and backend cache hits confirmed reuse. This established the accounting/concurrency question; it did not establish a deduplication speedup. The A/B/C results below are the subsequent controlled comparison, not speedups computed across different sharing workloads.

Describe alternatives you've tried

B and C each received three calibration candidates, with the same zero/high-sharing calibration workload and eligibility rule. Parameters were then frozen before held-out validation. All candidates, including unfavorable results, are retained below. C is the selected candidate from this bounded search, not a globally optimal threshold configuration; default A was itself faster than selected C on primary validation, so B/A is the main comparison.

B's accounting is rebuilt from ACTIVE membership on admission, pause, resume and completion/cancellation. Removing one program retains a shared-family charge while another ACTIVE program still uses it. A logical accounting decrease is not a claim of immediate backend memory release. The prototype changes accounting/admission checks in a local benchmark subclass, without editing installed Dynamo files, KV layout, or cross-replica placement. It assumes ACTING weight 1 and one compatible replica; no semantic similarity, adapters, or production cache index.

Held-out GPU results

Ratios are geometric means of paired batch ratios. Throughput ratios above 1 are better; P50/P95 ratios above 1 are worse. Intervals use 10,000 bootstrap resamples of whole paired batches, seed 20260919; programs/tokens are not treated as independent repetitions. The three-block secondary checks have limited precision.

Absolute arithmetic means for the 12 primary blocks (these are a different aggregation from the paired geometric ratios):

Program completion time runs from scheduled arrival through the third inference response, including scheduler waiting and simulated tool waits. Batch time runs from the batch arrival origin to the last completed program; throughput is completed programs divided by that time. Per-batch P50/P95 use nearest-rank percentiles of 32 program times; they are not a long-running production latency distribution.

Observed:

• All 54/54 held-out validation rows completed and passed input, lifecycle, route, capacity and accounting audits: 18 paired blocks, each containing A/B/C. Across qualification, overhead checks, calibration and validation, 78 valid rows completed 2,238 programs / 6,714 inference calls. One additional pre-measurement infrastructure failure is retained.
• At 50/90% sharing, B admitted all 32 programs without budget pauses; A/C initially admitted 28 and recorded 32 pause events per batch. At zero sharing, all arms recorded 32 pause events. Pause counts explain behavior; they are not the success metric.
• Backend preemption counter increments were zero in every validation row; backend generation-counter deltas matched actual streamed output tokens. No validation request failures were observed. The backend log scan found no ERROR/traceback/OOM records; authoritative backend failure/OOM counters remain unavailable, and this is not a general safety guarantee.
• At 90% sharing, each arm had the same per-batch request-usage-derived sum(prompt_tokens - cached_tokens) = 132,608; at 50%, 315,392. Thus the measured gain did not require reducing this observable prompt-work proxy. At zero sharing, B's mean proxy was approximately 0.9% higher than A's. This proxy is not total engine recomputation.
• The zero-sharing B/A P95 interval was 0.981–1.007, within the predeclared 1.05 margin for these three held-out blocks. This does not establish regression freedom for arbitrary zero-sharing workloads.

Possible explanation: earlier admission gives the last-finishing programs less scheduler waiting, improving makespan and P95, while increased concurrency makes some earlier-finishing programs contend longer and worsens P50. The observations support a scheduling opportunity, not faster individual GPU token computation.

Reproduction scope and controls

Actual shared-prefix ratios were reconstructed from token IDs at every turn, rather than assumed constant:

Every row used a fresh router with the same backend and the same two-request warmup sequence (512 input / 16 output tokens per request), then an explicit backend cache reset and empty-state checks. Warmup and fixture construction were excluded from timing. All arms used the same prefix validation and measurement wrapper; no client concurrency semaphore. This used Dynamo's native token endpoint, so there was no chat-template-added common prefix or HTTP frontend in the measurement.

Before calibration, three N=8, 50%-sharing sampler-on/off pairs gave a batch-time ratio geometric mean of 0.9996, below the predeclared 5% overhead gate. This isolates incremental sampler cost, not every logging cost. Measured rows had a 240 s deadline. Failures stopped the campaign; unfavorable valid samples were not rerun or replaced, and validation sample counts were not extended after seeing results.

B reserves: 256 / 512 / 1,024 tokens per ACTIVE program. C threshold/target/hysteresis triples: 0.97/0.90/0.05, 0.99/0.95/0.02, 1.00/1.00/0.00. Each candidate used the same seed 520 at 0% and 90% sharing, with A references. Eligibility required zero-sharing batch time and P95 no more than 5% above A, zero observed preemption increments, complete output and fresh generation counters. Among eligible candidates, select fastest high-sharing throughput, then freeze.

The slightly faster high-sharing B candidates were not chosen by overlooking zero-sharing regressions.

Limitations and next step

1. B knows the synthetic sharing relationships in advance, validates actual request prefixes against them, and does not infer backend residency. A production implementation needs trustworthy existing block identities/shared references and conservative behavior for unknown or stale information. Complete physical shared-reference/residency measurements and total engine recomputation remain TBD.
2. Backend cache-hit token counts are not deductible physical bytes. The tested vLLM usage gauge counts non-free blocks and excludes reclaimable cached blocks; its denominator excludes the null block. It cannot simply be subtracted from ThunderAgent's logical retention estimate to obtain a physical estimation error. GPU capacity, any offload capacity and growth reserve must remain distinct.
3. Every one of the 36 B/C validation batches had at least one generated sequence different from paired A, despite identical initial token IDs and matched output lengths. Actual generated histories were retained and fed into later turns. The cause has not been isolated, and semantic task equivalence/quality was not evaluated.
4. The P50 regression is material. These are synthetic finite batches on one GPU/model/backend in eager mode, not a claim about real tools, sustained arrivals, CUDA graph configurations, offload, adapters or clusters. C's search and secondary sample sizes are small.

The result supports continuing a narrowly scoped accounting investigation. A useful next step would be to agree on an existing source of same-replica, cache-compatible block identity/reference information, then validate a conservative implementation with the same A/B/C controls, explicit growth reserve, lifecycle accounting invariants and throughput/P50/P95/preemption gates. This report does not request merging the oracle prototype. The input/metric semantics also relate to admission-control follow-ups #14823.

Data and audit trail

The appendix includes all 79 A/B/C campaign attempts as a compact CSV, preserving original numeric precision and TBD values. Only the local raw-file-path column is removed. All rows are included, but the headline comparisons use only stage=validation and audit=PASS with complete A/B/C blocks. A small standalone analysis script is included to reproduce the paired intervals from this CSV.

The full harness, frozen manifests, actual token histories, time series and raw logs are retained locally; they are not attached to this issue, so this issue alone does not provide a full GPU rerun bundle. The complete A/B/C archive is approximately 4.4 GB uncompressed / 66.7 MB compressed. Local raw-data reconstruction matched the remote row table, audit, comparisons and report byte-for-byte. These hashes identify retained artifacts; they are not download links:

archive SHA256: 5228236e96f5fdf00d0f5ed8a9614e2b3e66a7743fd4ba9fdab5f4d095210903
original rows.csv SHA256: 45430a8dd54a782d39b685434cfce4656b4f2c0770c82eabc7ae84c35b9ece17
comparisons.json SHA256: 8f208279c8a770547ab2806fbab280a5a625432697c6605af415bf9e8987e373
public compact CSV SHA256: c1c36f3b279b0ff810c470451abfc1fd1a4c3c73e5f4fdb0ad294be768fc97fb

One early overhead row failed before measured work because transient discovery selected a stopped router. It and the preceding valid overhead row remain in the appendix. Before calibration, the harness was fixed to verify the intended router identity and select it using existing Client.direct, identically for all arms; this did not select backend placement. The entire overhead stage was repeated under separate v2 IDs with unchanged settings. A later postprocessing 0 versus 0.0 grouping-key correction fixed one zero-sharing Boolean gate; raw measurements and numerical intervals were unchanged, and the original summary was retained.

stage,arm,share,programs,completed,audit,batch_seconds,programs_per_second,p50_program_seconds,p95_program_seconds,scheduler_pauses,scheduler_resumes,forced_resumes,first_turn_cached_tokens,backend_preemptions,backend_recomputed_tokens,backend_generation_tokens_delta,backend_failures,backend_oom,client_failed_programs,row_id,seed,candidate,sampling_off,phase,tool_wait,jitter_ms,generation_counter_matches,capacity_tokens,peak_sampled_active_programs,computed_prompt_usage_proxy
overhead,A,0.5,8,0,FAIL,TBD,TBD,TBD,TBD,TBD,TBD,TBD,TBD,TBD,TBD,TBD,TBD,TBD,0,overhead_0_off,510,a,True,0.25,2.0,0.0,False,TBD,TBD,TBD
overhead,A,0.5,8,8,PASS,25.24284603074193,0.31692147510852076,25.230464980006218,25.24284603074193,0,0,0,57344,0.0,TBD,6144.0,TBD,TBD,0,overhead_0_on,510,a,False,0.25,2.0,0.0,True,468736,8,78848
qualification,A,0.9,2,2,PASS,17.102302964776754,0.11694331483421402,17.102134324610233,17.102302964776754,0,0,0,14720,0.0,TBD,1536.0,TBD,TBD,0,qual_A,500,a,False,0.25,2.0,0.0,True,468736,2,19328
qualification,B,0.9,2,2,PASS,17.17427995055914,0.11645320827176142,17.16035968065262,17.17427995055914,0,0,0,14720,0.0,TBD,1536.0,TBD,TBD,0,qual_B,500,b1,False,0.25,2.0,0.0,True,468736,2,19328
qualification,C,0.9,2,2,PASS,17.09497794881463,0.11699342379898656,17.08036397024989,17.09497794881463,0,0,0,14720,0.0,TBD,1536.0,TBD,TBD,0,qual_C,500,c2,False,0.25,2.0,0.0,True,468736,2,19328
calibration,A,0.0,32,32,PASS,105.78584540635347,0.3024979370073464,74.73229671269655,105.78481394797564,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_A_s0,520,a,False,0.25,2.0,0.0,True,468736,28,617536
calibration,A,0.9,32,32,PASS,49.27877000719309,0.6493668570731178,34.27091285213828,49.27850256487727,32,32,0,412160,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_A_s90,520,a,False,0.25,2.0,0.0,True,468736,28,132608
calibration,B,0.0,32,32,PASS,109.2560160048306,0.2928900500873578,74.87545352056623,109.22850679606199,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_b1_s0,520,b1,False,0.25,2.0,0.0,True,468736,28,617216
calibration,B,0.9,32,32,PASS,40.079017516225576,0.7984227654044945,40.05761616677046,40.07858734577894,0,0,0,412160,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_b1_s90,520,b1,False,0.25,2.0,0.0,True,468736,32,132608
calibration,B,0.0,32,32,PASS,120.97897066548467,0.2645087805258506,89.62250036746264,120.95721035823226,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_b2_s0,520,b2,False,0.25,2.0,0.0,True,468736,27,1044160
calibration,B,0.9,32,32,PASS,39.83717757835984,0.8032697581814351,39.80313540250063,39.83665356040001,0,0,0,412160,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_b2_s90,520,b2,False,0.25,2.0,0.0,True,468736,32,132608
calibration,B,0.0,32,32,PASS,112.53944955021143,0.28434473536075583,82.8782959356904,112.52476649731398,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_b3_s0,520,b3,False,0.25,2.0,0.0,True,468736,26,841728
calibration,B,0.9,32,32,PASS,39.80668671056628,0.8038850415426794,39.76332542672753,39.802764557302,0,0,0,412160,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_b3_s90,520,b3,False,0.25,2.0,0.0,True,468736,32,132608
calibration,C,0.0,32,32,PASS,107.0550562441349,0.2989116172806004,76.91118652001023,107.04024543985724,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_c1_s0,520,c1,False,0.25,2.0,0.0,True,468736,28,584896
calibration,C,0.9,32,32,PASS,52.20042197033763,0.6130218644244616,36.02235784381628,52.1948421522975,32,32,0,412160,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_c1_s90,520,c1,False,0.25,2.0,0.0,True,468736,28,132608
calibration,C,0.0,32,32,PASS,106.14361491799355,0.301478332207954,77.92483263462782,106.13839511573315,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_c2_s0,520,c2,False,0.25,2.0,0.0,True,468736,28,585728
calibration,C,0.9,32,32,PASS,68.15925515443087,0.46948869860001335,45.70866207405925,68.15903060138226,32,32,0,412160,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_c2_s90,520,c2,False,0.25,2.0,0.0,True,468736,28,132608
calibration,C,0.0,32,32,PASS,115.09502013027668,0.27803114299627407,87.11892970651388,115.06264853477478,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_c3_s0,520,c3,False,0.25,2.0,0.0,True,468736,28,766784
calibration,C,0.9,32,32,PASS,62.436535239219666,0.5125204317855728,41.14880537614226,62.43631872534752,32,32,0,412160,0.0,TBD,24576.0,TBD,TBD,0,v2_cal_c3_s90,520,c3,False,0.25,2.0,0.0,True,468736,28,132608
validation,A,0.0,32,32,PASS,103.79934005439281,0.3082871238220917,74.15594271197915,103.6221171990037,32,32,0,0,0.0,TBD,24576.0,TBD,TBD,0,v2_heldout_s0_0_A,700,a,False,0.25,1.0,5.0,True,468736,28,623872
validatio