#13063·dynamo

fix(llm): stream_choice_chunk_from_template leaves nvext on the synthetic chunk

Author: keivenchangCreated Aug 12, 2026Updated Sep 19, 2026
Labelsbuglanguage::rustdynamo-runtimedynamo-llmStalefrontend

Overview

stream_choice_chunk_from_template builds a synthetic chunk by cloning a response template and clearing the fields that describe the template chunk's own generation. It clears usage and llm_metrics but not nvext, so a per-chunk NVIDIA extension is emitted a second time on the synthetic chunk.

merge_response_nvext append-merges completion_token_ids, so a repeat turns [42] into [42, 42] during non-streaming aggregation. Fields it does not append (such as prompt_logprobs) are overwritten instead.

Details

lib/llm/src/protocols/openai/chat_completions.rs:

rust
let mut response = template.clone();
response.inner.usage = None;
response.llm_metrics = None;
// nvext is not cleared

This predates #11251 and is present on main at the merge-base. Two sibling instances of the same defect were found and fixed in #11251 (both end-of-stream flushes in preprocessor.rs), which now share a single scrub_synthetic_chunk_metadata helper. This site was left out deliberately to keep that PR scoped to its own regressions.

Suggested fix

Route this site through the same shared scrub, so the field list lives in one place. The two preprocessor.rs sites each kept their own copy of the list and both copies missed nvext — that drift is what the shared helper exists to prevent.

Validation

Not reproduced end-to-end for this site specifically. The equivalent defect was reproduced for the preprocessor.rs sites: a regression test asserting nvext is emitted once failed at 2 != 1 before the fix.