#281·gpt-oss

Responses stub backend does not reset fake output on new requests

Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026

Summary

The Responses API stub backend accepts the standard new_request flag but ignores it. Its fake token sequence is stored in the module-global token_queue, so a new API request starts wherever the previous request stopped rather than from the beginning of the deterministic stub response.

Current behavior

python
def stub_infer_next_token(
    tokens: list[int], temperature: float = 0.0, new_request: bool = False
) -> int:
    global token_queue
    next_tok = token_queue.pop(0)
    ...

The server sets new_request=True on the first inference call for each StreamResponsesEvents request, and the Metal backend already uses that flag to reset its per-request output state.

Impact

Stub-backed API tests and development sessions are order-dependent across requests. A second request can receive a suffix of the fake response instead of the same deterministic response used by the first request.

Proposed resolution

When new_request=True, reset token_queue to fake_tokens.copy() before returning the first token.

Add a regression that consumes several tokens, starts a new request, and verifies the sequence restarts from fake_tokens[0].