#279·gpt-oss

Responses Ollama backend injects token 0 when no streamed token is ready

Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026

Summary

gpt_oss.responses_api.inference.ollama.infer_next_token() returns integer 0 when Ollama has not produced another token by the end of its short polling slice but the overall inactivity timeout has not expired:

python
return EOS_TOKEN if False else 0

The comment describes this as a harmless PAD/NOOP placeholder. The Responses API inference interface has no no-token sentinel, and the server appends every returned integer to self.tokens and passes it to the Harmony parser immediately.

Impact

A temporarily slow Ollama stream can inject token id 0 into the generated sequence. That synthetic token was never produced by the model and can corrupt parsing/output or alter the subsequent token history supplied to the backend.

Proposed resolution

Do not manufacture a token when no model token is ready. Keep waiting until one of the legitimate terminal conditions occurs:

  • a real streamed token becomes available;
  • the Ollama stream reports an error;
  • the stream completes without another token;
  • the existing overall inactivity timeout expires, in which case emit the existing EOS token.

Add regressions proving the normal no-token polling path never returns token 0 and that a delayed real token is returned unchanged.