GPQA debug mode ignores an explicit --examples sample count
Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026
Summary
The eval CLI resolves an explicit --examples value and passes it to GPQAEval, but also passes debug=True whenever --debug is set. Inside GPQAEval, the debug branch selects one fixed Espresso example and never consults num_examples.
So python -m gpt_oss.evals --debug --examples 2 --eval gpqa ... still evaluates the single hard-coded debug example instead of the requested two-example subset.
Proposed resolution
Treat the fixed GPQA debug example as the default only when no explicit sample count is supplied. When --examples is present, disable the fixed-example branch and use the normal deterministic subset sampler with one repeat. Add lightweight CLI-resolution coverage.
Source: openai/gpt-oss