#299·gpt-oss

GPQA debug mode ignores an explicit --examples sample count

Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026

Summary

The eval CLI resolves an explicit --examples value and passes it to GPQAEval, but also passes debug=True whenever --debug is set. Inside GPQAEval, the debug branch selects one fixed Espresso example and never consults num_examples.

So python -m gpt_oss.evals --debug --examples 2 --eval gpqa ... still evaluates the single hard-coded debug example instead of the requested two-example subset.

Proposed resolution

Treat the fixed GPQA debug example as the default only when no explicit sample count is supplied. When --examples is present, disable the fixed-example branch and use the normal deterministic subset sampler with one repeat. Add lightweight CLI-resolution coverage.