#298·gpt-oss

Eval CLI --examples crashes non-debug AIME and GPQA because n_repeats stays 8

Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026

Summary

The eval CLI passes n_repeats=8 to AIME and GPQA whenever --debug is false, even when the user supplies --examples.

Both constructors intentionally require n_repeats == 1 when a positive subset size is requested:

python
assert n_repeats == 1, "n_repeats only supported for num_examples = None"

So a documented command such as python -m gpt_oss.evals --examples 2 --eval aime25 ... reaches the constructor with num_examples=2, n_repeats=8 and fails before evaluation starts. GPQA has the same path.

Proposed resolution

When an explicit sample count is supplied, set n_repeats=1 for AIME and GPQA regardless of debug mode. Preserve 8 repeats only for full non-debug evaluations. Add lightweight regression coverage for explicit, debug, and full-run repeat selection.