#292·gpt-oss

HealthBench debug mode ignores an explicit --examples override

Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026

Summary

The eval CLI documents --examples as overriding the default sample count, but all three HealthBench cases hard-code num_examples=10 if debug_mode else num_examples.

As a result, python -m gpt_oss.evals --debug --examples 2 --eval healthbench ... still evaluates 10 examples. GPQA and AIME honor the same explicit override.

Impact

Debug runs can unexpectedly perform five times the requested model/grader work, which is particularly costly for HealthBench because each sample is graded across multiple rubric items.

Proposed resolution

Use the explicit --examples value whenever provided. Only fall back to HealthBench's debug default of 10 when debug mode is active and the user did not specify a count. Add focused coverage for explicit, debug-default, and full-eval cases.