AIME and GPQA treat num_examples=0 as the full dataset
Summary
The AIME and non-debug GPQA constructors guard subset selection with if num_examples:. Because zero is falsey, num_examples=0 silently skips sampling and evaluates the full dataset instead of zero examples.
This is inconsistent with the parameter contract (int | None, where None represents no restriction) and with HealthBench, which can represent an empty sample set.
Impact
A zero-example smoke/debug configuration can unexpectedly launch a full evaluation, causing substantial unnecessary inference work instead of returning an empty result.
Proposed resolution
Distinguish None from zero explicitly. Validate that sample counts are non-negative, use an empty example list for zero, and retain the existing subset sampling behavior for positive counts. Add regression coverage for zero and negative counts.
Source: openai/gpt-oss