#294·gpt-oss

AIME and GPQA treat num_examples=0 as the full dataset

Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026

Summary

The AIME and non-debug GPQA constructors guard subset selection with if num_examples:. Because zero is falsey, num_examples=0 silently skips sampling and evaluates the full dataset instead of zero examples.

This is inconsistent with the parameter contract (int | None, where None represents no restriction) and with HealthBench, which can represent an empty sample set.

Impact

A zero-example smoke/debug configuration can unexpectedly launch a full evaluation, causing substantial unnecessary inference work instead of returning an empty result.

Proposed resolution

Distinguish None from zero explicitly. Validate that sample counts are non-negative, use an empty example list for zero, and retain the existing subset sampling behavior for positive counts. Add regression coverage for zero and negative counts.