API retry helper retries forever (no max_tries/max_time) and the error-detection branch in its callers is dead code

PUBLIC-READY MINT R02E2-M3 (unbounded recovery loop in the model I/O path plus an unreachable error check left over from the pre-1.0 SDK; availability defect in the harness, not a score-integrity issue) Target: openai/evals @ 8eac7a7de5215c907fbddc30efdaf316913eccdd Files: evals/utils/api_utils.py (create_retrying), evals/completion_fns/openai.py (openai_completion_create_retrying, openai_chat_completion_create_retrying)

Summary

create_retrying is decorated with @backoff.on_predicate(wait_gen=backoff.expo, max_value=60, factor=1.5) with NO max_tries and NO max_time. The function body catches the retry exceptions and returns False, and backoff.on_predicate retries while the result is falsy (its default predicate is operator.not_). Therefore a completion call that keeps failing with a retryable error (RateLimit, APIConnection, APITimeout, InternalServerError) retries forever, with waits capped at 60 seconds each. The eval thread never gives up and never surfaces the failure.

Additionally, both callers contain:

result = create_retrying(...)
if "error" in result:
    raise openai.APIError(result["error"])

This branch is unreachable (create_retrying never returns a falsy or dict value: it either returns the response or loops forever), and it is stale for the openai 1.x SDK, where responses are pydantic objects rather than dicts. If it were ever reached with a falsy result it would crash with TypeError: argument of type 'bool' is not iterable rather than raising the intended APIError.

Repro (deterministic property, no network)

PoC: f5_unbounded_retry.py Runner: REPRO-unbounded-retry.sh

The PoC patches time.sleep to a no-op so backoff never waits, calls create_retrying against a function that always raises a retryable exception, and stops the loop with a watchdog thread after about 1.5 seconds of wall clock. Asserted at the pin:

• The module source configures no max_tries and no max_time.
• Backoff never gives up on its own: more than 100,000 attempts occur in 1.5 seconds with zero waits (the only thing that stops the loop is the PoC's own watchdog).
• "error" in False raises TypeError, demonstrating what the dead branch would do if reached.

Determinism: the asserted properties are fixed; the raw attempt count varies with machine speed (by design of the watchdog), which is why the PoC asserts the property (count > 500, no give-up exception) rather than the count.

Impact

• A persistent RateLimit or outage during a long eval hangs the worker threads (default 10) indefinitely. The run never finishes, no KeyboardInterrupt-safe partial path runs (the gentle-interrupt handling is in the eval loop, which is blocked inside the completion call), and CI or scheduled runs must be killed externally.
• The dead error branch means the intended "raise APIError on error payload" behavior cannot ever happen; error semantics for this path are undefined after the SDK migration.

Suggested fix

• Add max_tries or max_time to the backoff.on_predicate decorator (for example max_time=300), and let exhaustion raise the last exception.
• Remove or fix the if "error" in result: checks in both callers: with the 1.x SDK, error responses arrive as exceptions, so the callers should only handle exceptions.

PoC path

f5_unbounded_retry.py