HttpRecorder silently discards logged eval events; HTTP error responses never trigger the fallback at any failure rate

PUBLIC-READY MINT R02E2-M1 (evidence-plane silent drop: computed measurement events are lost from both the remote log and the local fallback; kin to ETB-04, silent exclusion, and ETB-19, evidence channel not dependable) Target: openai/evals @ 8eac7a7de5215c907fbddc30efdaf316913eccdd File: evals/record.py, HttpRecorder._send_event and _flush_events_internal; CLI flag --http-fail-percent-threshold in evals/cli/oaieval.py

Summary

When oaieval --http-run --http-run-url URL is used, the HttpRecorder can silently lose logged events with no trace in either destination:

1. flush_events advances _written_events before delivery, so events are marked written before they are sent.
2. In _send_event, a non-OK HTTP response (for example a 500 from the ingest endpoint) only logs a warning and increments failed_requests. The batch is never retransmitted and never written to the local_fallback_path.
3. The RuntimeError that activates the local fallback is raised only inside the except Exception branch (connection-level errors), never in the non-OK-response branch. So an endpoint that answers with error status codes loses every event, at a 100 percent failure rate, with only warnings.
4. Even on the exception path, the threshold compares failed_requests (a count of failed EVENTS) against len(self._events) (all events ever recorded), and the fallback then engages only for the single batch that tripped the threshold. Earlier failed batches stay lost.

The CLI help presents the threshold as benign: "5% of total HTTP requests can fail without causing any issues".

Repro (deterministic, loopback only)

PoC: f3_httprecorder_silent_drop.py Runner: REPRO-http-recorder-silent-event-loss.sh

The PoC stands up an ingest server on 127.0.0.1 (ephemeral port) that fails any batch containing events with sample_id beginning poc.drop, records 200 match events in batches of 10, and flushes. Asserted at the pin:

• All 20 batches are attempted; no error surfaces.
• Exactly 190 events reach the server; the 10 marked events reach nowhere: not the server, not the fallback file.
• The recorder reports all 200 events as written (_written_events == 200).
• The failed fraction is exactly 5.0 percent (not above the threshold), so the fallback never engages.
• A final report answered with HTTP 500 is also lost silently, at any failure rate, because the non-OK branch cannot raise.
• Only after the server is shut down (connection errors) does the RuntimeError path engage, and it recovers only the current batch; the 10 events lost earlier remain absent from the fallback file.

Determinism: two consecutive runs produce identical assertion output.

Impact

• Anyone using --http-run to centralize eval logs (the advertised use case for this recorder) can end up with logs that are missing samples, matches, or entire final reports while the run itself completes "successfully". Post-hoc analyses computed from the shipped logs silently diverge from the reported final report.
• Because delivery failures are warnings, the operator has no failing exit status and no local copy; the loss is discoverable only by cross-checking counts against the in-memory report, which is not persisted in http mode unless the threshold trips.
• The documented 5 percent tolerance is actually a tolerance for permanent, unrecoverable data loss, and for HTTP status failures there is no tolerance boundary at all.

Suggested fix

• Treat non-OK responses as delivery failures on par with exceptions: either retry with backoff or write the batch to the local fallback immediately.
• Do not advance _written_events until the batch has been delivered or durably persisted locally.
• Compute the failure threshold over batches or requests, not events-versus-total-events, and make the default behavior fail loud (raise) rather than warn when events are being dropped.
• On fallback, also replay the events already marked failed, or record an explicit gap marker so downstream consumers know events are missing.

PoC path

f3_httprecorder_silent_drop.py