Reviewer calibration corpus: OpenReview camera-ready replacement leaks the accept label (blocks #653 first scored run)

Summary

The #653 corpus freeze completed on 2026-09-06 (12 ICLR 2026 papers, pre-registered seed, verify PASS), and the first cross-model review of the branch found that the frozen corpus cannot serve as a calibration gold set: the PDF layout itself reveals the label. This issue records the finding, the paths that were probed and ruled out, the branch state, and the decision that is still open.

Finding

OpenReview replaces an accepted ICLR paper's PDF with the camera-ready revision. Rejected papers keep the submission PDF. Verified on the frozen corpus and on a fresh 30-paper sample of the accepted pool:

	accepted (6/6 frozen, 30/30 sampled)	rejected (6/6 frozen)
page-1 header	Published as a conference paper at ICLR 2026	none
author block	named authors	Anonymous authors
line numbers	none	present (000, 001, …)
content	post-review revision	submission-time text

Any seat reading the first line can infer the decision, so an FNR/FPR profile measured on this corpus would not be a measurement of reviewer judgement. The same asymmetry holds for NeurIPS 2025 (accepted: named authors; rejected: anonymous), so it is a property of OpenReview's camera-ready workflow, not of one venue.

Paths probed and ruled out (2026-09-06, authenticated account)

• GET /notes/edits?note.id=<accepted id> (and with the Revision / Camera_Ready_Revision invitations): 0 edits — submission-time revisions are not readable by an ordinary account. Same for TMLR.
• API v1 references?referent=<id>&original=true: 404 for API-v2 venues.
• Anonymous requests: Cloudflare challenge.

The only clean route is to capture submission PDFs before decisions are released and attach labels afterwards. Nearest window: ICLR 2027 (submissions public ~2026-10, decisions ~2027-01).

Options and decision (2026-09-06)

• A. Wait for ICLR 2027: capture submission-time PDFs in October, freeze labels in January. The only route to a publishable profile. Cost: the first measured profile arrives after 2027-01.
• B. Harness dry-run on the current corpus, no numbers published: exercise the 228-call dispatch, scoring, and the heldout-measurement/1.1 envelope end to end, labelled as a harness rehearsal. Can be combined with A.
• C. Strip layout cues and publish with a caveat: not recommended — the accepted-side text is still the post-review revision, so the profile would be biased toward the panel looking accurate, against the suite's own claims discipline.

Decision: A + B. The frozen 2026-09-06 manifest stays in the PR marked SUPERSEDED — layout leaks the label; harness rehearsal only; the ICLR 2027 submission-time capture becomes the gold corpus. RUN_PLAN subject/judge ids move to the current generation before first dispatch (no amendment needed pre-dispatch).

Branch state

653-reviewer-calibration-first-run (pushed, no PR yet), rebased on main, 6 commits, 52 tests green, real-corpus verify PASS:

• corpus freeze (corpus/papers.json label-free, manifests/gold_labels.json), scripts/fetch_calibration_corpus.py (authenticated operator tool), scripts/_calibration_pdf_text.py (shared hashing; pypdf emits lone UTF-16 surrogates from math fonts — 61 in one manuscript — so the normalization rule is recorded in the manifest and pinned by verify and the dispatcher);
• review fixes already applied: dispatch/verify enforce the same manuscript invariants (page count, normalization rule), frozen cards pass the same plain-file guard as PDFs (a symlinked card could previously reach gold_labels.json), scorer refuses duplicate panel records and gold papers without a complete ensemble, adjudication overrides require the verbatim raw excerpt, transport-failure artifacts keep partial stdout/stderr, every call records RFC-3339 timing and prompt/output hashes;
• the numeric score axis was removed entirely: the seat contract is categorical and calibration_mode_protocol.md Phase 2 forbids AUC; the scorer now reports the protocol's full-tier table plus categorical replicate stability;
• RUN_PLAN/README now target heldout-measurement/1.1 (1.0 is closed to new rows since #664); the pre-registration record and execution-manifest builders are still to be written.

Open items before the first scored run

• Decide A / B / C — A + B; the frozen manifest stays as a marked rehearsal artefact.
• RUN_PLAN model currency (subject and judge ids) updated on the branch.
• heldout-measurement/1.1 pre-registration record + execution manifest emission — --stage manifest + scripts/build_calibration_measurement_row.py (branch @ 6094b11, 2026-09-06).
• Second cross-model review round after the corpus decision — done 2026-09-06 (gpt-6-astra xhigh; 12/13 applied on the branch).
• Schedule the ICLR 2027 submission-time capture. (The layout-tell check in freeze / verify landed 2026-09-06 @ 4a0d835; the capture schedule is still open.)

Blocks #653.