Reviewer paired controls: author-identity cues (affiliation, seniority) must not move findings, severity, or decision (paper-derived: arXiv:2609.07713 §7.2)

Author: Imbad0202Created Sep 15, 2026Updated Sep 15, 2026
Labelsenhancementresearchpaper-derivedstatus/needs-design

Motivation (paper-derived)

Wang, Li et al. (2026), The Emerging AI Paper-Review Arms Race (arXiv:2609.07713v1), §7.2 reports that controlled studies find LLM reviewers respond to evaluator-irrelevant author information such as institutional affiliation, seniority, or publication history (Vasu et al. 2026; experimentally reproduced, on other reviewer systems). The survey files this under "redistributed error": automation can move bias rather than remove it.

For an author-side panel this is an output-quality question, not a venue-fairness question: if the panel is more lenient toward a prestigious author block, the author receives softer revision guidance; if harsher, unnecessary extra work.

What ARS already has (checked on main at b06ceaf)

  • A prompt-level rule only: academic-paper-reviewer/references/review_criteria_framework.md row "Prestige bias: Evaluate the manuscript evidence, not institutional or author status."
  • #394 (submission-package verifier) checks the author's own package for blind-review residue. That is the opposite direction from this issue, which asks how the panel reacts to identity cues in its input.
  • No held-out set varies the author block while holding the manuscript fixed.

So this is a measurement gap; the ethical rule already exists.

Proposed shape: one held-out paired-control seed, no runtime change

A seed under evals/heldout/ reusing an existing synthetic manuscript (e.g., reviewer_seeded_defects/manuscripts/ms01_quant_defective.md) with the author block as the only variable:

  • arm-a: fictional high-prestige institution and senior-sounding author line;
  • arm-b: fictional unknown institution and early-career author line;
  • arm-c: no author block (title only).

Everything else, including the planted defects, is byte-identical. Fresh-context dispatch per arm, replicates per the #574 E4 discipline. Observables: the set of findings, severity per finding, remedy specificity, and the decision. Expectation: invariant across arms. A heldout-measurement report per evals/heldout/MEASUREMENT_CONTRACT.md; measure the current model first.

Possible follow-up (not proposed now)

If the measured sensitivity is non-trivial, a later issue may design an intake-time author-block redaction with a Review Panel Provenance field recording whether the block was stripped. That is a mechanism decision for after the measurement, not part of this issue.

Non-goals

  • No new fairness-governance framework; GOVERNANCE.md already scopes what the simulated panel's outputs are.
  • No claim that the ARS panel has this bias before the seed is run.

Relations

  • Companion to #871 (same harness, same fixtures; can share the equivalence-attestation format).
  • Source: dual read of the paper on 2026-09-15 (Claude Fable 5.1 and Codex gpt-6-astra, independent reads, then compared).

Source: Imbad0202/academic-research-skills