Committee-correspondence follow-ups from #854: scope by determination, grade a false committee claim

Author: Imbad0202Created Sep 24, 2026Updated Sep 24, 2026

Context

In one #853 verification run of plugin-evals case 03 (ICLR reviews, claude-opus-5), the revision coach applied the #668 committee-correspondence variant to peer reviews and said the user had identified a real committee. #900 added a guard sentence in three files and a regex grader, no-committee-branch. Review raised two more changes, left out of #900 because each needs a maintainer decision. On Claude Opus 5.5 the #900 retest did not reproduce the misroute (0 of 7 runs with the guard, 0 of 7 without), so neither item has a failing run to fix on the current model.

Items

1. Scope the variant by the determination its outputs assume

  • Now: Routing Rule 7 sends a request to the variant only when the user explicitly identifies a real committee or institutional review office. The #900 guard adds that journal or conference reviewers, editors, area chairs, and program committees are peer review.
  • Change: define the variant by the kind of institutional determination its outputs assume; its schema is shared/contracts/human_subjects/committee_correspondence.schema.json. Before taking the branch, the model quotes the user's own words that name the committee.
  • Why: the guard excludes the cases it names. A definition by what the outputs assume would also cover cases the guard does not name, and the quote makes the model show where the committee came from.
  • Cost: it changes which inputs reach the variant, so a committee outside the new definition would no longer get it. Case 04 would need a new run.

2. Grade a false "you identified a committee" claim

  • Now: case 03 has four graders: skill-fired, pushback-per-reviewer, no-fabrication, and no-committee-branch. None checks whether the answer claims the user named a committee. The regex reads the variant's output lines, so it also misses a run that only announces the branch, or only writes the bundle's files.
  • Change: a with-only llm grader for case 03 that fails when the answer says or assumes the user named a committee, since the input names only a venue's reviewers.
  • Why: the misrouted #853 run stated that claim.
  • Cost: an llm rubric needs calibration against stored runs before its verdicts can be trusted (see "Judge style matters" in plugin-evals/README.md), and that means paid eval runs. The #900 eval runs cost US$6.74.

Non-goals

No change to the #900 guard text or to the variant's outputs.

Follow-up from #854.

Source: Imbad0202/academic-research-skills