#36130·storybook

`agent-scan` false positive: #36085 graded `automated` despite disclosed AI assistance and full proof of work

Author: valentinpalkovicCreated Sep 2, 2026Updated Sep 14, 2026

cc @huang-julien @Sidnioulz

The issue: Our PR triage treats the agent-scan:* label as the single source of truth for AI-generated handling and deliberately never second-guesses it. That contract only works if disagreements between the scan and the PR's actual content flow back to the scanner. Here is the first clearly documented one.

#36085 (author: @ethanstoner, new contributor) was labeled agent-scan:automated by the workflow on 2026-08-30. Against that grade, the PR body contains:

  • both required template sections, filled in
  • a 4-step "Manual testing" section with the exact sandbox commands to reproduce
  • the real pre-fix failing-test output (snapshot mismatch, quoted verbatim)
  • the exact commands that were run
  • an explicit AI-disclosure statement, exactly as CONTRIBUTING asks for:

Per CONTRIBUTING's AI policy: this PR was written with AI assistance (Claude). A real person (me) is behind it, has reviewed the diff, and will respond to review comments.

A disclosed AI-assisted session is what agent-scan:mixed exists to describe, and triage explicitly does not treat mixed as AI-generated. I would argue this one should have been graded mixed.

The cost: Because of the automated grade, the author received the proof-of-work verification request, got the verification:pending label, and landed in the "Human verification" board column - for proof the PR already contained. Not a great first-contribution experience.

Questions:

  1. Is this tunable on our side (workflow config), or does it need to go upstream to MatteoGabriele/agentscan-action (we pin v1.8.0)?
  2. Can we use this issue as the standing collection point for future scan-vs-content disagreements from triage? Triage already records them in its run reports; they just had nowhere to go until now.

Happy to pull more cases from the triage run reports if useful.

WDYT?