`agent-scan` false positive: #36085 graded `automated` despite disclosed AI assistance and full proof of work
cc @huang-julien @Sidnioulz
The issue:
Our PR triage treats the agent-scan:* label as the single source of truth for AI-generated handling and deliberately never second-guesses it.
That contract only works if disagreements between the scan and the PR's actual content flow back to the scanner.
Here is the first clearly documented one.
#36085 (author: @ethanstoner, new contributor) was labeled agent-scan:automated by the workflow on 2026-08-30.
Against that grade, the PR body contains:
- both required template sections, filled in
- a 4-step "Manual testing" section with the exact sandbox commands to reproduce
- the real pre-fix failing-test output (snapshot mismatch, quoted verbatim)
- the exact commands that were run
- an explicit AI-disclosure statement, exactly as CONTRIBUTING asks for:
Per CONTRIBUTING's AI policy: this PR was written with AI assistance (Claude). A real person (me) is behind it, has reviewed the diff, and will respond to review comments.
A disclosed AI-assisted session is what agent-scan:mixed exists to describe, and triage explicitly does not treat mixed as AI-generated.
I would argue this one should have been graded mixed.
The cost:
Because of the automated grade, the author received the proof-of-work verification request, got the verification:pending label, and landed in the "Human verification" board column - for proof the PR already contained.
Not a great first-contribution experience.
Questions:
- Is this tunable on our side (workflow config), or does it need to go upstream to
MatteoGabriele/agentscan-action(we pin v1.8.0)? - Can we use this issue as the standing collection point for future scan-vs-content disagreements from triage? Triage already records them in its run reports; they just had nowhere to go until now.
Happy to pull more cases from the triage run reports if useful.
WDYT?
Source: storybookjs/storybook