OEP-0004: Close the reviewer learning loop

Summary

OEP-0004 proposes closing the loop between human feedback and the reviewer's per-repository learning. The outcomes dataset already exists, but several feedback paths do not currently reach it, so the analyzer learns from a partial picture.

Background

PR #1365 added the outcomes dataset and wired emission points into update_finding, resolve_finding_thread, and the GitHub reaction handlers. GitHub and Slack reactions were explicitly listed as learning signals. In practice, several of those paths are disconnected today.

Gaps

1. GitHub reactions never reach the app. Reactions are omitted from the supported webhook event list, so GitHub deliveries are rejected before dispatch and the handlers that exist for them have no callers.

2. Human thread resolutions are silent. When a person resolves a finding's review thread on GitHub, the reconciliation that runs in response updates the finding's status but records no outcome. Only the agent's own tools currently emit, which means the most common human action teaches nothing.

3. Outcome examples are not unique per finding. The example identity includes the label source alongside the finding, so a single finding can accumulate multiple examples, and reading outcomes can return the same finding as both confirmed and dismissed.

4. Recorded commit SHAs misclassify resolutions. The SHA written with an outcome is taken from the run configuration rather than the HEAD observed at resolution, so a resolution that involved no new commit can be labelled as fixed by a commit.

5. Continual runs do not receive the prompt they are asked to refine. The continual analyzer is told to edit the current repository prompt, but the stored prompt is never placed in its context.

Proposal

Record exactly one outcome for every terminal finding transition, no matter which actor caused it: an agent tool, webhook reconciliation, or a reaction. Key each example by repository and finding, keeping the most recent transition, and record the SHA observed at resolution so the fixed-by-commit signal stays accurate. Subscribe to reaction webhooks behind the existing repository allowlist and organization gate, ignore bot senders, and never surface missing findings or lookup failures as webhook errors. Finally, pass the stored prompt and its analysis summary into continual runs so the analyzer refines what exists instead of rebuilding it.

Open questions

• When a reaction is removed, should the previously recorded outcome be deleted or replaced with a neutral one?
• Is an explicit "won't fix" dismissal distinct from a thumbs-down false positive, or should both be treated as dismissals?
• Should existing per-label-source examples be migrated, or overwritten naturally as new transitions occur?

Status

This proposal remains a Draft. Publishing it neither accepts the design nor authorizes implementation. Ongoing design feedback is welcome here.