ce-retune: distinguish one streak's probability from evidence after candidate selection

Author: 22nsukCreated Sep 13, 2026Updated Sep 13, 2026

skills/ce-retune/references/noise-floor.md describes p^N as an exact test after estimating the baseline from A/A and archive runs. It also permits another streak after changing a failed candidate. The user guide repeats the one-armed-test claim.

For a fixed known p, independent runs and one pre-specified attempt, the calculation is valid. It is not automatically a significance level for a selected winner after repeated attempts, and an independently estimated baseline still has uncertainty.

A numerical counterexample (not measured CE performance): with p=0.58 and N=8, one planned all-success attempt has probability 0.0128063. Across 20 independent attempts under that same null, the probability of at least one success is 1-(1-p^N)^20 = 0.227235. Changing the candidate does not erase earlier selection opportunities.

Keep the operational streak bar, A/A measurement, fixed-build confirmation and stop-on-failure behavior. Qualify the statistical interpretation, retain all attempts, and distinguish candidate-selection evidence from a final check whose results were not used to select the candidate. When the assumptions or overall error control are unsupported, report the observed confirmation without an exact-significance claim.

Scope: the streak guidance and its user-facing explanation only; no new statistical engine, dependencies, fixed sample minimum, or changes to ce-optimize.

Source: EveryInc/compound-engineering-plugin