Proposal: noise-aware repeats and A/B replay for the experiment loop

Summary

I'd like to gauge interest in upstreaming noise-aware repeats and A/B replay for the experiment loop. Both target the same problem: distinguishing real wins from benchmark jitter on noisy signals (ML training, Lighthouse, microbenchmarks, anything thermal/time-sensitive). The confidence score already estimates a noise floor; these features let the agent act on it by gathering more samples and re-running marginal candidates under controlled conditions.

I have a working implementation (with tests) on my fork and am happy to open a focused PR rebased on current main if this aligns with the project's direction.

Motivation

The current confidence score is advisory — it flags when the best improvement is within the noise floor, but the agent has no first-class way to reduce that noise. Today it can only eyeball a single sample per run. For fast/jittery benchmarks a single measurement is often inside the noise band, so the keep/discard decision is effectively a coin flip.

Proposed additions

1. Noise-aware repeats (run_experiment)

• New optional repeats parameter on run_experiment; alternatively autoresearch.sh can emit multiple METRIC name=value samples.
• The harness aggregates samples per metric and reports median, standard deviation, and 95% CI.
• log_experiment persists the repeat statistics on the run entry for post-hoc analysis.

2. A/B replay (replay_experiment)

A new gated experiment tool for marginal/discarded candidates that may have lost to noise:

• Takes explicit baseline_ref and candidate_ref (git refs).
• Creates temporary git worktrees so the active worktree is never disturbed.
• Optional setup_command run in each worktree.
• Runs an alternating sequence (default ABBA) to cancel warmup/thermal/time-drift bias.
• Compares medians and 95% CIs, warns when intervals overlap ("within noise").
• Appends a type: "replay" entry to autoresearch.jsonl.

Question for maintainers

1. Is this in scope for the project, or would you prefer a slimmer cut (e.g. repeats only, replay later)?
2. Any preferences on the tool surface — parameter names, default sequence (ABBA vs. configurable), or where replay stats live in the JSONL schema — before I open the PR?

Happy to adjust to match conventions. Thanks!