test: per-chip-class estimator accuracy CI gate (~25% median)
Author: saman-mbCreated Aug 29, 2026Updated Aug 30, 2026
LabelsenhancementTesting
Context
Follow-up from #969 / PR #971. The MoE Tier-2/Tier-1 fixtures and the pre-existing leaderboard replay land, but the issue asked for an offline harness replaying data/benchmark_cache.json + data/community/ with a per-chip-class ~25% median error CI gate. That gate was not added (today's overall replay band is still deliberately loose [0.5, 2.0]).
Ask
- Add
llmfit-core/tests/estimator_accuracy.rs(or extend the existing replay) asserting median |error| ≤ ~25% per chip class - Record baseline medians in the PR that lands it; allow per-class exceptions if a class is already above 25% today rather than turning
mainred - Keep hermetic (synthetic
SystemSpecs/ embedded cache only)
Related: #969, #971
Source: AlexsJones/llmfit