#972·llmfit

test: per-chip-class estimator accuracy CI gate (~25% median)

Author: saman-mbCreated Aug 29, 2026Updated Aug 30, 2026
LabelsenhancementTesting

Context

Follow-up from #969 / PR #971. The MoE Tier-2/Tier-1 fixtures and the pre-existing leaderboard replay land, but the issue asked for an offline harness replaying data/benchmark_cache.json + data/community/ with a per-chip-class ~25% median error CI gate. That gate was not added (today's overall replay band is still deliberately loose [0.5, 2.0]).

Ask

  • Add llmfit-core/tests/estimator_accuracy.rs (or extend the existing replay) asserting median |error| ≤ ~25% per chip class
  • Record baseline medians in the PR that lands it; allow per-class exceptions if a class is already above 25% today rather than turning main red
  • Keep hermetic (synthetic SystemSpecs / embedded cache only)

Related: #969, #971