Feature: model_health_status table — per-model per-key persistent pass/fail tracking
Author: FesshompaCreated Sep 15, 2026Updated Sep 15, 2026
Summary
Adds a model_health_status table (and companion model_health_observations event log) to track per-model, per-key health persistently across restarts — currently health is only tracked at the key level (api_keys.status) and in-memory per-request failure windows that vanish on restart.
Problem
- Key-level health (
api_keys.status) tells you a key is failing, but not which specific model on that key is the problem. - The in-memory
modelFailureTimestampswindow inlib/fallback-loop.tstracks per-model failures but is lost on restart. - Operators have no persistent audit trail of when did this model start failing / what error class was last seen.
Solution
Two tables
model_health_status — current snapshot per (platform, model_id):
status:unknown|working|failing(mirrors existing health vocabulary)last_observation_at,last_working_at,last_failure_atfailure_count_in_window,window_start_ms— mirrors the in-memory window so the snapshot is self-contained- Primary key:
(platform, model_id)
model_health_observations — append-only event log:
error_class(auth, out_of_credits, rate_limited, upstream_error, etc.) +error_message- Written only on failures (not on successes) to avoid log growth on the healthy path
- Indexed on
(platform, model_id, observed_at)
Integration points
lib/fallback-loop.ts: callsrecordModelHealthFailing()fromrecordRetryableFailureandrecordModelHealthWorking()+clearModelFailure()fromrecordUpstreamSuccessroutes/health.ts: newGET /api/health/modelsendpoint returns the full snapshotlib/clear-model-failure-windows.ts: on boot,resetAllModelHealth()clears the snapshot so a restart does not carry forward stale statusdb/migrate/defaults.ts: migration20260915_000001_model_health_statusregistered
What this does NOT do (by design)
- Does not change routing/scoring logic — routing still uses key-level health + cooldowns. The model-health tables are observation/recording only. This keeps the PR focused.
- Does not bench models at the model level — that can be a follow-up once operators have visibility.
Test coverage
server/src/__tests__/services/model-health.test.ts (vitest):
- One failure writes a failing row with
failure_count_in_window = 3 - Three consecutive failures keep status failing
- Success clears failing to working
- Different models tracked separately
Migration
Reversible: down drops both tables. Already applied to the running instance (verified live).
Source: tashfeenahmed/freellmapi