#1234·freellmapi

Feature: model_health_status table — per-model per-key persistent pass/fail tracking

Author: FesshompaCreated Sep 15, 2026Updated Sep 15, 2026

Summary

Adds a model_health_status table (and companion model_health_observations event log) to track per-model, per-key health persistently across restarts — currently health is only tracked at the key level (api_keys.status) and in-memory per-request failure windows that vanish on restart.

Problem

  • Key-level health (api_keys.status) tells you a key is failing, but not which specific model on that key is the problem.
  • The in-memory modelFailureTimestamps window in lib/fallback-loop.ts tracks per-model failures but is lost on restart.
  • Operators have no persistent audit trail of when did this model start failing / what error class was last seen.

Solution

Two tables

model_health_status — current snapshot per (platform, model_id):

  • status: unknown | working | failing (mirrors existing health vocabulary)
  • last_observation_at, last_working_at, last_failure_at
  • failure_count_in_window, window_start_ms — mirrors the in-memory window so the snapshot is self-contained
  • Primary key: (platform, model_id)

model_health_observations — append-only event log:

  • error_class (auth, out_of_credits, rate_limited, upstream_error, etc.) + error_message
  • Written only on failures (not on successes) to avoid log growth on the healthy path
  • Indexed on (platform, model_id, observed_at)

Integration points

  • lib/fallback-loop.ts: calls recordModelHealthFailing() from recordRetryableFailure and recordModelHealthWorking() + clearModelFailure() from recordUpstreamSuccess
  • routes/health.ts: new GET /api/health/models endpoint returns the full snapshot
  • lib/clear-model-failure-windows.ts: on boot, resetAllModelHealth() clears the snapshot so a restart does not carry forward stale status
  • db/migrate/defaults.ts: migration 20260915_000001_model_health_status registered

What this does NOT do (by design)

  • Does not change routing/scoring logic — routing still uses key-level health + cooldowns. The model-health tables are observation/recording only. This keeps the PR focused.
  • Does not bench models at the model level — that can be a follow-up once operators have visibility.

Test coverage

server/src/__tests__/services/model-health.test.ts (vitest):

  • One failure writes a failing row with failure_count_in_window = 3
  • Three consecutive failures keep status failing
  • Success clears failing to working
  • Different models tracked separately

Migration

Reversible: down drops both tables. Already applied to the running instance (verified live).

Source: tashfeenahmed/freellmapi