Feature: observed capability scoring — learn tool/vision support from live traffic, not catalog declarations

Summary

Add observed capability scoring to the bandit layer — track, per model+key, whether the endpoint actually supports tool calling and vision input in practice, not just whether the catalog claims it does.

This closes a gap surfaced by #1230: a user's trail showed provider_bad_request from every candidate model (kimi-k2-thinking, deepseek-v3.2-think, GLM-5.3) when making a tool-calling request. The catalog has supports_tools=true for all three, but they silently reject tool calls — the router keeps offering them because the bandit only sees "failed with bad request" and applies a generic penalty, not "this model doesn't support tools". The next tool call request hits the same three, again and again.

Why catalog declarations are unreliable

Free-tier aggregators and custom relays frequently advertise capabilities based on:

• Generic provider claims ("this model family supports function calling") rather than实测 behavior
• Copy-pasted metadata without verification
• Partial support (e.g., supports simple JSON but not structured tool calling)

The result: a model with supports_tools=true in the catalog may return 400 on every tool-call attempt, or silently drop the tools field and produce garbage. The router has no way to tell the difference from the failure class alone.

Proposed mechanism

Data model

Add two rolling counters per (platform, model_id, key_id) bucket — the same granularity the existing reliability/speed axes already use:

Failure classification (new)

When a request with tools fails, classify the failure into one of:

• tool_incompatible — 400 body mentions "tool_calls", "function_call", "unsupported tools", "function calling not supported", or the response contains no tool_calls field despite a tools block in the request
• vision_incompatible — 400 body mentions "image", "vision", "multimodal" + rejection, or 400 on image payload
• generic_bad_request — falls through to existing handling

The classifier reads the error message AND (for 200 responses) inspects the response body for presence/absence of tool_calls.

Scoring impact

The new counters feed a capability confidence factor that multiplies into the base reliability score — like the existing headroomFactor:

capabilityFactor = clamp(observedToolFailureRate * weight, 0.2, 1.0)
effectiveScore = baseScore * capabilityFactor

For requests WITHOUT tools, capabilityFactor = 1.0 — tool-incompatible models are not penalized for pure-chat usage. This is the key differentiator from a blanket reliability penalty.

Defaults

• weight = 0.3 — a model that fails tool calls 100% of the time runs at 70% of its normal score on tool requests; on pure-chat it stays at 100%
• Observation window = 7 days (same as reliability), with the same 2-day half-life decay
• Exploration floor = 5% for unmeasured models (prevents starvation while gathering evidence)

Configuration

• cap_observe_enabled (default: true) — kill switch
• cap_tool_failure_weight (default: 0.3) — how harshly to penalize tool incompatibility
• cap_vision_failure_weight (default: 0.5) — vision is rarer, penalize more aggressively
• Per-model overrides via existing MODEL_ROUTING_OVERRIDES mechanism

Implementation notes

• The classifier should be added to lib/error-classify.ts alongside existing helpers
• Counters live in the existing model_stats aggregate table (no new schema needed — extend the shape)
• The dashboard GET /api/fallback/scores already exposes per-model reliability/speed/intelligence — add toolCapability and visionCapability axes
• Integration with existing PR #1256 endpoint health state machine: tool-incompatible failures are structural at the model level (not the endpoint level), so they feed the bandit, not the quarantined endpoint

Expected impact

For the #1230 scenario: once kimi-k2-thinking accumulates enough tool-fail observations, its effective score on tool requests drops below the threshold where the router would pick it over an unknown-but-never-tried model that actually supports tools. The router self-corrects without manual intervention.

PR plan (one concern per PR)

1. Failure classifier extension: tool_incompatible / vision_incompatible classes + observation counters in model stats
2. Capability confidence factor: multiplicative guardrail on bandit scores, gated by request context (only applied when tools/images present)
3. Dashboard visibility: add capability axes to the routing scores view