#41579 adaptive_router: thompson_sample has no alpha/beta floor — one persisted alpha=0 cell bricks the router with HTTP 500 (gammavariate) · litellm

Bug Report

Summary

The thompson_sample function in litellm/router_strategy/adaptive_router/bandit.py has no epsilon floor on the Beta shape parameters. When any cell has alpha=0 (or beta=0), random.betavariate(0, beta) raises ValueError: gammavariate: alpha and beta must be > 0.0, which crashes every request routed through the adaptive router — not just the one with the poisoned cell. The error is raised pre-deployment-selection, so no model is ever chosen.

A second defect in update_queue.py's upsert CREATE branch allows alpha=0 rows to be written to the database in the first place: it seeds the row's absolute alpha with the raw delta_alpha, and deltas are never negative (satisfaction → +1 alpha; failures → +beta), so a first failure-only flush writes alpha=0 directly.

Related issues/PRs

#35590 — closed (same crash, fixed on the load side by #39955)
#29397 — closed (same crash, fixed on the load side by #39955)
#39955 — merged (fixes load_state_from_db to add the persisted delta to a cold-start prior instead of assigning it as an absolute cell; first released in v1.102.0)

This issue covers two defects that remain unfixed on main as of 2026-09-17:

Draw-side floor (still missing): thompson_sample has no epsilon floor in ANY release, including main. The load-side fix (#39955) prevents the most common poison path (one-sided first flush), but any code path that produces a non-positive cell (e.g., a future learning update, a corrupt write, a manual DB edit) still crashes the router with a ValueError that is unrecoverable without DB intervention.
Write-side floor (still missing): The upsert CREATE branch in update_queue.py still seeds alpha with the raw delta_alpha:
```
data={
    "create": {
        "router_name": router,
        "request_type": rt,
        "model_name": model,
        "alpha": payload["delta_alpha"],   # <-- raw delta, can be 0
        "beta": payload["delta_beta"],
        "total_samples": int(payload["samples_added"]),
    },
```
A failure-only first flush (delta_alpha=0, delta_beta>0) writes alpha=0 directly to LiteLLM_AdaptiveRouterState. On the next pod restart, the loader (even with #39955's prior+delta fix) adds 0 to the cold-start prior's alpha — which is fine, but the row itself is still poison-shaped and will crash any code path that reads it as an absolute (e.g., debugging queries, monitoring tools, or a rollback to pre-#39955).

Reproduction

Environment: litellm 1.95.0, Postgres persistence enabled, adaptive-router virtual group configured.

Steps:

Start the proxy with an adaptive-router virtual group

Manually insert a poisoned row (simulating a failure-only first flush):

INSERT INTO "LiteLLM_AdaptiveRouterState" (router_name, request_type, model_name, alpha, beta, total_samples)
VALUES ('my_router', 'general', 'my_model', 0, 1, 1);

Restart the proxy (so load_state_from_db runs)
Send any request to model=my_router (the adaptive router alias)

Expected: the request routes to a model (the poisoned cell ranks last, it does not win picks) Actual: HTTP 500 ValueError: gammavariate: alpha and beta must be > 0.0 — every request, not just the poisoned cell's

Direct interpreter repro (what thompson_sample does on the loaded cell):

import random
random.betavariate(0.0, 1.0)
# ValueError: gammavariate: alpha and beta must be > 0.0

Observed poisoned rows (production, 2026-08-22 and 2026-08-26)

From a live deployment running v1.95.0 with Postgres persistence:

request_type	model_name	alpha	beta	vs. cold-start prior
general	glm-4.7-flash	0	1	should be ~(3, 7)
analytical_reasoning	glm-4.7-flash	0	1	should be ~(3, 7)
analytical_reasoning	glm-5.2	1	3	should be ~(9.5, 0.5)

13 of 17 rows had alpha=0. The adaptive-router group served 0 successful requests out of 1841 attempts over 3 weeks (no alert fired because the crash is pre-deployment-selection — the existing LitellmNoDeployments alert watches post-routing metrics only).

Suggested fix

Draw-side floor (bandit.py::thompson_sample): floor both shape params at a small epsilon (e.g., 1e-9) before calling r.betavariate:

def thompson_sample(cell: BanditCell, rng: Optional[random.Random] = None) -> float:
    r = rng if rng is not None else random
    a = cell.alpha if cell.alpha > 0.0 else 1e-9
    b = cell.beta if cell.beta > 0.0 else 1e-9
    return r.betavariate(a, b)

Semantics: Beta(eps, beta) samples ≈0 → a fully-poisoned cell ranks last (correct) instead of raising (incorrect).

Write-side floor (update_queue.py::upsert create): floor the create payload's alpha/beta so no new poison can be written:

"alpha": payload["delta_alpha"] if payload["delta_alpha"] > 0.0 else 1e-9,
"beta": payload["delta_beta"] if payload["delta_beta"] > 0.0 else 1e-9,

Apply_delta floor (bandit.py::apply_delta): clamp the learning update's result at the same epsilon so in-memory learning can never produce a non-positive cell.

Version

litellm 1.95.0 (deployed). Verified the draw-side floor is still missing on main as of 2026-09-17 by fetching bandit.py from the main branch — thompson_sample returns r.betavariate(cell.alpha, cell.beta) with no floor.

File/line references (v1.95.0)

litellm/router_strategy/adaptive_router/bandit.py — thompson_sample (line ~81): no epsilon floor
litellm/router_strategy/adaptive_router/bandit.py — apply_delta (line ~64): no clamp on result
litellm/router_strategy/adaptive_router/adaptive_router.py — load_state_from_db (line ~141): fixed by #39955 in v1.102.0, but still broken in v1.95.0
litellm/router_strategy/adaptive_router/update_queue.py — upsert CREATE branch: seeds alpha with raw delta

Impact

Severity: High — one poisoned cell bricks the entire adaptive router (every request 500s, not just the poisoned cell's)
Recovery: requires manual DB intervention (DELETE FROM "LiteLLM_AdaptiveRouterState" WHERE alpha <= 0) + pod restart; restarts alone never fix it (poison reloads from DB)
Detection gap: the crash is pre-deployment-selection, so existing deployment-level alerts do not fire