#1039·llmfit

Weekly model refresh fails since 2026-08-31: HF 429 rate limiting degrades the scrape and trips the #963 guard

Author: AkcialiCreated Sep 14, 2026Updated Sep 16, 2026
LabelsbugAutomation

@AlexsJones flagging this since it is easy to miss.

The weekly-model-update workflow fails at "Refresh model database" on its last three scheduled runs (#11 on 08-31, #12 on 09-07, #13 on 09-14: https://github.com/AlexsJones/llmfit/actions/runs/34798219787), each after about 4m30. Last green run: #10 on 08-24 (PR #945). No automated PR since, so it is easy to miss. The update-benchmarks job is fine.

Tail of run #13:

Merged with existing database (12937 models):
  Updated: 4716, Retained historical: 8221
  Architecture heads restored from prior catalog: 1759
ERROR: config.json fetch dropped attention-head metadata for 1759 models (limit 25). Prior values were kept, but refusing to ship this scrape so the mass drop cannot land quietly again.

The guard from #963 is doing its job. The cause is HuggingFace rate limiting: every discovered model in the log comes out as 0 downloads, ctx 4096, which is what the scraper yields when fetch_model_info and fetch_config_json fail.

Reproduced outside CI (--threads 8 --no-gguf-sources, no token): -n 200 on a fresh quota gives 26 HTTP 429 and exit 0; -n 1000 right after, quota exhausted, gives 192 HTTP 429 out of 192 curated fetches, then ERROR: config.json fetch dropped attention-head metadata for 76 models (limit 25) and exit 1. Same signature.

HF states the policy in the 429 response: ratelimit-policy: "fixed window";"api";q=500;w=300, with ratelimit: "api";r=<remaining>;t=<reset>. The scraper makes two to three requests per model (info, config.json, and a second info fetch per discovered model since the listing omits likes), so -n 5000 --threads 8 is on the order of 13,000 requests in 4m30, above any per-window quota even with HF_TOKEN. Nothing in the script handles 429 or reads Retry-After: fetch_model_info logs "skipping", fetch_config_json swallows every exception, so a burst silently zeroes downloads, resets context and drops architecture fields. The green runs after the 08-03 -n 5000 --threads 8 change may have shipped degraded data the same way.

Not proposing a fix, this is your scraper and guard. Reporting it because it also gates the release-date backfill in #176. Repro logs available if useful.