AzureUrlReader doesn't back off or fall back to cache on Azure DevOps rate-limit signals

Issue Labels

• Please familiarize yourself with the issue labels used in this project: LABELS.md

Search Terms

azure devops rate limit, AzureUrlReader retry-after, catalog backend x-ratelimit, ado 401 shared identity, azure devops 429 catalog

️ Project Area

Catalog

Need

We run a large Backstage catalog (35k+ entities) reading from Azure DevOps via integrations.azure with a single shared service-principal identity. Under aggregate load, ADO rate-limits this identity per its documented TSTU/resource-category scheme and responds with x-ratelimit-delay (soft-throttle, request often still succeeds) and/or Retry-After (hard back-off signal), sometimes alongside a 429, sometimes alongside a 200/203 that was simply delayed.

AzureUrlReader.readUrl() never inspects these headers, so:

• There's no backoff between calls to the same identity/host once ADO starts throttling — the catalog engine keeps calling at full rate, which in our case drove the shared SPN from soft-throttled into fully rate-limited, surfacing as 401s across thousands of entities that had nothing wrong with their credentials.
• There's no fallback to previously-read content — a throttled/rejected read becomes a hard processing error (e.g. PlaceholderProcessor / catalog status.items error) even though the file hasn't changed and was read successfully moments earlier.

Anyone running a large multi-org catalog against Azure DevOps with a single shared identity is likely to hit this same cascading-failure pattern once aggregate call volume nears ADO's per-identity budget.

Proposal

Teach AzureUrlReader.readUrl() to react to ADO's own rate-limit signals instead of only inspecting response.ok/response.status:

1. Parse x-ratelimit-delay / Retry-After off every response and track a per-host "don't call again until" deadline (per-host because the identity/budget is shared across every repo on that host, not per-URL).
2. While inside that window, skip the network call and serve the last successfully-read content for that URL from an in-memory cache instead (bounded/LRU-evicted, to keep memory usage predictable on large catalogs).
3. On a failure that came with those headers (or a 429) and a cache entry exists, fall back to cache rather than throwing.
4. Once the deadline passes, resume live reads normally; a fresh success overwrites the cached entry.

We've validated this shape as a local, out-of-tree patch (replacing the compiled AzureUrlReader.cjs.js in our deployment) and it measurably reduces cascading catalog-processing failures during a shared-identity throttling episode, without ever serving content staler than ADO's own signaled cooldown period. Happy to share the reference implementation (~90 lines) as a starting point for a real PR against @backstage/integration / @backstage/backend-defaults.

Alternatives

• Request a higher ADO rate limit for the SPN. Doesn't scale with catalog growth and depends on Microsoft support responsiveness; we filed a ticket for this and it sat unaddressed for a long time.
• Split the SPN per org/workload. Reduces blast radius but doesn't fix the underlying issue, and re-couples orgs back together the moment any one of them shares a budget again.
• Front ADO with our own caching proxy. Works, but means running and maintaining extra infrastructure just to compensate for the reader not respecting headers ADO already sends.

We think handling the signal in AzureUrlReader itself is the most direct fix and needs no extra infrastructure or config from consumers.

Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit a PR?

Yes, but I would like some more guidance