/api/compilers is 12.6MB for C, 84% of it duplicated possibleOverrides

Summary

GET /api/compilers/c with the field set the frontend requests returns 12.6 MB (13,243,930 bytes) of JSON for 1038 compilers. 84% of that is possibleOverrides, and almost all of it is the same data repeated verbatim.

This is worth fixing on its own merits, but it also has a concrete failure mode in the wild — see "Why this matters" below.

The numbers

Measured against production (https://godbolt.org, 2026-08-21), using exactly the field list CompilersService.compilerFields sends:

Now the duplication. Across all 1038 compilers there are 1234 override objects, but only 70 distinct ones, totalling 200,656 bytes. The single worst offender:

{"type":"options","name":"toolchain","display_title":"Toolchain",…,"values":[…640 entries…]} — 63,380 bytes, byte-identical, repeated across 149 compilers = 9.44 MB, i.e. 71% of the entire response.

That one comes from getPossibleToolchains() (lib/base-compiler.ts:4315), which delegates to this.env.getPossibleToolchains() — an environment-wide list. It is by construction the same for every compiler that gets it, and we ship 149 copies.

arch is the same story at smaller scale (~5 KB × 121 values, repeated per compiler in a group); stdver appears on 937 compilers.

Rough ceilings:

• dedupe whole possibleOverrides arrays (80 distinct values): 12.6 MB → ~4.3 MB
• dedupe individual override objects (70 distinct): 12.6 MB → ~2.3 MB (an 82% cut)

Why this matters

Beyond the obvious (bandwidth, parse time, memory on mobile), we have a Sentry event that this size makes more likely:

SyntaxError: JSON.parse: expected ':' after property name in object
                         at line 1 column 1441800 of the JSON data
SentryCapture Context: fetchCompilersForLang(c)
GET /api/compilers/c?fields=…&hash=… [200]

Analysis of that event:

• Not a server bug. outputList (lib/handlers/api.ts:228) is a single synchronous res.send(array) → JSON.stringify. Nothing can interleave. Prod confirms: ETag W/"ca161a-…" (0xca161a = 13,243,930 = exact body length), and 25 consecutive fetches came back byte-identical and strictly valid (no NaN/Infinity, compact round-trip identical).
• Not truncation. SpiderMonkey distinguishes end of data after property name when ':' was expected from expected ':' after property name in object at … column N. We got the second, which only fires when there is a character there and it isn't :. So the body arrived complete-looking with garbage in the middle.
• The offset is the tell. Column 1441800 → 0-based index 1441799, and 1,441,792 = 22 × 65536. The corruption is 7 bytes past a 64 KiB boundary — about a 1-in-8000 coincidence. That's the signature of a dropped/duplicated buffer chunk in the transport chain (nginx → CloudFront → intermediary → Firefox disk cache), not anything content-dependent.
• Nothing catches it. The response goes out HTTP/2 with content-encoding: br and no content-length, so a corrupt-but-complete body can't fail a length check. And it's cache-control: public, max-age=600, so Firefox persists 12.6 MB to disk cache — a poisoned entry replays the same broken parse for the full TTL.

So: a rare transport glitch we don't control, whose probability scales with payload size. Making the payload 5× smaller is the honest mitigation.

Constraints

No breaking changes, backwards or forwards. In particular:

• ?fields=all must keep returning exactly today's shape — lib/compiler-finder.ts:81 uses it to discover compilers on remote CE instances, and those instances run arbitrary older/newer versions of CE.
• A client asking for fields=…,possibleOverrides,… must keep getting today's inline shape forever. Stale cached frontend JS will keep asking for it long after any deploy.
• Third-party API consumers exist. Nothing that currently works may stop working.
• Whatever we add, old clients must tolerate it (they ignore unknown fields) and new clients must tolerate its absence (talking to an older remote instance).

That means the change has to be opt-in and additive, never a change in meaning of an existing field.

Possible approaches

Not decided — this is the part that needs design discussion.

1. New sibling field, same endpoint. Client asks for possibleOverridesRef (or similar) instead of possibleOverrides; response gains a shared table. Problem: the endpoint currently returns a bare array, so there's nowhere to hang a sidecar table without changing the top-level shape — which would break fields=all consumers unless gated behind another param.
2. New endpoint. e.g. GET /api/overrides/:language returning the deduped table, with compilers referencing entries by id. Cleanest separation, costs an extra round trip, and needs a story for how a client knows whether the endpoint exists (404 → fall back to inline).
3. Group-level hoisting. Overrides are already largely a property of the compiler group, and toolchain is a property of the environment. Serving them at that granularity is arguably the correct model rather than a compression trick.
4. Opt-in response envelope. ?format=2 (or an Accept variant) switches to {compilers: [...], overrides: {...}}. Old clients never pass it and are untouched.

Orthogonal cheap wins, independent of the above:

• fullVersion (2.0%) is version + "\n" for most compilers — omit when derivable.
• possibleRuntimeTools (4.1%) is also near-identical across compilers and dedupes the same way.

Also worth doing regardless (small, self-contained)

• fetchCompilersForLang (static/services/compilers.service.ts) does a bare await response.json(). Read .text(), try/catch the parse, and on failure retry once with cache: 'reload'. That specifically rescues the poisoned-browser-cache case, which otherwise sticks for the full 10-minute TTL.
• Enrich the Sentry capture with body length, the ±80 chars around the reported column, and x-cache / x-amz-cf-id / content-encoding. Today the event says a parse failed and nothing about what actually arrived; one enriched event would settle cache-vs-proxy immediately.