Worker mode stores large non-cacheable results in one place and reports another

Written by Claude (an LLM), at Matt's request, while auditing our S3 usage. Matt has spot-checked the main claims, but please treat the details as needing a second pair of eyes. Rewritten to be readable after the first version was, fairly, called a wall of text.

The short version

In compilation-worker mode, a large result that we don't want to cache gets written to one S3 location but reported to the reader as being at a different one. Anything fetching it looks in the wrong place.

How

When a worker produces a result over 31KiB that isn't cacheable, base-compiler.ts (~3162 and ~3450) does:

await this.env.tempCachePutWithTTL(key, resultString, TEMP_STORAGE_TTL_DAYS, undefined);
result.s3Key = `temp/${BaseCache.hash(key)}`;

tempCachePutWithTTL (lib/compilation-env.ts:184) only writes to temp/ if the cache object happens to be an S3Cache:

if (this.cache instanceof S3Cache) {
    return this.cache.putWithTTLAndPath(key, ..., 'temp', creator);
} else {
    return this.cache.put(key, jsonString, creator);   // no temp/
}

Our deployed config is layered — cacheConfig=InMemory(25);S3(storage.godbolt.org,cache,us-east-1) — and createCacheFromConfig returns a MultiCache for that, not an S3Cache. So the instanceof is false, the else-branch runs, and the object lands at cache/<hash>.

But s3Key is set to temp/<hash> regardless of which branch ran. ce-router resolves it as prefix + key (result-waiter.ts:74-95, prefix defaults to cache/), so it looks for cache/temp/<hash>, which doesn't exist. Nothing under temp/ or cache/temp/ exists in the bucket at all.

The irony: drop the temp/ prefix and the fallback would work, because cache/ + bare hash is exactly where it wrote.

Second, separate problem

The fallback calls plain put with the same key cacheGet reads. So a result explicitly marked okToCache: false gets stored under the cacheable key and can be served as a cache hit until the bucket's 1-day expiry removes it.

How much this matters

Depends where worker mode is actually live. compilequeue.is_worker=true appears in amazon.properties, staging.properties and beta.properties — but Matt says the SQS compilation queue is not currently configured to run in prod, so please don't read this as a live prod incident. Beta looks like the place it would bite today. Worth confirming before anyone changes routing.

Notes for a fix

• putWithTTL/putWithTTLAndPath only set the Expires HTTP header. S3 does not delete on that; only lifecycle rules delete. So ttlDays has no effect on retention either way, which is misleading.
• cachePutWithTTL has no callers at all.
• There's no test coverage here. A regression test must build its cache via createCacheFromConfig with a production-shaped config string — a directly constructed S3Cache passes while our real config fails, which is likely how this shipped.