Inline cache export drops a usable remote and exports records with no results

Opening this as a fresh issue as asked in https://github.com/moby/buildkit/issues/3730#issuecomment-5660901514.

Two defects in the v1 cache exporter make a cache record that has a usable remote get exported with no results at all. A record without results breaks the key chain in the exported manifest, so the next build importing that manifest misses, rebuilds everything locally, and then exports a complete manifest again — which is the "cache works every other build" symptom reported in #3730 and #2274.

On the previous closure: #4796 fixes cache/remotecache/inline/inline.go, the uncompressed-digest provider-linking path used by the Moby adapters without the containerd backend. It does not touch cache/remotecache/v1/chains.go or cache/remotecache/v1/utils.go, where both defects below live, and both are present on master today (9bc3354b43dd7a5cb70984fabf6a2730afa58038).

On the requested docker version + build sequence: I do not have one. I could not reproduce the end-to-end alternation locally — it needs a compression variant whose digest differs from the main remote and a lazy blob in the same chain, and I could not construct that combination (force-compression disables the variants, an imported zstd cache stays lazy). I tried the docker-container driver, dockerd with the containerd image store, and dockerd on overlay2; the manifest was stable in all of them. What I have instead is a deterministic unit-level reproduction against master, below.

Where the extra remotes come from

A cache record can carry several remotes. In solver/exporter.go, when CompressionOpt is set — always the case for the inline cache exporter — the compression variants of a chain are appended to results before the main remote. Some of those variants come from getAvailableBlobs in cache/remote.go, which attaches the plain content store as their provider; for a blob that is still lazy, that provider cannot resolve the descriptors.

Defect 1 — item.addResult treats distinct remotes as duplicates

https://github.com/moby/buildkit/blob/9bc3354b43dd7a5cb70984fabf6a2730afa58038/cache/remotecache/v1/chains.go#L345-L365

for i, d := range rr.Result.Descriptors {
    if d.Digest != r.Result.Descriptors[i].Digest {
        continue
    }
}
exists = true

The continue applies to the inner loop, so exists = true is reached unconditionally: the descriptor comparison never rejects a candidate. Any two results with the same CreatedAt and the same number of descriptors are treated as duplicates. Compression variants of a chain have exactly that shape, so only the first one added survives and the main remote is dropped.

Defect 2 — marshalItem gives up after bestResult()

https://github.com/moby/buildkit/blob/9bc3354b43dd7a5cb70984fabf6a2730afa58038/cache/remotecache/v1/utils.go#L213

marshalItem marshals only bestResult(). When marshalRemote rejects it — its provider cannot Info the descriptors — it returns "" and the record is exported with no results at all, silently, instead of trying another result of the same record.

Reproduction

The unit tests in #7139 (cache/remotecache/v1/marshal_test.go) reproduce both defects deterministically. Against a clean checkout of master with only that test file applied:

$ go test ./cache/remotecache/v1/ -run 'TestAddResult|TestMarshal' -v
--- FAIL: TestAddResultKeepsDistinctRemotes (0.00s)
    marshal_test.go:71: "[{... 0x6e47bf630510  0}]" should have 2 item(s), but has 1
--- FAIL: TestMarshalFallsBackToUsableRemote (0.00s)
    marshal_test.go:90: "[]" should have 1 item(s), but has 0
--- PASS: TestMarshalNoUsableRemote (0.00s)
--- PASS: TestMarshalPrefersNewestUsableRemote (0.00s)
FAIL

The first is defect 1: a usable remote silently discarded as a duplicate. The second is defect 2: a record exported with no layer result even though a usable remote was available.

Fix

#7139 — fix the descriptor comparison so distinct remotes are kept, and let marshalItem fall through to the next candidate when a remote cannot be marshalled. No interface changes.

This does not address why a remote ends up with a provider that cannot resolve its descriptors in the first place (see #5595 and @sipsma's analysis in https://github.com/moby/buildkit/pull/5560#issuecomment-2521716555); it makes the exporter stop discarding the good remote it already has.