Embedded registry: peers advertise layers they do not have, causing 404s and > stalled pulls

Environmental Info:

K3s Version:

RKE2 v1.35.7+rke2r1 (Kubernetes v1.35.7+rke2r1), containerd 2.2.6-k3s1

The affected code is in k3s (pkg/spegel), which RKE2 vendors.

Node(s) CPU architecture, OS, and Version:

Linux 7.1.8-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
Fedora CoreOS 44

Cluster Configuration:

3 servers, 3 agents on the lab cluster where this was investigated. Originally seen on a 12-node airgapped cluster with a Harbor mirror, where it is far more visible.

Describe the bug:

With embedded-registry: true, nodes advertise layer digests they do not actually hold. Peers resolved to them get a 404, and the pull stalls while the client works through bad candidates.

The cause is oci.WithContentEvents(false) at pkg/spegel/spegel.go:188. Spegel defaults this to true (ContainerdConfig{ContentEvents: true}). With content events enabled, a layer is advertised when containerd emits /content/create, i.e. once the blob is committed. With them disabled, advertisement comes from /images/create, which walks the manifest and advertises every child digest.

That walk cannot detect a missing layer. images.Children() only reads content for manifest and index media types; for a layer it returns nil, nil without touching the content store, so the errdefs.ErrNotFound branch that handleEvent relies on is never taken and the digest is advertised regardless.

Upstream describes exactly this in spegel-org/spegel@a348813, "Do not advertise content missing from the content store":

The walk does not read leaf content like layers, so their existence has to be checked before advertising.

WithContentEvents(false) was introduced by 0e575f3 ("Bump spegel to v0.7.0"), whose message only discusses not enabling QUIC. Nothing there explains disabling content events, so this may be an unintended side effect, if it was deliberate, knowing why would help.

Steps To Reproduce:

• Installed RKE2 v1.35.7+rke2r1 with embedded-registry: true and a mirror configured in registries.yaml.
• Remove an image from every node but one, then pull it on all nodes at once.
• 2 to 3 nodes out of 9-12 stall while the rest complete almost instantly. Which nodes stall varies between runs.

Expected behavior:

A node only advertises content it can actually serve, so peers resolved from the DHT return the blob rather than a 404.

Actual behavior:

Waves of 404s from peers, and pulls that hang from ~15 seconds to 7 minutes observed:

request to peer failed ... expected one of the following status [200 OK, 206 Partial Content], but received 404 Not Found {"errors":[{"code":"MANIFEST_UNKNOWN","message":"could not get manifest sha256:…"}]}

During a stall, ss -tinHO shows the connection to the peer established and live but carrying only keepalives, and ctr content active shows the ingest counter barely moving. Nothing in the client bounds this wait: the OCI client is built with Timeout: 0 and BaseTransport() sets no ResponseHeaderTimeout, so a peer that accepts a connection and never answers is never abandoned.

Additional context / logs:

Two things made this much harder to diagnose than it needed to be, and small fixes would help anyone hitting it:

• pkg/registry/registry.go:474 logs request to peer failed without naming the peer, although failure.peer.Host is used on the very next line. In a 12-node mesh there is no way to tell which peer returned the 404.
• Spegel's debug endpoints (/debug/web/, /debug/web/stats, /debug/web/measure) are not registered on the supervisor router and return 404; pkg/spegel/spegel.go:300 only wires /v2/.

Measured on my lab cluster with a purpose-built image (8 layers, 1 MB to 1 GB, incompressible): the manifest lands at +0.0 s on every node while the 1 GB layer only arrives at +60 to +65 s, so the window where a node holds a manifest without its layers is wide.