CRI: PullImage hangs forever when image_pull_progress_timeout = "0s" (default transfer service path)

Description

Hey everyone! I ran into this one while playing with a slow registry mirror on my test cluster. Pulls kept getting killed by the progress timeout, so I set image_pull_progress_timeout = "0s" expecting it to mean "never give up on a slow pull". The code even logs no timeout and will not start pulling image ... reporter, so it really does look like a supported value.

Instead, every image pull through the transfer service just hangs. Forever. The PullImage call never returns; it only unwinds when the client (kubelet, crictl, whatever) gives up and cancels the context. Since the transfer service is the default pull path in containerd 2.x (use_local_image_pull defaults to false), this basically bricks image pulls on the whole node, and nothing in the logs tells you why.

Steps to reproduce the issue

1. Configure containerd with:

   toml复制

   version = 3
   
   [plugins."io.containerd.cri.v1.images"]
     image_pull_progress_timeout = "0s"

2. Restart containerd.

3. Pull any image via CRI, e.g. crictl pull registry.k8s.io/pause:3.10 (or just let kubelet schedule a pod with an image that isn't cached yet).

Describe the results you received and expected

Received: PullImage never finishes. No pull error, no watchdog firing. The request just sits there until the client disconnects. crictl hangs indefinitely, and kubelet only shows a generic context-deadline-exceeded way downstream, which sent me looking in completely the wrong direction at first.

Expected: "0s" should disable the progress watchdog and the pull should go on normally, just without the no-progress cancellation. At least that's what the docs and the code's own log line suggest.

What version of containerd are you using?

main @ f6132dbe1f482cbe0aebc4bd3d8d7a184fb4a2aa (v2.x development branch; the affected code path is the default in 2.x releases).

Any other relevant information

Root cause. I went down a bit of a rabbit hole on this one, and the culprit is the progress reporter on the transfer pull path. It's built around an unbuffered channel whose only reader is never started when the timeout is zero:

1. newTransferProgressReporter creates the channel unbuffered (internal/cri/server/images/image_pull.go:951):

   go复制

   pc: make(chan transfer.Progress),

2. start() bails out early when timeout == 0, so no goroutine ever reads from pc (internal/cri/server/images/image_pull.go:1011-1015):

   go复制

   func (reporter *transferProgressReporter) start(ctx context.Context) {
       if reporter.timeout == 0 {
           log.G(ctx).Infof("no timeout and will not start pulling image %s reporter", reporter.ref)
           return
       }

3. But the progress func we hand to the transfer service still sends to pc, and its only escape hatch is a context that gets cancelled after the transfer returns (internal/cri/server/images/image_pull.go:342-343 and 1075-1082):

   go复制

   err = c.transferrer.Transfer(rctx, reg, is, transfer.WithProgress(transferProgressReporter.createProgressFunc(rctx)))
   rcancel()

   go复制

   return func(p transfer.Progress) {
       select {
       case reporter.pc <- p:   // blocks forever: nobody is reading
       case <-ctx.Done():       // rctx: only cancelled by rcancel() after Transfer returns
           return
       }
   }

4. On the transfer side, HandleProgress calls the progress func synchronously inside its own event loop (core/transfer/local/progress.go:114 and friends), and the pull won't return until that loop is done (core/transfer/local/pull.go:128-129):

   go复制

   go progressTracker.HandleProgress(ctx, tops.Progress, NewContentStatusTracker(store))
   defer progressTracker.Wait()

So the very first progress event deadlocks the whole chain: HandleProgress blocks in pf(...), waitC never closes, progressTracker.Wait() never returns, Transfer never returns, rcancel() is never reached, and ctx.Done() never fires. Textbook circular wait, and it fires deterministically on the first layer byte that reports progress.

Suggested fix. A few ways to break the cycle, smallest first:

• In createProgressFunc, don't block when the reporter was never started: guard the send with a timeout == 0 / started check, or make it non-blocking with select + default and just drop the event.
• Or always start the consumer goroutine and only skip the timeout check when timeout == 0.
• Or reject "0s" in config validation, if "no timeout" isn't actually meant to be a thing.

Not sure which direction you'd rather go, but I'm happy to send a PR once you point me at one.

Logs. The only CRI log line anywhere near the hang is this (misleadingly calm) one:

no timeout and will not start pulling image <ref> reporter

A goroutine dump taken during the hang shows HandleProgress parked on the reporter.pc <- p send and the pull goroutine parked in ProgressTracker.Wait.

Possibly related: #13909 involves the same reporter, but it's the mirror image of this bug: there the watchdog is running (timeout > 0) and cancels a healthy pull because extracting/extracted events leak the active-request counter; here the watchdog is never started (timeout = 0) and the pull deadlocks because nothing ever drains the progress channel. Different trigger, different mechanism, different fix. Handling those extra events wouldn't unblock an unbuffered channel with no reader, and vice versa. (Also not to be confused with the closed #11650: that one was about genuinely slow networks defeating the progress watchdog on 1.7, an enhancement request. This one deadlocks with a perfectly healthy registry and network, on a brand-new pull.)

Show configuration if it is related to CRI plugin.

version = 3

[plugins."io.containerd.cri.v1.images"]
  image_pull_progress_timeout = "0s"