Kubernetes backend: WaitStep hangs until the pipeline timeout when a service pod is deleted before the informer cache syncs

Component

agent

Describe the bug

We hit this on every pipeline with services: and finally caught a live goroutine dump, so I can offer a root cause. Short version: the guard added in #6623 narrows the window but does not close it, because it runs before the informer's cache is synced. The affected code in WaitStep is byte-for-byte identical in v3.17.0, v3.18.1 and main (b4b4660), so this is still present today.

Environment and frequency

Woodpecker server and agent 3.17.0, Kubernetes backend, k3s, two agents, three repositories whose pipelines declare services: [postgres:16-alpine].

Over a six-day window (2026-09-01 .. 2026-09-07) we recorded 33 hangs: 28 in one repository, 5 in another, spread over both agents. This is not a rare race for us — it is the normal outcome whenever the working steps are short relative to the service pod's lifetime. Untouched, a hung workflow ends exactly at the repository timeout (3600 s in our case); the service step stays running in the API and UI until then, which is the symptom reported in this issue.

One detail that made this hard to measure: the pipeline status carries no signal of the hang in any variant. A healthy run, a run cancelled through the API, and a run left to hit the 3600 s timeout all end up as success at the pipeline level. The only reliable indicator is the gap between the end of the last working step and the end of the workflow.

Goroutine dump from a live hang

Taken with SIGQUIT (GOTRACEBACK=all) on the agent while a workflow was hung:

goroutine 46012 gp=0x396bfd7874a0 m=nil [select, 12 minutes]:
runtime.selectgo(...)
go.woodpecker-ci.org/woodpecker/v3/pipeline/backend/kubernetes.(*kube).WaitStep(...)
	/src/pipeline/backend/kubernetes/kubernetes.go:356
go.woodpecker-ci.org/woodpecker/v3/pipeline/runtime.(*Runtime).completeStep(...)
	/src/pipeline/runtime/step.go:155
go.woodpecker-ci.org/woodpecker/v3/pipeline/runtime.(*Runtime).runDetachedStep.func1()
	/src/pipeline/runtime/step.go:242

Line 356 in v3.17.0 is the select in WaitStep. Above it, goroutine 36 sits in sync.WaitGroup.Wait at pipeline/runtime/workflow.go:82 — that is uploadWait.Wait() waiting on this very detached-step goroutine.

Worth noting for anyone who suspected the log stream (we did): the whole dump contains no waitForLogs, TailStep or tailLogs frame at all. The pods/log stream ended normally; the hang is one step further, in WaitStep.

Mechanism

WaitStep (pipeline/backend/kubernetes/kubernetes.go, identical in v3.17.0 / v3.18.1 / main):

si := informers.NewSharedInformerFactoryWithOptions(...)  // UpdateFunc: podUpdated, DeleteFunc: podDeleted
stop := make(chan struct{})
si.Start(stop)          // informer starts asynchronously: LIST, then WATCH
defer close(stop)

// If the pod was deleted before the informer started, no events will
// ever arrive. Check explicitly so we don't hang forever.
if _, err := e.client.CoreV1().Pods(ns).Get(ctx, podName, ...); kube_errors.IsNotFound(err) {
    return &types.State{ExitCode: 0, Exited: true}, nil
}

select {
case <-finished:
case <-ctx.Done():
    return nil, ctx.Err()
}

finished is closed only from podUpdated (phase Succeeded/Failed/Unknown) or podDeleted — both driven by the informer. There is no cache.WaitForCacheSync between si.Start() and the guard, so the guard's Get can observe a state that the informer never will:

1. si.Start() kicks off the initial LIST asynchronously.
2. The guard Get succeeds — the service pod still exists (possibly already Terminating).
3. The workflow teardown deletes the service pod.
4. The informer's initial LIST completes after the deletion, so the pod never enters the cache; the WATCH is then established from that LIST's resourceVersion, so the deletion is already in the past and no event is delivered.
5. DeleteFunc never fires, finished is never closed, and the select blocks until ctx.Done() — i.e. the workflow timeout.

The 5 s resync does not help: resync replays objects from the cache, and this pod was never in it.

The comment above the guard states the intent exactly right; the guard is just placed before the synchronization it depends on.

Why only services: a service pod is deleted by the teardown, i.e. concurrently with completeStep starting its informer. A regular step's pod ends by a phase change that podUpdated sees through an established WATCH. That the informer is started at teardown time, and not when the service starts, is also what explains the hit rate — if the WATCH had been up for the whole nine minutes of the pipeline, losing the event would be rare luck.

Why cancelling the pipeline "fixes" it: the server signals cancellation, the agent cancels the workflow context, ctx.Done() fires — that is the only branch of that select which ever fires other than a delivered event.

Suggested fix

Any of these closes the window; the first is the minimal one:

1. cache.WaitForCacheSync(stop, si.Core().V1().Pods().Informer().HasSynced) before the guard Get. After a synced cache, the WATCH is running from the LIST's resourceVersion, so either the Get already returns NotFound (guard returns) or every later deletion is delivered as an event.
2. Equivalently, repeat the Get after cache sync.
3. As defence in depth against any lost watch event, add a third select branch with a ticker that re-checks the pod's existence periodically.

We can reproduce the hang several times a day on our cluster.

Workarounds, for anyone stuck on a released version

Both are user-side patches, not fixes:

• Injecting the missing event. A small controller watches for the hang and then creates and deletes a pod with the same name as the deleted service pod, in the agent's namespace. The informer delivers the delete event, finished closes, and the service step ends as success (not killed, as it does when the pipeline is cancelled). 33 out of 33 hangs recovered this way in the window above, ~30 s from detection, no agent restarts. It needs create/delete on pods in the agent's namespace.
• Letting the service pod terminate itself before teardown. podUpdated also closes finished on PodSucceeded/PodFailed/PodUnknown, so a service that exits by itself at the end of the last working step should never enter the race. We have not tested this — it is a suggestion from reading the code, not something backed by a run.

Relation to #6669

@hhamalai suspected context/error handling in #6669 as the cause of the remaining occurrences. The dump above points elsewhere: the goroutine is blocked in the select, with a live context, waiting for an event that will never arrive. #6669 may well fix a different corner case, but it would not close this race.

Steps to reproduce

1. Run the agent with the Kubernetes backend (WOODPECKER_BACKEND=kubernetes). Any version from v3.17.0 to current main will do — WaitStep is byte-for-byte identical across them.
2. Give a repository a pipeline with a long-running service and a working step that is short relative to the pipeline as a whole:

services:
  - name: postgres
    image: postgres:16-alpine
    ports: [5432]
    environment:
      POSTGRES_PASSWORD: somepass

steps:
  - name: test
    image: postgres:16-alpine
    commands:
      - pg_isready -h postgres -t 30

1. Run the pipeline repeatedly and watch the pods (kubectl get pods -n <agent namespace> -w).
2. Observe a run in which the working steps finish and the teardown deletes the pods, but the service step stays running in the API and the web UI, and the workflow does not end. Left alone, it ends at the pipeline timeout — 3600 s in our configuration — and the service step is then reported failure with context deadline exceeded.
3. This is a race, so it does not fire on every run. It gets more likely the shorter the working steps are: in our three repositories it happened 33 times over six days, and in one window it hit 5 out of 5 consecutive runs of one repository.
4. To confirm that a given run hit this race and not another failure mode, take a goroutine dump of the agent — kill -QUIT the agent process with GOTRACEBACK=all set, then read kubectl logs <agent pod> --previous. The signature is a goroutine blocked in kubernetes.(*kube).WaitStep → runtime.(*Runtime).completeStep, with no waitForLogs, TailStep or tailLogs frame anywhere in the dump.

Note when looking for affected runs: the pipeline status carries no signal of the hang. A healthy run, a cancelled run and a run left to hit the timeout all end up as success at the pipeline level. The only reliable indicator is the gap between the end of the last working step and the end of the workflow.

Expected behavior

No response

System Info

server  v3.18.0
agent   v3.17.0 (StatefulSet, 2 replicas)
backend kubernetes
k3s     v1.31.4+k3s1

Additional context

No response

Validations

• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/versions]