Shutdown sequence never finishes when a service does not stop: stopServices ends with an unbounded wait, so the node never powers off

Bug Report

Description

stopServices ends with an unbounded wait, so one service that does not stop blocks the whole shutdown sequence and the node never powers off.

The last line of internal/app/machined/pkg/system/system.go#L408 is return conditions.WaitForAll(stoppedConds...).Wait(ctx), on the caller's context. The 30 s deadline declared 26 lines above it, at #L381-L383, is shutdownCtx, and shutdownCtx is used only inside the per-service goroutines, for allDeps.Wait(shutdownCtx) at #L398 — the wait for each service's reverse dependencies. The final wait, for the services themselves, gets the bare ctx.

The callers on this path attach no deadline of their own. internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_controller.go contains neither context.WithTimeout nor context.WithDeadline; Controller.run is #L276, runPhase #L341, runTask #L372, and each passes the sequence context straight through.

What is lost when it blocks is everything that comes after stopServices in stopAllPhaselist — umount, unmountBind, unmountPromotable, unmountSystem, volumeFinalize, stopEverything, and then the power-off task itself, v1alpha1_sequencer.go#L509-L557 and Sequencer.Shutdown #L362-L382. The stopServices phase itself is StopServicesEphemeral, which stops cri with its reverse dependencies, and the power-off task is Shutdown #L1292-L1308. The machine stays up, with its filesystems mounted, until someone removes power.

I hit this on a bare-metal node running KubeVirt. The containerd shim backing a VirtualMachineInstance would not go away, so the cri service could not be stopped, and talosctl shutdown never completed.

Every neighbouring wait on the same path is bounded, which is what narrows the defect to this one call:

• stopAndRemoveAllPods caps its CRI work at three minutes, v1alpha1_sequencer_tasks.go#L929, and waits for the kubelet lifecycle finalizers under a 30 s deadline at #L873.
• The process runner escalates SIGTERM to SIGKILL after GracefulShutdownTimeout, runner/process/process.go#L487.
• The containerd runner bounds the post-SIGKILL reap at killWaitTimeout = 30 * time.Second, runner/containerd/containerd.go#L47-L52 and #L347-L354, whose own comment states the principle exactly: "a task stuck in uninterruptible sleep never reaps, and waiting for it forever would hold up whatever is driving the stop."

There is also no escape hatch on the client side. talosctl shutdown --help offers only --force if true, force a node to shutdown without a cordon/drain, which skips the cordon and drain and not the service stop, while talosctl reboot has had -m, --mode with [default force powercycle] since #12049 closed #11782. An operator facing a node whose services will not stop has nothing to reach for.

Reproduction

A deterministic Go unit test inside the Talos tree, needing no cluster, no Kubernetes, no KubeVirt and no root: talos-shutdown-repro.zip, attached in a comment below. It holds README.md, shutdown_bounded_test.go, fix.patch and run.sh; run.sh runs the test twice against a clean checkout — unpatched, then with fix.patch applied — and reverts the patch afterwards, leaving the checkout as it was found.

The test stands in for the VMI's shim with a service whose runner does not return when its context is cancelled, which is the only property of the field situation that stopServices can observe.

Run today against a clean checkout of main at 7f5327d with Go 1.26.8, the two decisive results are below.

Logs

Unpatched — the shutdown never returns, and the test gives up after its own 40 s deadline:

=== RUN   TestSystemServicesSuite/TestShutdownBoundedWithStuckService
2026/09/13 13:08:53 service[stuck-shim](Running): Running
    shutdown_bounded_test.go:119:
        	Error:      	Shutdown did not return
        	Messages:   	still stopping services after 40s; a service which does not stop blocks the shutdown sequence forever
--- FAIL: TestSystemServicesSuite/TestShutdownBoundedWithStuckService (40.04s)
FAIL	github.com/siderolabs/talos/internal/app/machined/pkg/system	40.114s

Patched — the wait is bounded, the service that did not stop is named, and the sequence carries on:

=== RUN   TestSystemServicesSuite/TestShutdownBoundedWithStuckService
2026/09/13 13:09:57 service[stuck-shim](Running): Running
2026/09/13 13:10:27 gave up waiting for stuck-shim to stop after 30s, continuing
--- PASS: TestSystemServicesSuite/TestShutdownBoundedWithStuckService (30.07s)
ok  	github.com/siderolabs/talos/internal/app/machined/pkg/system	30.156s

There is nothing useful in the node's own log for the field incident: the sequence simply stops emitting after the stopServices phase begins, which is the symptom.

Environment

• Talos version:

$ talosctl -n <node> version
Client:
	Tag:         v1.13.9
	SHA:         3ebd10a7
	Built:
	Go version:  go1.26.6
	OS/Arch:     linux/amd64
Server:
	NODE:        <node>
	Tag:         v1.13.9
	SHA:         3ebd10a7
	Built:
	Go version:  go1.26.6
	OS/Arch:     linux/amd64
	Enabled:     RBAC

• Kubernetes version: v1.36.4, with KubeVirt v1.9.0. (The node name is redacted above; no kubectl version block is quoted because I have none recorded verbatim from the incident, and I will not reconstruct one.)
• Platform: bare metal, single node, metal runtime mode.

The node had been upgraded v1.12.6 → v1.12.11 → v1.13.9 on the evening before the incident. The code is byte-identical across those versions and on main: v1.12.6, v1.13.9, main at 7f5327d.

Why the docker provisioner cannot show this

talosctl cluster create docker builds a working lab, but talosctl shutdown against it is rejected with rpc error: code = FailedPrecondition desc = method is not supported in container mode. Container mode masks the capability — mode.go#L91-L103 returns all ^ uint64(Reboot|Shutdown|Upgrade|Rollback|MetaKV) for ModeContainer, checked at v1alpha1_server.go#L461 before Controller.Run — and no flag lifts it. That is why the reproduction is a unit test rather than a lab run.

Proposed fix

Filed as #14405, two commits:

1. fix: bound the wait for services to stop on shutdown — names the timeout the function already declares (stopServicesTimeout, still 30 s) and applies it to the final wait as well as the reverse-dependency wait; on expiry it logs which services did not stop and lets the sequence carry on, so the filesystems are unmounted and the machine is powered off. A cancellation coming from the caller is still returned as an error. This is the change in fix.patch.
2. feat: add a force mode to shutdown, mirroring reboot's — ShutdownRequest.Mode with the same shape as RebootRequest.Mode, so talosctl shutdown -m force powers a node off without stopping the services first, the way talosctl reboot -m force already reboots.

Happy to take the fix alone if the second commit is unwelcome.

Not verified

The end-to-end confirmation on hardware — a scratch node with KubeVirt and one running VMI, talosctl shutdown reaching power-off with the patched machined — has not been done. The unit test proves the missing deadline and its fix; it does not prove that a real VMI's shim was the only thing holding cri open on my node.

Prior art already checked

#11775, #9625, #11782, #12049, #13934, #13622, #7137, #12599, #10143, #14296. None of them is this defect.

I used Claude (Anthropic) as an assistant while investigating this and preparing the patch; I have reviewed and tested everything here myself and I take responsibility for it.