Hot reload kills the sidecar when a pub/sub handler outlasts the graceful shutdown window

In what area(s)?

/area runtime

Expected Behavior

Editing a Configuration, Resiliency or HTTPEndpoint resource with hot reloading enabled restarts the sidecar runtime in process. The sidecar should come back.

Actual Behavior

If any pub/sub handler is still running when the restart begins, and it takes longer than the graceful shutdown window, the sidecar exits and does not come back:

level=info  msg="Received signal 'hangup'; restarting"
level=info  msg="Dapr is shutting down"
level=info  msg="Shutting down component orderpubsub (pubsub.redis/v1)"
level=fatal msg="Graceful shutdown timeout exceeded, forcing shutdown"

The fatal lands exactly 5s after the SIGHUP and the process is gone. The app is left without a sidecar until something restarts it, and the in-flight messages are never acked, so the broker redelivers them.

The subscription drain waits for in-flight handlers, up to drainMaxDuration of 30s, in this loop. The shutdown it runs inside is bounded by DefaultGracefulShutdownDuration, 5s. Any handler slower than that turns a config edit into a sidecar kill.

Steps to Reproduce the Problem

Self-hosted, with a Redis Streams pub/sub component and an HTTP subscriber whose handler sleeps 12s before returning {"status":"SUCCESS"}.

1. Run daprd with a Configuration that enables HotReload, and a Resiliency resource in the resources directory.
2. Publish 6 messages to the subscribed topic and wait ~4s, so the handlers are running.
3. Edit the Resiliency resource (any change — I changed timeouts.general from 30s to 45s).

The SIGHUP arrives, shutdown begins, and 5s later the process exits with the fatal above.

Control: the same edit with nothing in flight restarts cleanly and the sidecar keeps running.

Affected Versions

1.17 logs Hot reloading enabled. Daprd will reload 'Component' and 'Subscription' resources on change. and never watches Configuration, Resiliency or HTTPEndpoint, so there is no in-process restart to collide with the drain.

Possible regression source: #9582 (commit f2d61989b20a4a7e9238edbc19ab2fcbe76dc47a, first released in v1.18.0), which added the SIGHUP reconcilers that restart the runtime in process for Configuration, HTTPEndpoint and Resiliency. The drain and the 5s shutdown budget both predate it; that change is what made them reachable from a resource edit.

The same collision has already been hit once on the workflow side: #10100 describes a Configuration hot-reload leaving a workflow-hosting sidecar permanently unavailable because shutdown did not complete promptly. The pub/sub drain is the same shape and is still reached on master.

Verification

Built from master ece6ec75e and from the v1.18.4 and v1.17.14 tags, run on macOS, --log-level info, no placement or scheduler.

Reload landing mid-flight, same six-message scenario each time:

The same sequence with an idle subscription restarts and stays up.

A goroutine dump taken 3s after the SIGHUP shows the shutdown parked in the drain, with the handlers still waiting on the app:

goroutine 190 [sleep]:
time.Sleep(...)
github.com/dapr/dapr/pkg/runtime/subscription.(*Subscription).Stop(...)
	pkg/runtime/subscription/subscription.go:523
github.com/dapr/dapr/pkg/runtime/processor/subscriber.(*Subscriber).StopPubSub.func1(...)
	pkg/runtime/processor/subscriber/subscriber.go:301

github.com/dapr/dapr/pkg/runtime/subscription/postman/http.(*http).Deliver(...)
	pkg/runtime/subscription/postman/http/http.go:96
github.com/dapr/dapr/pkg/runtime/subscription.New.func1(...)
	pkg/runtime/subscription/subscription.go:365

That dump was captured on a build carrying one unrelated commit on top of master, so its line numbers are shifted; Stop:523 there is the drain loop at subscription.go:505 on master. The mid-flight kill itself reproduces on a clean master build.

Release Note

RELEASE NOTE: