Workflow waits get CANCELLED "context canceled" when daprd's own runtime shuts down; return UNAVAILABLE instead

In what area(s)?

/area runtime

What version of Dapr?

Reproduced on 1.18.0. The same code is on master (e170747).

Expected Behavior

When daprd shuts down or restarts (rollout, pod restart, hot-reload, a config change that restarts the runtime) while a client is blocked in WaitForInstanceCompletion or WaitForInstanceStart, the call should end with a status that tells the client the sidecar went away and it should retry, such as UNAVAILABLE. The workflow is still running, and a re-issued wait on the new daprd would succeed.

Actual Behavior

The wait fails with CANCELLED and the message context canceled. SDKs treat CANCELLED as final, so the caller sees a failed wait for a workflow that then completes normally.

How it happens:

1. WaitForInstanceCompletion goes to WatchWorkflowRuntimeStatus (actors.go#L801). That calls router.CallStream (#L824), and the orchestrator actor holds the stream open until case <-ctx.Done(): return true, ctx.Err() (stream.go#L131-L132).
2. CallStream wraps its context with r.withContext (router.go#L165-L170). That registers a cancel function for every in-flight call (router.go#L454-L476).
3. When the runtime context ends, router.Run marks the router closed and calls every registered cancel (router.go#L105-L119). The caller's context is still alive, but the wait returns context.Canceled.
4. grpc-go turns that into codes.Canceled "context canceled". The API server stops with GracefulStop, so the client receives that status instead of a connection reset.

Call and CallReminder go through the same withContext wrapper, so any in-flight actor call is affected the same way. The workflow wait matters most, because it is held open for minutes or hours.

Steps to Reproduce the Problem

1. Run placement, scheduler and daprd 1.18.0 natively, with an in-memory actor state store.
2. Register a workflow whose single activity sleeps for 60 s. Schedule it, and call wait_for_workflow_completion(instance_id) from the Python SDK with no timeout.
3. About 10 s in, send daprd SIGTERM.

The client fails with StatusCode.CANCELLED and details context canceled, and daprd logs Daprd shutdown gracefully. Without the SIGTERM, the same wait returns normally after 60 s.

Suggested fix

In the wait paths (WatchWorkflowRuntimeStatus and the gRPC executor's wait handlers), when the router's own context ended but the caller's context didn't, return codes.Unavailable ("runtime shutting down, retry") instead of ctx.Err(). SDKs already retry UNAVAILABLE on these waits. Also consider draining in-flight actor streams before cancelling them on shutdown.

Related

• #8332 (closed): graceful shutdown cancels in-flight requests, in general
• #10428: status watches on a stalled workflow retry with backoff
• #10408, #10307: nearby workflow wait and cancel fixes
• #10337: daprd not recovering after a control-plane restart

Client-side discussion: dapr/python-sdk#1246

Release Note

RELEASE NOTE: FIX Workflow waits (WaitForInstanceCompletion / WaitForInstanceStart) now return UNAVAILABLE instead of CANCELLED when the sidecar shuts down mid-wait, so clients can retry against the restarted sidecar.