Actor Resiliency timeout causes `context canceled` for localhost HTTP calls within Actor method (Dapr 1.18.1)
Issue Template for Dapr GitHub
Title
Actor Resiliency timeout causes context canceled for localhost HTTP calls within Actor method (Dapr 1.18.1)
Environment
| Item | Version |
|---|---|
| Dapr Runtime | 1.18.1 |
| Dapr SDK (NuGet) | 1.18.4 (Dapr.Actors, Dapr.Client, Dapr.AspNetCore) |
| Deployment | Kubernetes |
| .NET Version | .NET 10 |
| Dapr State Store | Redis (Dragonfly) |
Architecture
Our application uses a Manage + PST co-hosted pod architecture:
Pod: oamanage
├── Container: oamanage (Manage - Business Logic + Actors)
├── Container: oamanagepst (PST - Persistence API)
├── Container: daprd (Dapr sidecar)
└── Container: nginx (reverse proxy)The Manage container hosts Dapr Actors (including PointActor). Within PointActor's method, it calls the PST container via http://localhost:{port}/api/... using ServiceStack's JsonServiceClient. This is a localhost call within the same Pod and does NOT go through Dapr service invocation.
Problem
When a Dapr Resiliency CRD assigns a timeout policy to PointActor, the Actor method fails with context canceled. The error originates from the internal localhost HTTP call to the PST service, even though:
- The HTTP call goes to
localhost, entirely bypassing Dapr's service invocation - The
timeoutis set to5m(5 minutes), generously long - The same code works normally when the
timeoutpolicy is removed
Configuration
Dapr Resiliency CRD (kubectl get resiliency -n dev oamanage -o yaml):
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: oamanage
namespace: dev
spec:
scopes:
- oamanage
policies:
timeouts:
general: 5s
important: 5m # <-- PointActor uses this
largeResponse: 10s
retries:
general:
duration: 5s
maxRetries: 3
policy: constant
targets:
actors:
dev_OA_PointActor:
timeout: important # <-- This causes context canceledObserved Behavior
With timeout: important:
Sidecar log (daprd):
Found Actor Policy for type dev_OA_PointActor: {Timeout:important}
api error: code = Internal desc = error invoke actor method:
failed to read response data: context canceledThe PointActor method calls httpContext.GetAsync<QueryResponse<Point>>(...) which internally uses JsonServiceClient to make an HTTP request to http://localhost:{pst-port}/api/GetPoint?MemberId=xxx. This localhost HTTP call fails with context canceled.
With timeout removed:
Everything works normally. The localhost HTTP call completes successfully.
New Actor without timeout policy:
New Actors (e.g., DevTestActor) that are not listed in the Resiliency targets work fine, even when they make the same localhost HTTP calls.
Reproduce Steps
- Deploy a Dapr-enabled app with Dapr 1.18.1
- Define a Resiliency CRD with a
timeoutpolicy for a specific Actor type - In the Actor's method implementation, make a localhost HTTP request (e.g., to a co-hosted service within the same Pod)
- Invoke the Actor method
- Observe
context cancelederror
Expected Behavior
The Resiliency timeout should only apply to the Actor invocation gRPC call (i.e., the outer boundary from caller to Actor). It should NOT propagate to or affect any internal HTTP requests that the Actor method makes to localhost services.
Analysis
We believe Dapr 1.17+ introduced a stricter enforcement of the Resiliency timeout policy at the gRPC context level. The gRPC context (with the timeout) is being incorrectly propagated to the HTTP client connection within the Actor method, specifically the localhost HTTP call made by JsonServiceClient.
Since JsonServiceClient sends the HTTP request to localhost -- which does NOT go through Dapr's iptables interception or service invocation -- the gRPC context from the Actor Resiliency policy should have no bearing on this localhost call. But Dapr 1.18 appears to be associating them, causing the HTTP connection to be prematurely canceled.
Workaround
Remove the timeout policy from the Actor's Resiliency target:
targets:
actors:
dev_OA_PointActor:
retry: general # Keep retry
# timeout: important # Remove timeout (use application-level timeout instead)Timeout control is then implemented at the application level via JsonServiceClient.Timeout.
Downgrade Note
This issue did NOT occur on Dapr 1.16.x with the same Resiliency configuration and same application code.
Please let me know if you need any additional information, such as full sidecar logs, application code, or a minimal reproduction.
Source: dapr/dapr