Flaky tests: TestV1DirectMessagingEndpointsWithResiliency subtests failing intermittently
Description
The TestV1DirectMessagingEndpointsWithResiliency test in pkg/api/http is flaking with multiple subtest failures. Errors indicate context canceled rather than the expected resiliency behavior (retries, circuit breaker tripping, etc.), suggesting a timing/race condition in the test setup.
Where it showed up
Showing up on PR #9731 (unrelated changes):
- PR: https://github.com/dapr/dapr/pull/9731
- Failing job: https://github.com/dapr/dapr/actions/runs/25569842345/job/75063083221?pr=9731
Failing subtests
All under pkg/api/http TestV1DirectMessagingEndpointsWithResiliency:
Test_invoke_direct_message_retries_with_resiliency— got HTTP 500 instead of 200; only 1 call made instead of 2Test_invoke_direct_message_retries_on_unsuccessful_statuscode_with_resiliency— got 500 instead of 450; 0 calls instead of 2; response body showsERR_DIRECT_INVOKE: failed to invoke, id: failingApp, err: context canceledTest_invoke_direct_message_fails_with_timeout— 0 calls instead of 2;superfluous response.WriteHeader callwarning at directmessaging.go:255Test_invoke_direct_messages_fails_after_exhausting_retries— 0 calls instead of 2; samesuperfluous response.WriteHeaderwarningTest_invoke_direct_messages_can_trip_circuit_breaker— 1 call instead of 5; never received "circuit breaker is open" error; gotcontext canceledinstead
Sample failure output
=== FAIL: pkg/api/http TestV1DirectMessagingEndpointsWithResiliency/Test_invoke_direct_message_retries_on_unsuccessful_statuscode_with_resiliency
directmessaging_test.go:973:
Error: Not equal:
expected: "failingKey2"
actual : "{\"errorCode\":\"ERR_DIRECT_INVOKE\",\"message\":\"failed to invoke, id: failingApp, err: context canceled\"}"
=== FAIL: pkg/api/http TestV1DirectMessagingEndpointsWithResiliency/Test_invoke_direct_messages_can_trip_circuit_breaker
directmessaging_test.go:1015:
Error: "{...err: context canceled\"}" does not contain "circuit breaker is open"
Messages: Should have received a circuit breaker open error.Impact
Blocking unrelated PRs from getting a green CI. The PR in question (#9731) does not touch direct messaging, HTTP API, or resiliency code.
Suspected cause
The repeated context canceled errors and superfluous response.WriteHeader call at directmessaging.go:255 suggest the request context is being canceled before the resiliency policy can execute its retries — likely a test-level timeout or context lifetime issue rather than a production bug.
Source: dapr/dapr