Data races between overlapping bulk subscribe retry attempts

In what area(s)?

/area runtime

What version of Dapr?

1.18.4

Reproduced on 1.18.4 (latest), and present unchanged in 1.17.14 and 1.16.20: in all three, BulkSubIngressDiagnostics is a plain unguarded struct and applyBulkSubscribeResiliency passes &psm to every attempt.

Expected Behavior

Two resiliency attempts of the same bulk delivery should never mutate the same state concurrently. When a policy timeout abandons an attempt, whatever that attempt keeps writing should be safe against the attempt that replaced it.

Actual Behavior

When a resiliency policy timeout fires, NewRunnerWithOptions returns ctx.Err() to the caller while the operation goroutine is still running (pkg/resiliency/policy.go:141-158), and the retry loop starts the next attempt immediately. The abandoned attempt runs to completion, so two attempts of the same bulk delivery are in flight at once, mutating state they share through todo.BulkSubscribeCallData.

Two races follow, on any bulk subscribe route whose policy has both a timeout and a retry, whenever the app is slower than the timeout:

1. Diagnostics. BulkSubIngressDiagnostics.StatusWiseDiag is written by both attempts. It is a plain map, so this is not only a lost counter — it can abort daprd with a fatal concurrent map writes, which is unrecoverable. Elapsed races the same way. There are ~20 such write sites across postman/http/http.go, postman/grpc/grpc.go and bulksubscription.go.

WARNING: DATA RACE
Write at 0x00c000713398 by goroutine 64:
  postman/http.(*http).DeliverBulk()   http.go:353   (bscData.BulkSubDiag.Elapsed = elapsed)
  subscription.(*Subscription).applyBulkSubscribeResiliency.func2()  bulkresiliency.go:61
  resiliency.NewRunnerWithOptions.func2.1.1()  policy.go:124

Previous write at 0x00c000713398 by goroutine 62:
  postman/http.(*http).DeliverBulk()   http.go:353
  ...
Goroutine 62 (finished)   <- the abandoned attempt
Goroutine 64 (running)    <- the attempt that replaced it

2. Message. The resiliency accumulator in applyBulkSubscribeResiliency narrows the delivery set by writing back into the caller's BulkSubscribedMessage, whose address is handed to every attempt as BulkSubMsg: &psm. An abandoned attempt can still be ranging over PubSubMessages while the accumulator replaces it.

WARNING: DATA RACE
Read at 0x00c0001b1740 by goroutine 41:
  todo.PopulateBulkSubscribeResponsesWithError()  helpers.go:95
  postman/http.(*http).DeliverBulk()              http.go:264

Previous write at 0x00c0001b1740 by goroutine 39:
  subscription.(*Subscription).applyBulkSubscribeResiliency.func1()  bulkresiliency.go:53  <- the accumulator

Nothing here is test-only. The trigger is an app that is slower than the configured timeout — a slow consumer, a GC pause, an overloaded app — which is ordinary operation, not an edge case.

Steps to Reproduce the Problem

The diagnostics race already reproduces on master with the existing tests, on roughly 1 run in 3:

go test -tags="allcomponents unit" -race -count=1 -run 'TestBulkSubscribeResiliency' ./pkg/runtime/subscription/

The flaky subtests are TestBulkSubscribeResiliency/fail_ALL_entries_due_to_timeout and TestBulkSubscribeResiliencyWithLongRetries/Fail_all_events_with_timeout_and_then_Open_CB_-_long_retries — both configure a 1s timeout against an app channel that takes 3s, which is exactly the overlap above. Because Go attributes a detected race to whichever test happens to be running, the reported subtest varies between runs.

The message race needs an attempt that completes while an earlier one is still abandoned, which no existing test covers. To reproduce:

1. Configure a bulk subscribe route with a resiliency policy of timeout: 1s and a constant retry of 1s.
2. Have the app take 3s to answer the first bulk delivery, then answer subsequent ones immediately with a mix of SUCCESS and RETRY so the accumulator actually filters.
3. Attempt 1 is abandoned at t=1s; attempt 2 completes and is accumulated at t≈2s, writing psm.PubSubMessages; attempt 1 wakes at t=3s and reads it.

A test covering this is included in the linked PR.

Release Note

RELEASE NOTE: FIX Data races between overlapping bulk subscribe retry attempts, which could abort the sidecar with a fatal concurrent map write.