BulkPublishEventAsync hangs in DaprRuntimeCapabilities when daprd has no gRPC reflection service (regression in 1.18)

[Bug] BulkPublishEventAsync hangs in DaprRuntimeCapabilities when daprd has no gRPC reflection service (regression in 1.18)

In what area(s)?

/area client

What version of Dapr?

• Dapr.Client / Dapr.Common: 1.18.10 (NuGet)
• daprd: 1.18.4 (daprio/daprd:1.18.4, self-hosted via Docker Compose)
• .NET: 10 (app migrated from .NET 8)
• Works: all Dapr.* packages pinned to 1.17.x, with the same .NET 10 app and the same daprd:1.18.4 sidecar
• Worked before migration: Dapr.Client 1.16 on .NET 8

Expected Behavior

DaprClient.BulkPublishEventAsync publishes the batch, or fails fast with a clear error. If the runtime capability check cannot use gRPC reflection, it should fall back optimistically, as the code in SupportsMethodAsync intends, and not block.

Actual Behavior

BulkPublishEventAsync never reaches the actual bulk publish RPC. It blocks until the caller's CancellationToken fires:

System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeoutAsync(TaskNode asyncWaiter, Int64 millisecondsTimeout, CancellationToken cancellationToken)
   at Dapr.Common.DaprRuntimeCapabilities.GetMethodsForServiceAsync(String serviceName, CancellationToken cancellationToken)
   at Dapr.Common.DaprRuntimeCapabilities.SupportsMethodAsync(String fullyQualifiedMethodName, CancellationToken cancellationToken)
   at Dapr.Common.VersionAwareDaprClient.BulkPublishEventAsync(BulkPublishRequest request, CallOptions options)
   at Dapr.Client.DaprClientGrpc.MakeBulkPublishRequest[TValue](String pubsubName, String topicName, IReadOnlyList`1 events, Dictionary`2 metadata, CancellationToken cancellationToken)
   at <app code>

The caller is blocked on _gate (SemaphoreSlim(1,1)), so another capability lookup is holding the semaphore and never completes.

Observed timing / what we could capture:

• The only stack trace we could obtain is the one above (the waiting side). The call holding the semaphore never throws or returns, so its stack cannot be captured from application logs.
• The failing call runs from a background timer 60 seconds after startup, when the app is already serving requests. WaitForSidecarAsync returned immediately before app.Run(), so the sidecar was healthy. This is not a startup race.
• The call blocks for the full duration of the caller's token. We tested with a 30s timeout and with a 300s timeout, and both expired with the identical stack trace. Waiting longer does not help, which indicates the reflection lookup never completes rather than being slow.
• Pinning Dapr.* to 1.17.x, with nothing else changed, resolves the problem completely.

daprd does not expose gRPC reflection

Running the same lookup the SDK performs (FileContainingSymbol) against the sidecar, from inside the app's network namespace:

$ docker run --rm --network container:<app-container> fullstorydev/grpcurl -plaintext -max-time 10 \
    localhost:50001 describe dapr.proto.runtime.v1.Dapr

Failed to resolve symbol "dapr.proto.runtime.v1.Dapr": rpc error: code = Unknown desc = cannot proxy request: missing dapr-callee-app-id or dapr-app-id metadata

The reflection service is not registered on the daprd API server. The call falls through to the service-invocation gRPC proxy, which rejects it immediately with Unknown.

With grpcurl the error comes back instantly. In the .NET SDK, however, the reflection call (a bidi stream opened by ServerReflectionInfo) appears to never complete on .NET 10. The server ends the stream with an error while the client request stream is still open. The comment in SupportsMethodAsync about an "HTTP/2 stream setup race on .NET 9+" suggests this path is already known to be fragile.

Analysis (from src/Dapr.Common/DaprRuntimeCapabilities.cs on master)

1. GetMethodsForServiceAsync holds _gate across network I/O, and the reflection call has no deadline of its own. It only uses the caller's token.
2. If that lookup hangs, every later capability check on the same DaprClient blocks on _gate. DaprClient is typically a DI singleton, so all bulk publishes are affected.
3. RpcException is handled optimistically (return true). But OperationCanceledException is only swallowed when the caller's token is not cancelled. Once the caller times out, the exception propagates.
4. Results are only cached on success. A failed or cancelled lookup is retried by the next caller, which hits the same hang.

In short: against a sidecar without reflection, one stuck lookup becomes a permanent block for every bulk publish on that client.

Steps to Reproduce the Problem

1. Run daprd:1.18.4 as a sidecar (Docker Compose, network_mode: service:<app>, -app-port 80, gRPC on 50001).
2. .NET 10 ASP.NET Core app with Dapr.Client 1.18.10, DaprClient registered via DI.
3. After the app has fully started, call:

   using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
   await daprClient.BulkPublishEventAsync("pubsub", "topic", items, cancellationToken: cts.Token);

1. The call never completes and throws OperationCanceledException from SemaphoreSlim.WaitUntilCountOrTimeoutAsync.
2. Downgrade all Dapr.* packages to 1.17.x and repeat: the bulk publish succeeds.

Suggested fixes

• Give the reflection lookup its own short deadline (e.g. a few seconds), independent of the caller's token.
• Treat reflection being unavailable (Unknown, Unimplemented, deadline exceeded, internal cancellation) as a cached "assume supported" result, so the lookup happens once per client and not once per call.
• Avoid holding _gate across the network call (e.g. cache a Lazy<Task<...>> per service).
• Consider a public opt-out for the capability check (e.g. make UseRuntimeCapabilities public or add an option), for environments where reflection is not available.

Workaround

Pin all Dapr.* packages to the latest 1.17.x release.

Release Note

RELEASE NOTE: FIX Runtime capability detection no longer blocks SDK calls such as BulkPublishEventAsync when the Dapr sidecar does not expose gRPC reflection.