PubSub: no root span created when inbound message has no traceparent (breaks observability for DLQ re-queues / external publishers)

In what area(s)?

/area runtime

Expected Behavior

When the sidecar delivers a pub/sub message whose inbound CloudEvent carries no W3C trace context (no traceparent, no legacy traceid), Dapr should still start a new root span for the delivery, named pubsub/<topic>, and inject it into the metadata passed to the app. Every delivered message then has an end-to-end trace, whether or not the producer set trace headers.

Actual Behavior

No span is created at all. The message is delivered normally, but the hop is invisible in tracing and the app receives an empty traceparent header — so it has nothing to continue from and nothing to cleanly detect as absent. The gap propagates through the whole downstream call chain, not just the one hop.

Span creation is gated on the trace context already existing, rather than falling back to a new root. There are four independent copies of this gate, pinned to v1.18.4 because the line numbers drift:

All four share the same shape:

iTraceID := cloudEvent[contribpubsub.TraceParentField]
if iTraceID == nil {
    iTraceID = cloudEvent[contribpubsub.TraceIDField]
}

if traceID, ok := iTraceID.(string); ok {          // <-- gate
    sc, _ := diag.SpanContextFromW3CString(traceID)
    ctx, span = diag.StartInternalCallbackSpan(ctx, "pubsub/"+msg.Topic, sc, spec)
}

When neither field is present the branch is skipped, span stays nil, and every later if span != nil guard silently no-ops.

1.18 added a guard on some paths for a non-string trace id, which logs at debug level and still creates no span. The nil case remains silent everywhere.

Secondary issues in the same area, unchanged from the original report:

• subscriptions.go:396 silently discards SpanContextFromW3CString parse errors — a malformed traceparent yields an invalid SpanContext and the code continues as if it were valid.
• pkg/runtime/subscription/subscription.go:348 seeds the resiliency runner with context.Background(), severing cancellation/deadline (and span) propagation from the outer handler ctx. Not the proximal cause, but worth fixing alongside.

Steps to Reproduce the Problem

Dapr 1.18.4, self-hosted, Redis pub/sub, Zipkin, samplingRate: "1". A subscriber on topic orders that logs its inbound traceparent header.

The Redis component uses the topic as the stream key and a data field, so a raw XADD is a faithful stand-in for any non-Dapr publisher — no cloud broker needed.

1. Control, published through Dapr:

curl -X POST http://localhost:3510/v1.0/publish/pubsub/orders \
  -H 'Content-Type: application/json' -d '{"orderId":"CASE1-via-dapr"}'

1. External publisher, straight to the broker with no trace context:

docker exec dapr_redis redis-cli XADD orders '*' data '{"specversion":"1.0","type":"com.dapr.event.sent","source":"external-system","id":"CASE2-no-traceparent","datacontenttype":"application/json","data":{"orderId":"CASE2-external"},"pubsubname":"pubsub","topic":"orders"}'

Observed. Both messages are delivered; only the control is traced:

APP-RECEIVED traceparent="00-20fa547b905bdb1cfa166aa0d13ad37e-a1dbda0979694042-01" body={... "traceparent":"00-20fa547b905bdb1cfa166aa0d13ad37e-80a94f78e9645aab-01" ...}
APP-RECEIVED traceparent="" body={"specversion":"1.0","source":"external-system","id":"CASE2-no-traceparent", ...}

Zipkin holds one trace for the delivery path — the control's, a publish span with a pubsub/orders child:

{"traceId":"20fa547b905bdb1cfa166aa0d13ad37e","id":"80a94f78e9645aab","name":"/v1.0/publish/pubsub/orders"}
{"traceId":"20fa547b905bdb1cfa166aa0d13ad37e","parentId":"80a94f78e9645aab","id":"a1dbda0979694042","name":"pubsub/orders"}

The external message produced no span.

This also reproduces via a manual DLQ re-queue on Azure Service Bus — moving messages from the DLQ back to the active queue through the portal/CLI drops the AMQP application properties carrying traceparent — which is how it was originally hit in production. The runtime code path is broker-agnostic.

Proposed Fix

Call StartInternalCallbackSpan unconditionally and parse a parent only when one is present. StartInternalCallbackSpan (pkg/diagnostics/tracing.go:167) already handles an invalid parent correctly; the callers never reach it.

var sc trace.SpanContext
if traceID, ok := iTraceID.(string); ok {
    sc, _ = diag.SpanContextFromW3CString(traceID)
} else if iTraceID != nil {
    log.Debugf("...")
}
ctx, span = diag.StartInternalCallbackSpan(ctx, "pubsub/"+msg.Topic, sc, spec)

Checked against a real SDK tracer provider — a zero parent yields a new root, a valid parent is still inherited:

zero parent  -> valid=true traceID=1518101967406ff7cd618e473a1e85b0 spanID=58e84f5a5f714031
valid parent -> traceID=20fa547b905bdb1cfa166aa0d13ad37e (inherited) spanID=c15543ca6e312712
spans exported: 2
  name="pubsub/orders" traceID=1518101967406ff7cd618e473a1e85b0 parentValid=false   <- new root
  name="pubsub/orders" traceID=20fa547b905bdb1cfa166aa0d13ad37e parentValid=true    <- child of remote parent

Applied at all four gates. Sampling is still governed by the tracing spec, so no spans are created where tracing is disabled.

Release Note

RELEASE NOTE: FIX PubSub subscription delivery now starts a new root span when the inbound message has no traceparent, so externally-published or manually re-queued (DLQ) messages remain traceable end-to-end.