#3004·hyperdx

[Feature Request] Trace-level AND search across spans

Author: milansanjeevCreated Aug 26, 2026Updated Sep 18, 2026
Labelsfeatureexternal

Summary

The search bar evaluates AND at the span/row level. There is no way to express a trace-level filter: return traces where property A appears on some span and property B appears on a (possibly different) span in the same trace.

That is a common query when correlation IDs are stamped once at the edge and service name (or other attributes) live on downstream spans.

Problem

A query like:

SpanAttributes.tuid:"abc123" AND ServiceName:checkout-service

Expected (Jaeger / Haystack / many APM UIs): traces that contain both predicates, even if they sit on different spans.

Actual: 0 results unless a single span has both fields.

This is easy to misread as a bug. The search bar is doing what it was built to do — filtering rows. The gap is that trace-level, cross-span AND is not expressible there.

Why this happens

Typical OpenTelemetry instrumentation stamps a correlation ID (request ID, tenant ID, user ID, etc.) on the gateway/entry span. Downstream services appear as separate spans in the same trace and often do not copy that ID onto every child.

So a real trace can look like:

Span ServiceName tuid
entry gateway-service abc123
child checkout-service (unset)
  • Trace-level AND matches this trace
  • Span-level AND matches nothing

Current workaround

Intersect two TraceId sets outside the where bar:

  1. TraceIds where any span has tuid = X
  2. TraceIds where any span has ServiceName = Y
  3. Fetch full traces for the intersection

That works against ClickHouse, but it is not something a user can do from the search bar.

Proposed behavior

Support a trace-scoped search mode (or syntax) where boolean operators apply at the trace:

  • A AND B → traces that contain a span matching A and a span matching B
  • Keep today’s span-level AND as an explicit mode, since it is still useful (“this span is slow and this span is checkout-service”)

Jaeger, Haystack, and several commercial APM products default to the trace-level interpretation for this class of query.

Additional context

Verified directly in ClickHouse on otel_traces-style data: the traces exist; no single row satisfies both predicates. Happy to share a minimal SQL sketch of the TraceId intersection if useful.