You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew.
A box for each service, arrows between them, latency on every edge.
It feels like magic, and — more dangerously — it feels complete. "The agent traces everything" is the sentence repeated in every onboarding doc.
This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see.
The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a backed by MySQL, a backed by PostgreSQL, and Kafka carrying events between them.
A user is created, an event is published, a notification is sent.
I didn't want to draw that topology.
A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter.
I wanted the dependency graph generated from live traffic, so it would always reflect what the system actually does.
Grafana Tempo does exactly this.
Its processor reads matched client/server span pairs out of trace data and emits a metric — — that Grafana renders as a node graph.
No edge is ever wired by hand.
The topology is derived, continuously, from real spans.
The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: Then I looked for the one edge I actually cared about — , the asynchronous hop over Kafka.
It wasn't there.
The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken.
The event isn't getting across.
Go debug the consumer.
So I checked.
And the consumer was completely fine. had consumed every event and written every corresponding row to PostgreSQL.
Its database edge was lit.
Liveness was perfect; the feature worked end to end.
That is the trap.
A missing edge looks exactly like a broken feature, and the instinct is to go "fix" something that was never broken.
The defect wasn't in the message path at all — it was in the observability of the message path.
Those are two different failure domains that happen to render identically on a dashboard.
So I stopped trusting the picture and went to the source of truth: the trace store.
Evidence over assertion Two TraceQL queries settled it.
Zero Kafka messaging spans anywhere in the system.
Zero traces spanning both services.
The agent — this build, under this Spring Boot version — simply was not instrumenting the Kafka client.
Nothing errored.
No warning was logged.
The map wasn't wrong about the data; the data was never produced.
This is the part worth sitting with: a green dashboard would have let me believe the chain was fully traced.
The absence of red is not the presence of coverage.
How the graph is actually built The pipeline that produces the map is worth seeing, because one of its links is also a silent path you have to consciously turn on: Three deliberate decisions shaped it: Generated edges, not declared ones.
Tempo reads client/server span pairs and emits the edge metric.
The graph is a test, not a drawing — it fails when reality diverges from expectation, which is exactly what makes it valuable.
One processor, scoped on purpose.
I enabled only — deliberately not .
The latter generates RED/latency series and would have inflated cardinality for a deliverable that was strictly about topology.
Minimal blast radius, single responsibility.
A push boundary you must open explicitly.
Tempo remote-writes its generated metrics into Prometheus — the inverse of every other component, which Prometheus scrapes.
That requires flipping Prometheus into a receiver with .
Forget it, and the metrics silently never land.
Same lesson as the Kafka gap, one layer down: the data paths that fail quietly are the ones nobody turned on.
One missing edge, two real causes There's a subtlety the graph forced me to articulate.