#885·fonoster

[Bug] Call-end events carry no ref; CDR correlation relies solely on in-process memory and is lost on apiserver restart

Author: psandersCreated Sep 7, 2026Updated Sep 7, 2026
Labelsbug

Summary

A call's end event from Routr carries no ref, so correlating it back to the call's CDR relies entirely on an in-process cache inside the apiserver. If the apiserver restarts between a call's start and end events, that call's CDR is orphaned and permanently incomplete.

Steps to Reproduce

  1. apiserver receives a routr.call.* start event whose extraHeaders include X-Call-ReftransformEvent.ts:70-71 sets ref from it and the point is written with that tag.
  2. apiserver restarts before the same call's end event arrives. This wipes the in-memory refByCallId cache in createInfluxDbPub.ts (line 51).
  3. The end event arrives with extraHeaders: {} (confirmed in production — see below). The guard at transformEvent.ts:65 (extraHeaders && Object.keys(extraHeaders).length > 0) is false, so the whole header-processing block is skipped — no ref is set on the end event, not even a synthetic one.
  4. GetCall(ref) queries via createFetchSingleCall.ts:39, which filters on the ref tag before pivoting. The end point (written with no ref tag) is invisible to that query.

Expected Behavior

A call's CDR should show a complete record — status and duration populated — regardless of whether the apiserver process restarted between the call's start and end events.

Actual Behavior

The CDR is stuck with a start and no status, duration: 0, permanently — the end event's InfluxDB write never got a ref tag and nothing rescues it after restart.

Evidence (production apiserver logs, 2026-08-30)

18:08:53.675 received nats event {"callId":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","extraHeaders":{...,"X-Call-Ref":"222c6f81-d69a-432e-bb41-551d407ce70f",...},"startTime":"2026-08-30T18:08:53.674Z",...}
18:08:53.675 writing event to InfluxDB {"name":"cdr","tag":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","data":{...,"ref":"222c6f81-d69a-432e-bb41-551d407ce70f",...}}

18:10:03.490 received nats event {"callId":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","extraHeaders":{},"endTime":"2026-08-30T18:10:03.488Z","hangupCause":"NO_ANSWER"}
18:10:03.490 writing event to InfluxDB {"name":"cdr","tag":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","data":{"endedAt":1788113403.488,"status":"NO_ANSWER"}}

Note the end event's own write has no ref field at all. It only survives correlation because createInfluxDbPub's refByCallId cache (populated by the start event, same process) reapplies the tag before the point is written. Had the process restarted between these two lines, that reapplication would not happen, and this call's CDR would be stuck incomplete forever.

The chain

  1. Routr publishes call-start and call-end on routr.call.*; apiserver subscribes in mods/apiserver/src/events/nats.ts:46, wired from mods/apiserver/src/index.ts:49.
  2. The start event's extraHeaders include X-Call-Ref, so mods/apiserver/src/events/transformEvent.ts:70-71 sets ref directly from it.
  3. The end event arrives with extraHeaders: {}. The outer guard at transformEvent.ts:65 is false, so the entire block — including the else branch at lines 72-80 that would otherwise mint/reuse a synthetic ref via syntheticRefByCallId — never runs for it. The end event leaves transformEvent with no ref whatsoever.
  4. The only thing that rescues correlation today is mods/apiserver/src/events/createInfluxDbPub.ts's own, separate in-memory cache — refByCallId (declared line 51, a createPerCallCache<string>(24h)) — which reapplies the remembered ref tag to any write for the same callId that doesn't carry one (lines 57-64). Its own comment (lines 46-50) documents exactly this: "not every event's payload carries ref."
  5. mods/apiserver/src/calls/createFetchSingleCall.ts:39 filters on the ref tag before pivoting, so a point written without that tag is invisible to GetCall(ref).

Worth noting: transformEvent.ts's syntheticRefByCallId and createInfluxDbPub.ts's refByCallId already share the same underlying primitive (createPerCallCache, in mods/apiserver/src/events/createPerCallCache.ts) — but as two independent instances solving two different problems (minting a ref for SIP-originated calls that never get one vs. reapplying a known ref to ref-less events). Neither is persisted, and both are wiped on restart.

Impact

  • Data loss on restart. Any call whose end event lands after an apiserver restart gets a CDR permanently stuck with a start and no status/duration.
  • The correlation is currently private to createInfluxDbPub. Any future consumer of the same call-end event that also needs its ref (e.g. something bridging call-end onto a live call-tracking stream) would have to duplicate this in-memory cache rather than reuse one, since it isn't a shared component today.

Suggested direction (not prescriptive)

  1. Have Routr echo X-Call-Ref back on call-end events too, so extraHeaders on the end event is non-empty and transformEvent.ts can set ref directly, the same way it already does for the start event. Cleanest fix — removes the in-memory dependency entirely — but requires a change (and its own release) in the routr repo. Checked fonoster/routr for a companion issue; found none currently open.
  2. Make the ref correlation survive a restart, e.g. writing a placeholder point tagged with ref at call-start time so pivot always has something to match the end point against, or backing the correlation with something durable instead of (or in addition to) the in-memory cache. Keeps the change inside fonoster/fonoster, but is more involved than the cache-lift originally considered, since a shared cache utility (createPerCallCache) already exists — the gap is persistence, not code reuse.

Priority

P1