[Bug] Call-end events carry no ref; CDR correlation relies solely on in-process memory and is lost on apiserver restart
Summary
A call's end event from Routr carries no ref, so correlating it back to the call's CDR relies entirely on an in-process cache inside the apiserver. If the apiserver restarts between a call's start and end events, that call's CDR is orphaned and permanently incomplete.
Steps to Reproduce
- apiserver receives a
routr.call.*start event whoseextraHeadersincludeX-Call-Ref—transformEvent.ts:70-71setsreffrom it and the point is written with that tag. - apiserver restarts before the same call's end event arrives. This wipes the in-memory
refByCallIdcache increateInfluxDbPub.ts(line 51). - The end event arrives with
extraHeaders: {}(confirmed in production — see below). The guard attransformEvent.ts:65(extraHeaders && Object.keys(extraHeaders).length > 0) is false, so the whole header-processing block is skipped — norefis set on the end event, not even a synthetic one. GetCall(ref)queries viacreateFetchSingleCall.ts:39, which filters on thereftag before pivoting. The end point (written with noreftag) is invisible to that query.
Expected Behavior
A call's CDR should show a complete record — status and duration populated — regardless of whether the apiserver process restarted between the call's start and end events.
Actual Behavior
The CDR is stuck with a start and no status, duration: 0, permanently — the end event's InfluxDB write never got a ref tag and nothing rescues it after restart.
Evidence (production apiserver logs, 2026-08-30)
18:08:53.675 received nats event {"callId":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","extraHeaders":{...,"X-Call-Ref":"222c6f81-d69a-432e-bb41-551d407ce70f",...},"startTime":"2026-08-30T18:08:53.674Z",...}
18:08:53.675 writing event to InfluxDB {"name":"cdr","tag":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","data":{...,"ref":"222c6f81-d69a-432e-bb41-551d407ce70f",...}}
18:10:03.490 received nats event {"callId":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","extraHeaders":{},"endTime":"2026-08-30T18:10:03.488Z","hangupCause":"NO_ANSWER"}
18:10:03.490 writing event to InfluxDB {"name":"cdr","tag":"DGoldhnEZ6enSoxJ2JmzNZBoGiDytSAI","data":{"endedAt":1788113403.488,"status":"NO_ANSWER"}}Note the end event's own write has no ref field at all. It only survives correlation because createInfluxDbPub's refByCallId cache (populated by the start event, same process) reapplies the tag before the point is written. Had the process restarted between these two lines, that reapplication would not happen, and this call's CDR would be stuck incomplete forever.
The chain
- Routr publishes call-start and call-end on
routr.call.*; apiserver subscribes inmods/apiserver/src/events/nats.ts:46, wired frommods/apiserver/src/index.ts:49. - The start event's
extraHeadersincludeX-Call-Ref, somods/apiserver/src/events/transformEvent.ts:70-71setsrefdirectly from it. - The end event arrives with
extraHeaders: {}. The outer guard attransformEvent.ts:65is false, so the entire block — including theelsebranch at lines 72-80 that would otherwise mint/reuse a synthetic ref viasyntheticRefByCallId— never runs for it. The end event leavestransformEventwith norefwhatsoever. - The only thing that rescues correlation today is
mods/apiserver/src/events/createInfluxDbPub.ts's own, separate in-memory cache —refByCallId(declared line 51, acreatePerCallCache<string>(24h)) — which reapplies the rememberedreftag to any write for the samecallIdthat doesn't carry one (lines 57-64). Its own comment (lines 46-50) documents exactly this: "not every event's payload carries ref." mods/apiserver/src/calls/createFetchSingleCall.ts:39filters on thereftag before pivoting, so a point written without that tag is invisible toGetCall(ref).
Worth noting: transformEvent.ts's syntheticRefByCallId and createInfluxDbPub.ts's refByCallId already share the same underlying primitive (createPerCallCache, in mods/apiserver/src/events/createPerCallCache.ts) — but as two independent instances solving two different problems (minting a ref for SIP-originated calls that never get one vs. reapplying a known ref to ref-less events). Neither is persisted, and both are wiped on restart.
Impact
- Data loss on restart. Any call whose end event lands after an apiserver restart gets a CDR permanently stuck with a start and no status/duration.
- The correlation is currently private to
createInfluxDbPub. Any future consumer of the same call-end event that also needs itsref(e.g. something bridging call-end onto a live call-tracking stream) would have to duplicate this in-memory cache rather than reuse one, since it isn't a shared component today.
Suggested direction (not prescriptive)
- Have Routr echo
X-Call-Refback on call-end events too, soextraHeaderson the end event is non-empty andtransformEvent.tscan setrefdirectly, the same way it already does for the start event. Cleanest fix — removes the in-memory dependency entirely — but requires a change (and its own release) in theroutrrepo. Checkedfonoster/routrfor a companion issue; found none currently open. - Make the ref correlation survive a restart, e.g. writing a placeholder point tagged with
refat call-start time so pivot always has something to match the end point against, or backing the correlation with something durable instead of (or in addition to) the in-memory cache. Keeps the change insidefonoster/fonoster, but is more involved than the cache-lift originally considered, since a shared cache utility (createPerCallCache) already exists — the gap is persistence, not code reuse.
Priority
P1
Source: fonoster/fonoster