Root-cause the intermittent internal-MCP 401 once failure-mode telemetry lands (WORLDMONITOR-XZ)
Symptom
A signed internal-MCP tool call intermittently fails with 401 invalid_internal_mcp_signature. Sentry surfaces it as WORLDMONITOR-XZ (list-global-tenders HTTP 401), auth_kind=pro, tool=get_procurement_opportunities — paying customers, on a paid tool.
The real blast radius is wider than Sentry shows
Axiom wm_api_usage, filtered to user_agent startswith 'worldmonitor-mcp-edge', 30 days to 2026-09-13:
| route | 401 / anon / auth_401 |
|---|---|
/api/economic/v1/list-global-tenders |
3 |
/api/intelligence/v1/get-country-risk |
3 |
/api/infrastructure/v1/get-bootstrap-data |
1 |
Only list-global-tenders reaches Sentry, because its tool calls assertToolFetchOk and raises. The other two swallow the 401 and return empty — the caller sees no data rather than an error. So the Sentry issue understates this, and there may be more routes affected that never surface at all.
For scale: the same proxy succeeds in the hundreds over the same window (748 get-country-risk, 512 get-intel-timeline). This is rare and transient, not a broken path.
Why it was not fixed directly
server/gateway.ts deliberately collapses five distinct failures into one indistinguishable 401:
- timestamp outside the ±30s
INTERNAL_MCP_TIMESTAMP_WINDOW_SECONDS - malformed signature envelope
- missing
X-WM-MCP-User-Id - failed HMAC compare
- replay-nonce already spent
That is correct security design — a caller who can tell those apart has a forgery oracle — but it also made the failure unreproducible. auth_kind=anon on the gateway side only says "no identity resolved". Any fix would have been a guess.
What has been done
PR #8086 ships the diagnostic: a distinct server-side reason per failure mode (internal_mcp_ts_window, internal_mcp_sig_mismatch, internal_mcp_replay, internal_mcp_no_user, internal_mcp_malformed_sig, internal_mcp_bad_request), with the caller-facing response held byte-identical and pinned by a test. No behavior changed.
What this issue tracks
After #8086 has been live ~1 week, run:
wm_api_usage
| where reason startswith 'internal_mcp_'
| summarize count() by reason, route, bin(_time, 1d)
The distribution names the cause, and the real fix follows:
internal_mcp_ts_windowdominant → clock skew between the MCP edge and the gateway. Widening the window is the obvious lever but weakens replay protection; the replay-cache TTL (2 * WINDOW + 5) must move with it.internal_mcp_replaydominant → something is re-sending an already-signed request. A transport-level retry of an idempotent GET would do this; an application-level retry would not (it re-signs with a fresh nonce). Compare with the checkout-retry/idempotency-lock race, which is the same shape.internal_mcp_sig_mismatchdominant → genuine signer/verifier divergence. Suspect a deploy window where the MCP edge function and the gateway run different deployments with differentMCP_INTERNAL_HMAC_SECRETvalues; correlate the timestamps against deploy times.
Worth also deciding separately whether get-country-risk and get-bootstrap-data should be swallowing this — a 401 that silently becomes an empty result is its own bug, independent of the root cause.
Refs WORLDMONITOR-XZ. Follow-up to #8086.
Source: koala73/worldmonitor