Bug Report: MaxListenersExceededWarning on every response since 3.17.0 (instrumentation-router adds a finish listener per router layer)
Description
Since 3.17.0, every HTTP response on the self-hosted api, worker and ws services emits:
(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected.
11 finish listeners added to [ServerResponse]. MaxListeners is 10.
Use emitter.setMaxListeners() to increase limitThe cause is @opentelemetry/instrumentation-router, which registers a prependOnceListener('finish') for every router layer a request traverses. Any request crossing more than ~10 layers passes Node's default maxListeners of 10.
This is not an actual memory leak — the listeners are attached per ServerResponse and released with it — but it produces a very large volume of useless log output on any deployment with ENABLE_OTEL=true. In our integration environment it accounted for roughly a 49% increase in the total log volume of the Novu services.
3.15.0 was unaffected with the identical ENABLE_OTEL=true configuration. Two changes in between look relevant:
@opentelemetry/instrumentation-router0.56.0→0.61.0- NestJS
10.4.18→11.1.21in 3.17.0, acrossapi,workerandws— precisely the three affected services
Note this is the router instrumentation, not the express one, so the ignoreLayersType: ['middleware'] configuration added to otel-init.ts for novuhq/novu#10722 does not suppress it, nor does OTEL_NODE_DISABLED_INSTRUMENTATIONS=express.
Reproduction steps
- Run
ghcr.io/novuhq/novu/api:3.18.0with MongoDB and a Redis/Valkey reachable. - Set
ENABLE_OTEL=trueandOTEL_EXPORTER_OTLP_ENDPOINTto any endpoint (a collector need not actually be listening). curl http://localhost:3000/v1/health-check- One
MaxListenersExceededWarningis logged per request.
️ Attribution of the listeners
Running the same image with NODE_OPTIONS=--require probe.js, where probe.js patches http.ServerResponse.prototype.{on,once,addListener,prependListener,prependOnceListener} and records the registering module per response:
=== 11 finish listeners on ONE ServerResponse ===
1. node internal: node:_http_server [.on()]
2. node internal: node:events [.prependListener()]
3. @opentelemetry/instrumentation-router @ 0.61.0 [.prependOnceListener()]
4. node internal: node:events [.prependListener()]
5. @opentelemetry/instrumentation-router @ 0.61.0 [.prependOnceListener()]
6. node internal: node:events [.prependListener()]
7. @opentelemetry/instrumentation-router @ 0.61.0 [.prependOnceListener()]
... pattern continues past 19 on this routeEach node:events entry is the OpenTelemetry context manager's bound wrapper around the preceding registration.
Expected behavior
Enabling OpenTelemetry should not emit a MaxListenersExceededWarning per request.
Actual behavior
One warning per HTTP response on every OTel-enabled Node service.
✅ Evidence this is not a real leak
- Every occurrence reports exactly 11 listeners at the trigger point — a constant overshoot, never an escalating count.
- Container working set stays flat:
api427 → 429 MiB against an 8 GiB limit,worker265 → 266 MiB,ws290 → 292 MiB, over hours of traffic. - Zero restarts, no OOM kills.
Workaround
Adding router to the disabled instrumentations removes it. Verified on api:3.18.0 over 5 requests to /v1/health-check:
OTEL_NODE_DISABLED_INSTRUMENTATIONS |
Warnings |
|---|---|
express |
4 |
express,router |
0 — boots clean, all requests 200 |
Possible fixes upstream
- Configure
@opentelemetry/instrumentation-routerinlibs/application-generic/src/tracing/otel-init.tsthe wayinstrumentation-expressalready is, so it does not attach a listener per layer. - Or raise the limit for response objects once during OTel setup (
res.setMaxListeners(n)/events.setMaxListeners), which silences the warning without dropping the spans. - Or disable
routerby default for self-hosted, alongside the existing express handling.
Environment
- Novu 3.18.0, self-hosted (
IS_SELF_HOSTED=true), officialghcr.io/novuhq/novu/*images - Node 20 (image default), Kubernetes,
ENABLE_OTEL=trueexporting OTLP/HTTP - Also reproduced locally with plain Docker, so it is not Kubernetes-specific
Source: novuhq/novu