Per-message TTL entries leak in the hash wheel when the message is removed by purge or rollup, causing unbounded CPU and memory growth

Observed behavior

If a message with a Nats-TTL header is deleted before its TTL fires (by a subject purge, or a rollup), its entry in the TTL hash wheel is never removed. When the TTL comes due, expireMsgs finds the message already gone and leaves the entry in place, so it is collected again on every pass, forever. The entries are persisted in thw.db and survive restarts. A stream purge does not clear them; only deleting the stream does.

Since every stale entry is already in the past, the expiry timer re-arms at its 250 ms floor and each pass walks the whole wheel, on every replica. The effect is CPU and memory growing without bound, with nothing in the logs.

In our case (2.14.6, R3 file stream driven by message schedules with Nats-Schedule-TTL: 30s + Nats-Schedule-Rollup: sub, ~0.5 fires/s) the wheel reached 1.13 million stale entries in 4 weeks for a stream holding 11 messages, and each broker burned 200-270m CPU at idle. Under a 100m limit this fully throttled the brokers and broke client boots.

The subject-delete-marker branch of expireMsgs already handles this case (if sm == nil { fs.ttls.Remove(...) }); the plain branch does not. memStore has the same gap.

Expected behavior

A TTL entry whose message is already gone should be dropped from the hash wheel on the next expiry pass, the same way the subject-delete-marker branch of expireMsgs already does. Only a genuine removal failure (write error, closed store) should keep the entry for retry.

With that, the wheel stays bounded by the number of live TTL messages, thw.db does not grow, the expiry timer goes back to firing only when something is actually due, and a server that already carries stale entries heals itself on the first pass after upgrade, without deleting the stream.

Server and client version

Server: nats-server v2.14.6 (official nats:2.14.6-alpine image, 3-node cluster). The code path is unchanged on main as of 3c80d8f (2026-09-11), and in v2.14.7-RC.1 and v2.15.0-RC.1.

Client: nats.go v1.52.0 (only relevant for publishing the schedule SETs; the leak is entirely server-side).

Host environment

Kubernetes v1.36.0 on Talos Linux v1.13.5 (kernel 6.18, containerd 2.2.5), 3 nodes, amd64, 8 vCPU / 16 GB each (Hetzner Cloud). NATS deployed with the official Helm chart 2.14.6, one server per node, JetStream file store on a 10 Gi block volume per server.

Container resources requests == limits, cpu: 500m, memory: 512Mi (Guaranteed QoS), GOMAXPROCS=2. Not environment-specific: the leak is in the store code and reproduces in a unit test on a single fileStore / memStore with no cluster involved.

Steps to reproduce

Unit-level (deterministic, no cluster needed), against main:

1. Create a fileStore with AllowMsgTTL: true (AllowRollup: true is not required for the store-level repro).
2. StoreMsg("test.a", nil, nil, 1) ten times (1 s TTL). fs.ttls.Count() is 10.
3. fs.PurgeEx("test.a", 0, 0) - removes all ten messages. fs.ttls.Count() is still 10; this path does not touch the wheel.
4. Wait > 1 s and call fs.expireMsgs() (or let the timer fire).
5. fs.ttls.Count() is still 10. Every subsequent pass collects the same ten entries again. The same sequence on a memStore gives the same result.

I have this as two regression tests (TestFileStoreMessageTTLRemovedOutOfBandDoesNotLeakTHW, TestMemStoreMessageTTLRemovedOutOfBandDoesNotLeakTHW); both fail on main and pass with a small change to expireMsgs that drops the entry when removeMsg reports the message is already gone. Happy to open a PR.

Server-level (how we hit it):

1. Stream with AllowMsgTTL, AllowRollup, AllowMsgSchedules, retention limits.
2. Publish a schedule SET with Nats-Schedule: @every 5s, Nats-Schedule-TTL: 30s, Nats-Schedule-Rollup: sub, targeting a subject on the same stream.
3. Let it run. Each fire is rolled up by the next one after 5 s; its 30 s TTL then finds nothing. msgs/thw.db grows by one entry per fire and never shrinks, CPU on every replica climbs with it, and nats stream purge does not reset it.

Submission acknowledgment

• I am a human being writing in my own words and not an AI agent. I will not use an AI agent to communicate on my behalf in this issue, either directly or via copy-paste.