watch_order_book (ccxt.pro): memory leak on kucoin/htx/gate since 4.5.77, still present in 4.5.84 (4.5.54 stable)

Summary

We run a long-lived process that subscribes to watch_order_book for a fixed set of symbols on several exchanges. On ccxt 4.5.54 its RSS is flat over weeks of uptime. On 4.5.77 we measured a sustained leak of roughly 70–90 MB/min. We have now re-tested on the current release, 4.5.84, in a separate isolated process/venv, side by side with a 4.5.54 control under the same load, and the leak is still there (numbers below) — growth accelerates within each cycle until the process hits our memory cap.

Context: in #30571 you asked us to upgrade and confirm the HyperLiquid error-frame fix (fb23ddb / #29006). We confirmed it with our reproducer on 4.5.84 (thank you), but we can't roll the upgrade out because of this leak, so we're filing it as the concrete blocker.

Environment

• ccxt: 4.5.84 (leaking), 4.5.77 (leaking, earlier test), 4.5.54 (stable)
• Python 3.12.3, uvloop 0.21.0, Linux 6.8.0 x86_64 (Ubuntu)
• ccxt installed with pip install --no-deps (only ccxt and curl-cffi changed to the version under test)

What we run

One process, three exchanges, one ccxt.pro instance per exchange:

• exchanges: kucoin, htx, gate
• ~50 symbols per exchange (~150 concurrent watch_order_book subscriptions)
• read-only public order-book streaming, no REST in the hot path, no private endpoints
• we process at most ~1 update per 2 s per symbol, but the watch_order_book subscriptions themselves run continuously

Control that narrows it to watch_order_book

In the same test, a second process on the same ccxt version (4.5.84), same exchanges, subscribed only to watch_liquidations (no order books) stayed flat at ~166 MB for the whole ~60 min. Only the watch_order_book process grew — so we believe the leak is in watch_order_book or the local order-book state kept per symbol, not in the websocket/connection layer in general.

Numbers (4.5.84)

RSS sampled every ~60–70 s. A 2500 MB cap restarts the process when hit; each cycle is one growth-to-cap run:

cycle	start RSS	end RSS	wall time	average
1	252 MB	2509 MB (cap)	~12 min	~188 MB/min
2	239 MB	~2450 MB (cap)	~25 min	~88 MB/min
3	120 MB	434 MB when we stopped the test	24 min	~13 MB/min, accelerating

Growth is not linear within a cycle: it starts slowly (tens of MB over several minutes) and accelerates sharply towards the end (hundreds of MB within one sampling interval near the cap). We saw the same shape on 4.5.77. That suggests the leaked amount scales with something that itself grows over the process lifetime (accumulated per-symbol / per-update state?) rather than a fixed per-message overhead.

Control (4.5.54): a process with the same exchanges, symbols and config, running on another of our hosts in the same time window, stayed in a 352–371 MB band for the full hour — consistent with the plateau we see from this code path in long-running use on 4.5.54.

Not done yet

• No bisect between 4.5.54 and 4.5.77 (4.5.54 is our last known-good version).
• No minimal standalone reproducer yet — ours runs inside our own service. We're happy to build one (single script, one exchange, N watch_order_book symbols, RSS printed once a minute) if that helps; tell us if it's useful, or if the numbers already point to a specific change.

Ask

Any pointers on what changed in watch_order_book / local order-book handling for kucoin/htx/gate between 4.5.54 and 4.5.77 would help us narrow it down faster than a manual bisect. We can also share the raw RSS time series (one row per minute).