[BUG] Binance WS subscription quota uses Quota::per_second(5) (max_burst=5) — guarantees 1008 'Too many requests' on cold start

Description

On a cold start with a large instrument universe, the Binance adapter's WebSocket subscription messages violate the venue's per-connection rate limit within the first second. Binance closes the connection with 1008 / 'Too many requests'; the client reconnects, immediately re-subscribes (WebSocket reconnected, restoring subscriptions → subs.all_topics()), and is closed again — a reconnect storm that only ends when the venue's penalty window expires (~2 minutes).

This is not a missing limiter: the limiter is on the send path (send_subscribe() → send_text(..., BINANCE_RATE_LIMIT_KEY_SUBSCRIPTION)), but the configured quota Quota::per_second(5) is a token bucket with max_burst = 5, i.e. 5 messages instantly, then one every 200 ms → up to 9 messages inside the first rolling second. Binance's limit is a strict rolling window of 5 incoming messages per second per connection.

Environment

• nautilus_trader 2.0.0rc5 (release wheel; also checked on develop, commit 1b0a49d)
• Binance USDⓈ-M Futures, testnet; BinanceDataClientConfig(product_type=USD_M, environment=TESTNET)
• Reproduced with both transport_backend=SOCKUDO (default) and TUNGSTENITE
• Reproduced with a direct connection and through a local HTTP proxy
• Python 3.12, Linux x86_64

Steps to reproduce (with NautilusTrader)

Subscribe 215 symbols × (bars + funding rates) = 430 subscription commands in one on_start():

for sym in symbols:                        # 215 symbols
    self.subscribe_bars(bar_type[sym])
    self.subscribe_funding_rates(iid[sym])

Observed (stdout=Debug):

[WARN]  nautilus_network::websocket::client: Received close frame, terminating:
        code=1008, reason='Too many requests'                      × 54 … 61 per run
[INFO]  nautilus_binance::futures::websocket::streams::client:
        WebSocket reconnected, restoring subscriptions             × 54 … 61
[ERROR] ...futures::websocket::streams::handler:
        Failed to send subscribe request: send failed: timeout waiting for active state

The number of 1008 closes tracks the number of subscription messages (measured: 75 commands → 9 closes, 215 → 27, 430 → 54–61).

Minimal reproduction without NautilusTrader (decisive)

The venue rule can be isolated with a raw socket, no credentials required:

import asyncio, json, websockets

async def main():
    async with websockets.connect("wss://stream.binancefuture.com/ws",
                                  ping_interval=None) as ws:
        # exactly what Quota::per_second(5) emits: burst of 5, then one per 200 ms
        for i in range(5):
            await ws.send(json.dumps({"method": "SUBSCRIBE",
                                      "params": ["btcusdt@kline_1m"], "id": i + 1}))
        for i in range(20):
            await asyncio.sleep(0.2)
            await ws.send(json.dumps({"method": "SUBSCRIBE",
                                      "params": ["ethusdt@kline_1m"], "id": i + 6}))
        await asyncio.sleep(3)

asyncio.run(main())
# → websockets.exceptions.ConnectionClosedError:
#   received 1008 (policy violation) Too many requests

Measured matrix (each case on a fresh connection, 4 s settle):

Two conclusions that matter for adapter design:

1. The limit is on messages, not streams — a single SUBSCRIBE carrying 50 streams is fine.
2. A one-off burst above 5 is silently dropped (no close frame), so a subscription can end up silently incomplete; only a sustained rate above 5/s triggers the close.

Suggested fix

crates/adapters/binance/src/common/consts.rs:

// before: max_burst = 5, replenish 200 ms → up to 9 messages in the first rolling second
pub static BINANCE_WS_SUBSCRIPTION_QUOTA: LazyLock<Quota> = LazyLock::new(|| {
    Quota::per_second(NonZeroU32::new(5).expect("non-zero")).expect("valid constant")
});

// after: single-cell quota → at most 4 messages in any rolling second
pub const BINANCE_WS_SUBSCRIPTION_MIN_INTERVAL_MS: u64 = 300;
pub static BINANCE_WS_SUBSCRIPTION_QUOTA: LazyLock<Quota> = LazyLock::new(|| {
    Quota::with_period(Duration::from_millis(BINANCE_WS_SUBSCRIPTION_MIN_INTERVAL_MS))
        .expect("valid constant")
});

Note that Quota::per_second(4) would not be a fix: it still sets max_burst = 4 (4 instantly + one per 250 ms = 8 in the first rolling second). A burst-1 quota is required.

Measured effect of the patch (locally built wheel, 215 symbols / 430 commands, no app-level pacing)

Wire-level verification of the patched build (temporary debug! in WebSocketClientInner::send_text, because release_max_level_debug removes all trace! calls):

messages sent            : 430
bytes                    : min=58 median=64 max=73   (all single-stream)
keyed_rate_limit         : True=430 / False=0
limiter_wait_ms          : min=0 median=299 p90=300 max=301
  └ zero-wait messages   : 3   (= exactly one per connection: the initial cell)
bursts (<20 ms clusters) : 430 × size-1  ⇒ zero bursting
max messages in any 1 s  : 12 globally (3 connections ⇒ ≈4 per connection; limit is 5)
1008 closes              : 0

Also: the limit is per connection, not per IP (measured)

Subscriptions are sharded across connections (Pool slot 1/2 connected, 200 streams per slot) and each connection gets its own keyed quota. Concurrent raw connections (burst 2, then one message per 300 ms each):

⇒ the per-connection limiter design is correctly shaped; only the quota semantics need fixing.

Suggested refinement: coalesce and pace

With the fix above, 430 subscriptions are sent as 430 single-stream messages, so the subscription ramp takes ~60 s. Since the venue counts messages, not streams, it would be better to keep batching (the current code already produces e.g. one 200-stream message) and pace the messages: e.g. one message per 300 ms carrying up to N streams ⇒ 430 streams subscribed within a couple of seconds with zero 1008.

Related defects found while investigating

1. log::trace! is compiled out of the release wheel. Cargo.toml enables the log crate's release_max_level_debug feature, so every trace! in nautilus_network (including Sending text frame ({} bytes) in crates/network/src/websocket/client.rs) is statically removed in release builds. Wire-level investigation currently requires a debug build — this is not documented.
2. Module-level log filters cannot raise verbosity. Logger::enabled() (crates/common/src/logging/logger.rs) gates on stdout_level / fileout_level before filter_policy.should_skip() runs, so stdout=Info;nautilus_network::websocket=Trace is a no-op by construction. The from_spec doc example (my_crate::module=Debug) implies otherwise.
3. from_spec puts keys without :: into component_level, which is matched by exact equality (crates/common/src/logging/config.rs), so nautilus_network=Trace matches nothing at all. Together with (2) this makes the obvious configuration silently useless.
4. LoggerConfig.__new__ has no module_levels parameter, so a LoggerConfig rebuilt in Python silently drops all module filters produced by from_spec.
5. The reconnect buffer bypasses the keyed quota. drain_reconnect_buffer() (crates/network/src/websocket/client.rs) writes with writer.send(...) directly, without await_rate_limit_or_closed(keys), while send_text() does apply it. Messages buffered during a disconnect are therefore replayed unmetered.
6. Reconnect re-subscribe is unconditional and full: every reconnect re-sends subs.all_topics(). Combined with a venue penalty window this forms the storm loop; a capped backoff for re-subscription would make the client well-behaved on its own.

Workaround available today (no patch)

Pace at the actor level with clock.set_timer, one subscription action per ≥330 ms (note one symbol = 2 actions for bars + funding rates). Measured: 1008 61 → 1, Failed to send subscribe request 40 → 0, missing 1-minute bars 12 → 0. It does not reach the adapter's internal re-subscribe, hence the residual single closure.