C# watchTradesForSymbols can return a trade twice when an append lands between getLimit and the cache snapshot

Operating System

Windows 11 x64

Programming Language

C#

CCXT Version

4.5.84

Description

Steps to reproduce

No credentials or WebSocket connection are needed. With newUpdates enabled (the default Pro trade-watch behavior), force one append between getLimit and filterBySinceLimit on a single ArrayCache:

using System;
using System.Collections.Generic;
using System.Linq;
using ccxt;
using ccxt.pro;

var exchange = new BaseExchange(new Dictionary<string, object>());
var cache = new ArrayCache();
Dictionary<string, object> Trade(string id, int timestamp) => new()
{
    ["symbol"] = "BTC/USDT", ["id"] = id, ["timestamp"] = timestamp
};

cache.append(Trade("a1", 1));
cache.append(Trade("a2", 2));
var firstLimit = cache.getLimit("BTC/USDT", null); // 2
cache.append(Trade("a3", 3)); // receive-thread append after count read
var first = exchange.filterBySinceLimit(cache, null, firstLimit, "timestamp", true);

cache.append(Trade("a4", 4));
var secondLimit = cache.getLimit("BTC/USDT", null); // 2
var second = exchange.filterBySinceLimit(cache, null, secondLimit, "timestamp", true);

Console.WriteLine(string.Join(",", first.Select(row => ((Dictionary<string, object>)row)["id"])));
Console.WriteLine(string.Join(",", second.Select(row => ((Dictionary<string, object>)row)["id"])));
// a2,a3
// a3,a4  (a3 was appended only once)

The same sequence occurs when a real WebSocket trade arrives after the watch method has read the count but before it copies the cache. Pass an omitted limit (null), not 0: an explicit zero introduces a separate full-cache replay and would obscure this race.

Expected behavior

Successive new-update resolutions should not both contain a trade that was appended only once, even when a receive-thread append races a watch call. The count and returned rows should describe the same cache state.

Actual behavior

The deterministic interleaving returns a3 in both calls without a repeated input. In a two-minute C# public WebSocket observation with the optional limit omitted, 135 observed caches produced 45,417 returned-row occurrences (42,553 tracked; 2,864 not tracked) and 289 occurrences where a previously returned ID was selected again without another append of that ID. All 289 met the recorded ordering: the trade was appended after the first count read, was present in its later snapshot, and was still present when the next count was read. One Binance XRP/USDT example had first count-read sequence 23, single append/snapshot sequence 24, and next count-read sequence 25; the second call returned that same ID with an effective limit of 2. These are occurrences, not distinct trade IDs; the observation is bounded and its instrumentation can affect scheduling. It does not measure the uninstrumented race rate or attribute earlier production totals to this race.

Root cause

ArrayCache.getLimit(symbol, limit) reads the symbol's update count and sets its clear flag under lockObject, then releases the lock. filterBySinceLimit subsequently invokes toArray and tail-slices the cache; even in v4.5.84, which copies SlimConcurrentList atomically under its own read lock, this snapshot is not atomic with the earlier update-count read. An append between the two reads can enter the first returned tail and, because it occurs after the counter was marked for clearing, be counted again in the next call. A per-symbol cache and one unique venue trade are sufficient.

Suggested fix

Consume the update count and choose the returned rows from one consistent cache snapshot, or use a stable delivery cursor across resolutions. Preserve independent per-symbol/global polling scopes and the bounded cache's eviction semantics. Add a deterministic C# test that pauses an append between count and snapshot, then checks that the same unique ID is returned only once. Test venue redelivery separately (a second append of the same ID is different from this defect). Switching getLimit(symbol, limit) to getLimit(null, limit) alone does not make the count and snapshot atomic.