[BUG] Huge performance degradation for the server connected with too many tracking bcast clients

@ShooterIT

Hello, I just found performance degradation during client TRACKING test.

Describe the bug

If the redis server is connected with too many tracking bcast clients, the other normal clients will receive huge latency for the normal commands requests.

To reproduce

I just ran the test on a Master-slave redis service. The master only received write requests of command 'SET', number of connected normal clients is 400, and OPS is about 1000. The slave received read requests of command 'GET', number of connected normal clients is 400, and OPS is about 18000. The slave was also connected with 1000 tracking bcast clients, the tracking prefix is just matched with the keys written to the master.

Then, the latency of 'GET' by average is more than 7ms. If I disconncted the tracking clients, the latency is less than 0.1ms. The latency degradation is very obviously relevant to the tracking clients.

analysis

I added some logs and found that the latency was from handleClientsWithPendingWrites. During calling this function, server need to write buffer to the sockets of clients(writeToClient), it cost at least 6us for calling writeToClient each time, so it cost more than 7000us to process total clients(including 1000 tracking clients) in handleClientsWithPendingWrites.

Then, the cost of entire event loop is also more than 7000us and even more. So, the slave server could not get into the next event loop as soon as possible to respond to the read request that had come.

` 3273:S 21 Jul 2025 10:51:32.507 * handleClientsWithPendingWrites cost: 7780us, processed_num_clis: 1183, buflen: 140137

3273:S 21 Jul 2025 10:51:32.516 * handleClientsWithPendingWrites cost: 7879us, processed_num_clis: 1150, buflen: 209930 3273:S 21 Jul 2025 10:51:32.525 * handleClientsWithPendingWrites cost: 7828us, processed_num_clis: 1166, buflen: 290028 3273:S 21 Jul 2025 10:51:32.534 * handleClientsWithPendingWrites cost: 7834us, processed_num_clis: 1178, buflen: 215100 3273:S 21 Jul 2025 10:51:32.542 * handleClientsWithPendingWrites cost: 7587us, processed_num_clis: 1155, buflen: 288464 `

For some reason, replying to the tracking clients must be handled by main thread. And the number of handled clients is unlimited.

`int handleClientsWithPendingWrites(void) { listIter li; listNode *ln; int processed = listLength(server.clients_pending_write);

listRewind(server.clients_pending_write,&li);
while((ln = listNext(&li))) {
    client *c = listNodeValue(ln);

     ... ...
    /* Let IO thread handle the client if possible. */
    if (server.io_threads_num > 1 &&
        !(c->flags & CLIENT_CLOSE_AFTER_REPLY) &&
        !isClientMustHandledByMainThread(c))
    {
        assignClientToIOThread(c);
        continue;
    }
    /* Try to write buffers to the client socket. */
    if (writeToClient(c,0) == C_ERR) continue;

    ... ...
}
return processed;

}`

` int isClientMustHandledByMainThread(client *c) {

if (c->flags & (CLIENT_CLOSE_ASAP | CLIENT_MASTER | CLIENT_SLAVE |
                CLIENT_PUBSUB | CLIENT_MONITOR | CLIENT_BLOCKED |
                CLIENT_UNBLOCKED | CLIENT_TRACKING | CLIENT_LUA_DEBUG |
                CLIENT_LUA_DEBUG_SYNC))
{
    return 1;
}
return 0;

}`

Expected behavior

Just in my opinion, In condition of client-side caching, with usage of thread pool, the redis server may establish many connection tracking clients, which should not have negative effect on the service for the other normal clients.

Additional information

Redis version: 8.2-in (tested on latest stable) Environment: Docker containers on Linux (kernel 5.15) CLient: Jedis