Congestion control for staked/unstaked nodes

Problem

If client sends txs to validator and network latency is large, tps is very low (see table below). Contrary, if we place client/server in one datacenter TPS will be 100k higher.

Problem setup

Single node private network:

• Client with bench-tps in Amsterdam.
• Validator in Toronto.

Looks like it is due to QUIC receive window size set here. As pointed out by @lijunwangs, it is a adjusted by constant QUIC_UNSTAKED_RECEIVE_WINDOW_RATIO see quic.rs:25 which is by default 1. Setting this parameter to 10 (max value for staked) improves situation but cannot beat UDP results.

Protocol	Ratio*	TPS
UDP	-	6375
QUIC	1(default)	138
QUIC	10(default for staked)	1256

• QUIC_UNSTAKED_RECEIVE_WINDOW_RATIO see quic.rs:25

For the sake of experiment, I tried also QUIC_MAX_UNSTAKED_CONCURRENT_STREAMS = 1024 (default 128) together with QUIC_UNSTAKED_RECEIVE_WINDOW_RATIO = 10: TPS = 1478.

Consequence

If client application uses TPUClient to send txs, the variance of TPS is high. It happens because TPUClient will try sending to the next leaders and TPS will be strongly correlated with the network latency. Contrary, if we pick ThinClient and send always to the closest validator, TPS will be higher because validator will forward txs to the leader and it will use stacked quic configuration.

Current implementation summary

Unstaked nodes: the window size is set to minimum (1*MTU) which leads to low PPS on networks with packets drops. Staked nodes: the window size depends on the stake only and is varied in the range 2..10.

Proposed solution

Atm, discuss a congestion control model which maximizes utilization of network taking into account validator load.

Preparing a setup for reproducible testing is important sub task.

Discussed a bit with @lijunwangs and @ilya-bobyr