#14599·substrate

Network stream events shouldn't be cloned

Author: sandreimCreated Jul 20, 2023Updated Aug 24, 2023
LabelsI3-bugI9-optimisation

Currently, the NetworkWorker creates one clone per event stream listener in https://github.com/paritytech/substrate/blob/master/client/network/src/service/out_events.rs#L219 .

For example, in a simple Versi test with 200 validators and 40 parachains we get around a bit above 1MB/s of traffic/events from the network: Screenshot 2023-07-20 at 12 30 23

Considering that there 5 such listeneres, I think this is a big problem with the current design, as in practice 5 additional copies are created wasting memory and CPU for free, raising the total internal traffic to more than 5MB/s. Screenshot 2023-07-20 at 12 41 08

We should strive for a 0 copy approach in our internal network stack architecture, for example we could use Bytes crate or just a simple Arc to avoid the unneeded extra copies. Additionally we could also do the per protocol filtering at lower level, instead of doing it in all of the consumers of these messages. It might worth looking into optimizing the protocol name check per messages as it seems wasteful since these are big strings based on genesis hashes. On connection we could negotiate some shorter name, or just an integer and use that instead.

FWIW this likely contributes to the CPU usage pattern we observe, see https://github.com/paritytech/polkadot-sdk/issues/702 .