Runtime Upgrade (Big PoV) leading to collator peer reputation dropping (network stalled)
Author: crystalinCreated Nov 24, 2021Updated Aug 24, 2023
Conditions
- Internal permanent network (4 validators, 4 collators)
- Relay runtime 9122 (kusama-local) client 0.12.1
- Parachain runtime Moonbase 902 (substrate v0.9.11) migrating to 1000 (substrate v0.9.12)
Event
- Parachain runtime upgraded from 902 to 1000
- Parachain stalled at the block applying the upgrade
Logs
https://drive.google.com/file/d/1tNixS4n4_HHiaGz5bMURWM5-wGyGFmWb/view?usp=sharing
Observations
PoV for the block applying the runtime upgrade is 3000kb:
PoV size { header: 0.244140625kb, extrinsics: 2.8232421875kb, storage_proof: 3100.509765625kb }Validators are reporting the collators:
validator2 2021-11-24 00:56:49.379 DEBUG tokio-runtime-worker parachain::collator-protocol: Fetching collation failed due to network error hash=0x91139d3d4a8c5c31febfb614a2a509681204535403fe0a769e5d0837095dd2fe para_id=Id(1000) peer_id=PeerId("12D3KooWEokXpahv1QL2aJceYDnfjR6d2B4bTpc1xnY2wP7cgkiS") err=Network(Timeout)
validator2 2021-11-24 00:56:49.379 DEBUG tokio-runtime-worker parachain::network-bridge: peer=PeerId("12D3KooWEokXpahv1QL2aJceYDnfjR6d2B4bTpc1xnY2wP7cgkiS") rep=CostMinor("Some network error") action="ReportPeer"
Conclusion
When the PoV is too big and can't be fetch in time by the validators, those are reducing the reputation of the collator and preventing it from sending any further block, stalling the network
Source: paritytech/substrate