Big Block Issue Optimization

Description

Background: Currently, if a large block is encountered, the SDK may fail to index properly. For example, Kusama: 25845986.

It behaves differently under HTTP and WS, and here’s the information we know so far:

• When using WS, it may cause an out-of-memory (OOM) error because the block is too large (at this point, we cannot do anything, and the process will be blocked), potentially leading to a 1006 connection closed error.
• When using HTTP, it may fail due to too many concurrent requests, resulting in many HTTP request failures.

Issues:

• How to detect large blocks?
• WS cannot retrieve large block results properly, causing OOM.
• HTTP cannot handle too many concurrent requests for large blocks, leading to timeouts and connection closures.

Optimizations:

• Implement an RPC response size detection mechanism. If the response exceeds the set size, record this block in the metadata, indicating that the current block height is a large block, which may warrant increasing the timeout.
• Add logs during the processing of large blocks to notify that the process is still running normally.