vminsert: optimize NewRelic ingestion memory usage by batching parsed rows
Is your feature request related to a problem? Please describe
VictoriaMetrics currently supports ingesting NewRelic infrastructure agent events via:
/newrelic/infra/v2/metrics/events/bulk
The current NewRelic parser path reads the full request body, parses it with fastjson, converts all events into an intermediate Rows structure, and then passes the parsed rows to the write callback.
A single NewRelic event can produce multiple VictoriaMetrics samples:
- numeric fields become samples
- string fields become labels attached to every sample
- timestamp is handled as the sample timestamp
When a request contains many events or large label values, the parser accumulates all parsed Row, Tag, and Sample data before writing anything. This can increase memory pressure during ingestion. OpenTelemetry ingestion already uses a more memory-friendly pattern: decoded samples are accumulated into an internal write context and flushed to the callback when the internal buffer becomes large.
Describe the solution you'd like
Keep using fastjson for NewRelic JSON parsing, but add a batch callback path for parsed rows.
Describe alternatives you've considered
When processing OpenTelemetry requests, VM will decode the protobuf payload streamly while simultaneously converting it into an internal row structure, which is a fully streaming process. In contrast, NewRelic payload are in json format, the fastjson used by the VM only support parsing the entire json body at once, then the subsequent conversion into internal rows can be performed streamly.
If we need to make the decoding part to be streamming, we need to swicth to other json library such as encoding/json.Decoder, but it will introduce significant changes
Additional information
No response
Source: VictoriaMetrics/VictoriaMetrics