vminsert: optimize NewRelic ingestion memory usage by batching parsed rows

Author: JayiceZCreated Sep 10, 2026Updated Sep 18, 2026
Labelsenhancementvminsert

Is your feature request related to a problem? Please describe

VictoriaMetrics currently supports ingesting NewRelic infrastructure agent events via:

  • /newrelic/infra/v2/metrics/events/bulk

The current NewRelic parser path reads the full request body, parses it with fastjson, converts all events into an intermediate Rows structure, and then passes the parsed rows to the write callback.

A single NewRelic event can produce multiple VictoriaMetrics samples:

  • numeric fields become samples
  • string fields become labels attached to every sample
  • timestamp is handled as the sample timestamp

When a request contains many events or large label values, the parser accumulates all parsed Row, Tag, and Sample data before writing anything. This can increase memory pressure during ingestion. OpenTelemetry ingestion already uses a more memory-friendly pattern: decoded samples are accumulated into an internal write context and flushed to the callback when the internal buffer becomes large.

Describe the solution you'd like

Keep using fastjson for NewRelic JSON parsing, but add a batch callback path for parsed rows.

Describe alternatives you've considered

When processing OpenTelemetry requests, VM will decode the protobuf payload streamly while simultaneously converting it into an internal row structure, which is a fully streaming process. In contrast, NewRelic payload are in json format, the fastjson used by the VM only support parsing the entire json body at once, then the subsequent conversion into internal rows can be performed streamly.

If we need to make the decoding part to be streamming, we need to swicth to other json library such as encoding/json.Decoder, but it will introduce significant changes

Additional information

No response

Source: VictoriaMetrics/VictoriaMetrics