Baike.dev
Log in
Back to tool/Back to issues
#26198·vector

disk_v2 permanently drops and acknowledges all events in an oversized EventArray

Author: dikuchanCreated Aug 25, 2026Updated Sep 9, 2026
Labelsdomain: buffers

A note for the community

  • Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

#25795 prevents an oversized disk_v2 record from tearing down the Vector topology by dropping the record and continuing.

However, as I understand, a disk buffer record isn't necessarily a single logical event. In the normal topology path it's an EventArray, which may contain many events produced by a transform.

For example, a VRL remap assigning an array to the root:

. = parse_json!(string!(.message))

turns one input event into many output events. OutputBuffer then coalesces all adjacent events of the same type into one EventArray, and disk_v2 serializes that entire array as one record.

As a result, #25795 can permanently discard a large number of valid, individually encodable events because Vector grouped them into one internal record. This is permanent data loss caused by an internal batching boundary that's invisible to the user.

I agree that retrying a single event which cannot fit into an empty disk record would create a broken loop. However, an EventArray isn't an atomic event: when its individual events are encodable, the aggregate record can succeed after being split. The drop policy could be applied after the record has been reduced to one independently unencodable event.

Configuration

data_dir: ./vector-data

sources:
  input:
    type: file
    include:
      - ./input.ndjson
    read_from: beginning
    ignore_checkpoints: true
    max_line_bytes: 268435456
    file_key: ""
    host_key: ""

transforms:
  expand:
    type: remap
    inputs:
      - input
    source: |
      . = parse_json!(string!(.message))

sinks:
  output:
    type: file
    inputs:
      - expand
    path: ./output.ndjson
    encoding:
      codec: json
    buffer:
      type: disk
      max_size: 536870912
      when_full: block

Version

Vector v0.57.0 and upstream commit a5f409e099a1ebc1fe0b4ae4eac39a19c4777771 from #25795 cherry-picked on top

Debug Output

2026-08-25T14:23:38.401602Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_limit_secs=10
2026-08-25T14:23:38.401721Z  INFO vector::app: Log level is enabled. level="trace"
2026-08-25T14:23:38.401813Z DEBUG vector::app: Building runtime. worker_threads=14 chunk_size_events=1000
2026-08-25T14:23:38.401954Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE
2026-08-25T14:23:38.403312Z  INFO vector::app: Loading configs. paths=["vector.yaml"]
2026-08-25T14:23:38.406361Z DEBUG vector::config::loading::secret: No secret placeholder found, skipping secret resolution.
2026-08-25T14:23:38.413809Z DEBUG vector::topology::builder: Building new source. component_id=input
2026-08-25T14:23:38.432131Z DEBUG vector::topology::builder: Building new transform. component_id=expand
2026-08-25T14:23:38.433169Z DEBUG vector::topology::builder: Building new sink. component_id=output
2026-08-25T14:23:38.434118Z DEBUG sink{component_kind="sink" component_id=output component_type=file}: vector_buffers::variants::disk_v2::ledger: Found existing data file. data_file="buffer-data-0.dat" file_size=0 total_buffer_size=0
2026-08-25T14:23:38.434166Z TRACE sink{component_kind="sink" component_id=output component_type=file}: vector_buffers::variants::disk_v2::ledger: Updated buffer size. previous_buffer_size=0 new_buffer_size=0
2026-08-25T14:23:38.434239Z DEBUG sink{component_kind="sink" component_id=output component_type=file}:validate_last_write: vector_buffers::variants::disk_v2::writer: Validating last written record in current data file. current_writer_data_file="./vector-data/buffer/v2/output/buffer-data-0.dat"
2026-08-25T14:23:38.434418Z DEBUG sink{component_kind="sink" component_id=output component_type=file}:validate_last_write:ensure_ready_for_write: vector_buffers::variants::disk_v2::writer: Opened data file for writing. data_file_path="./vector-data/buffer/v2/output/buffer-data-0.dat" existing_file_size=0
2026-08-25T14:23:38.442862Z DEBUG sink{component_kind="sink" component_id=output component_type=file}: vector_buffers::variants::disk_v2::reader: Seeking to last acknowledged record for reader. last_acknowledged_record_id=0
2026-08-25T14:23:38.442902Z DEBUG sink{component_kind="sink" component_id=output component_type=file}: vector_buffers::variants::disk_v2::reader: Synchronized with ledger. Reader ready. last_record_id_read=0
2026-08-25T14:23:38.643433Z  INFO vector::topology::running: Running healthchecks.
2026-08-25T14:23:38.643544Z DEBUG vector::topology::running: Connecting changed/added component(s).
2026-08-25T14:23:38.643594Z DEBUG vector::topology::running: Configuring outputs for source. component_id=input
2026-08-25T14:23:38.643666Z  INFO vector::topology::builder: Healthcheck passed.
2026-08-25T14:23:38.644779Z DEBUG vector::topology::running: Configuring output for component. component_id=input output_id=None
2026-08-25T14:23:38.644823Z DEBUG vector::topology::running: Configuring outputs for transform. component_id=expand
2026-08-25T14:23:38.644847Z DEBUG vector::topology::running: Configuring output for component. component_id=expand output_id=None
2026-08-25T14:23:38.644871Z DEBUG vector::topology::running: Connecting inputs for transform. component_id=expand
2026-08-25T14:23:38.645450Z DEBUG vector::topology::running: Adding component input to fanout. component_id=expand fanout_id=input
2026-08-25T14:23:38.645513Z DEBUG vector::topology::running: Connecting inputs for sink. component_id=output
2026-08-25T14:23:38.645538Z DEBUG vector::topology::running: Adding component input to fanout. component_id=output fanout_id=expand
2026-08-25T14:23:38.645658Z DEBUG vector::topology::running: Spawning new source. component_id=input
2026-08-25T14:23:38.645797Z DEBUG vector::topology::running: Spawning new transform. component_id=expand
2026-08-25T14:23:38.645819Z DEBUG source{component_kind="source" component_id=input component_type=file}: vector::topology::builder: Source pump supervisor starting.
2026-08-25T14:23:38.645857Z TRACE vector::topology::running: Spawning new sink. component_id=output
2026-08-25T14:23:38.645944Z DEBUG source{component_kind="source" component_id=input component_type=file}: vector::topology::builder: Source pump starting.
2026-08-25T14:23:38.645903Z DEBUG source{component_kind="source" component_id=input component_type=file}: vector::topology::builder: Source starting.
2026-08-25T14:23:38.646138Z  INFO source{component_kind="source" component_id=input component_type=file}: vector::sources::file: Starting file server. include=["./input.ndjson"] exclude=[]
2026-08-25T14:23:38.646136Z DEBUG transform{component_kind="transform" component_id=expand component_type=remap}: vector::topology::builder: Synchronous transform starting.
2026-08-25T14:23:38.646151Z DEBUG sink{component_kind="sink" component_id=output component_type=file}: vector::topology::builder: Sink starting.
2026-08-25T14:23:38.646292Z  INFO vector: Vector has started. debug="true" version="0.57.0" arch="aarch64" revision=""
2026-08-25T14:23:38.646333Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2026-08-25T14:23:38.646570Z TRACE source{component_kind="source" component_id=input component_type=file}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "expand" })
2026-08-25T14:23:38.646691Z DEBUG sink{component_kind="sink" component_id=output component_type=file}: vector_buffers::variants::disk_v2::reader: Opened data file for reading. data_file_path="./vector-data/buffer/v2/output/buffer-data-0.dat"
2026-08-25T14:23:38.647299Z DEBUG vector::utilization: component_id=output utilization=0.9953 internal_log_rate_limit=false
2026-08-25T14:23:38.647296Z TRACE vector: Beep.
2026-08-25T14:23:38.647320Z DEBUG vector::utilization: component_id=expand utilization=0.9947 internal_log_rate_limit=false
2026-08-25T14:23:39.649494Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-08-25T14:23:40.043670Z  INFO source{component_kind="source" component_id=input component_type=file}:file_server: vector::internal_events::file::source: Found new file to watch. file=input.ndjson
2026-08-25T14:23:41.054146Z DEBUG source{component_kind="source" component_id=input component_type=file}:file_server: vector::internal_events::file::source: Files checkpointed. count=0 duration_ms=6
2026-08-25T14:23:41.401097Z TRACE source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: Continue watching file. path="input.ndjson"
2026-08-25T14:23:41.783417Z TRACE source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: Read bytes. path="input.ndjson" bytes=139838891
2026-08-25T14:23:41.783584Z TRACE source{component_kind="source" component_id=input component_type=file}: vector::internal_events::file::source: Bytes received. byte_size=139838891 protocol="file" file=input.ndjson
2026-08-25T14:23:41.783688Z TRACE source{component_kind="source" component_id=input component_type=file}: vector::internal_events::file::source: Events received. count=1 byte_size=139838968 file=input.ndjson
2026-08-25T14:23:41.783797Z TRACE source{component_kind="source" component_id=input component_type=file}: vector_buffers::topology::channel::limited_queue: Sent item.
2026-08-25T14:23:41.783818Z TRACE source{component_kind="source" component_id=input component_type=file}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=139838970 output=_default
2026-08-25T14:23:41.783885Z DEBUG source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: event_throughput=1.000/sec bytes_throughput=139.839m/sec ratios={"discovery": 0.78015023, "other": 7.041566e-5, "reading": 0.21976103, "sending": 1.1185639e-5, "sleeping": 7.113535e-6}
2026-08-25T14:23:41.783900Z TRACE source{component_kind="source" component_id=input component_type=file}: vector_buffers::topology::channel::limited_queue: Internal log [Sent item.] is being suppressed to avoid flooding.
2026-08-25T14:23:41.783931Z TRACE source{component_kind="source" component_id=input component_type=file}: vector_core::fanout: Sent item to fanout.
2026-08-25T14:23:41.783982Z TRACE transform{component_kind="transform" component_id=expand component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=139838970
2026-08-25T14:23:42.062681Z DEBUG source{component_kind="source" component_id=input component_type=file}:file_server: vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2026-08-25T14:23:42.903075Z TRACE transform{component_kind="transform" component_id=expand component_type=remap}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "output" })
2026-08-25T14:23:43.187877Z TRACE source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: Internal log [Continue watching file.] is being suppressed to avoid flooding.
2026-08-25T14:23:43.432206Z ERROR transform{component_kind="transform" component_id=expand component_type=remap}:write_record_outcome:write_record:try_write_record_inner: vector_buffers::variants::disk_v2::writer: Record cannot be written to the disk buffer; dropping it. event_count=150000 encoded_len=134217728 max_record_size=134217728 error=failed to encode record: BufferTooSmall
2026-08-25T14:23:43.439348Z TRACE transform{component_kind="transform" component_id=expand component_type=remap}:flush: vector_buffers::variants::disk_v2::ledger: Updated buffer size. previous_buffer_size=0 new_buffer_size=0
2026-08-25T14:23:43.439396Z TRACE transform{component_kind="transform" component_id=expand component_type=remap}: vector_core::fanout: Sent item to fanout.
2026-08-25T14:23:43.439417Z TRACE transform{component_kind="transform" component_id=expand component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=150000 byte_size=139838891 output=_default
2026-08-25T14:23:43.648597Z DEBUG vector::utilization: component_id=output utilization=0.0995 internal_log_rate_limit=false
2026-08-25T14:23:43.648622Z DEBUG vector::utilization: component_id=expand utilization=0.2807 internal_log_rate_limit=false
2026-08-25T14:23:44.445921Z ERROR sink{component_kind="sink" component_id=output component_type=file}:sink{buffer_type="disk"}: vector_buffers::internal_events: Events dropped. count=150000 byte_size=134217728 intentional=false reason=unprocessable_events buffer_id=output stage=0
2026-08-25T14:23:48.647742Z DEBUG vector::utilization: component_id=output utilization=0.01 internal_log_rate_limit=false
2026-08-25T14:23:48.647754Z TRACE vector: Internal log [Beep.] has been suppressed 9 times.
2026-08-25T14:23:48.647772Z DEBUG vector::utilization: component_id=expand utilization=0.0281 internal_log_rate_limit=false
2026-08-25T14:23:48.647781Z TRACE vector: Beep.
2026-08-25T14:23:49.648718Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-08-25T14:23:51.082207Z DEBUG source{component_kind="source" component_id=input component_type=file}:file_server: vector::internal_events::file::source: Internal log [Files checkpointed.] has been suppressed 9 times.
2026-08-25T14:23:51.082242Z DEBUG source{component_kind="source" component_id=input component_type=file}:file_server: vector::internal_events::file::source: Files checkpointed. count=1 duration_ms=0
2026-08-25T14:23:51.593852Z TRACE source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: Internal log [Continue watching file.] has been suppressed 6 times.
2026-08-25T14:23:51.593879Z TRACE source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: Continue watching file. path="input.ndjson"
2026-08-25T14:23:52.084343Z DEBUG source{component_kind="source" component_id=input component_type=file}:file_server: vector::internal_events::file::source: Internal log [Files checkpointed.] is being suppressed to avoid flooding.
2026-08-25T14:23:53.139973Z TRACE source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: Internal log [Continue watching file.] is being suppressed to avoid flooding.
2026-08-25T14:23:53.398459Z DEBUG source{component_kind="source" component_id=input component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 0.87769675, "other": 9.254724e-5, "reading": 1.8577874e-5, "sending": 9.801494e-6, "sleeping": 0.12218227}
2026-08-25T14:23:53.648391Z DEBUG vector::utilization: component_id=output utilization=0.001 internal_log_rate_limit=false
2026-08-25T14:23:53.648421Z DEBUG vector::utilization: component_id=expand utilization=0.0028 internal_log_rate_limit=false

Example Data

Input data is generated with the following Python script:

import json

path = "./input.ndjson"
event_count = 150_000
payload = "x" * 900

with open(path, "w", encoding="utf-8") as stream:
    stream.write("[")
    for sequence in range(event_count):
        if sequence:
            stream.write(",")
        json.dump(
            {"sequence": sequence, "payload": payload},
            stream,
            separators=(",", ":"),
        )
    stream.write("]\n")

This input contains 150,000 events. Each event is less than 1 KiB and can be encoded independently. Only their combined EventArray exceeds the 128 MiB disk record limit.

Additional Context

No response

References

  • #18346
  • #25795

Source: vectordotdev/vector

View original on GitHubView discussion on GitHub