Data Loss After Log Rolling in Vector File Source

Author: suha-companyCreated May 20, 2025Updated Sep 9, 2026

Labelssource: file

A note for the community

Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When collecting access.log files with Vector, logs are rolled every 60 minutes or when exceeding 100 MB. However, occasionally after rolling, the newly created access.log is not detected, resulting in missing log entries.

Configuration

Log Rolling Schedule

access.log is rolled to access_YYYYMMDD_HHMM.log every 60 minutes or when it exceeds 100 MB.
Rolling is performed via rename, so the original file’s inode remains the same both access.log and access_YYYYMMDD_HHMM.log share the identical inode.

### Source (`cache_logs_access.yaml`)


type: file
ignore_checkpoints: false
fingerprint:
  strategy: "device_and_inode"
include:
  - /var/log/example/*/access.log


-------------------------------------------------------------------------------------------

### Transform (`cache_logs_access.parse.yaml`)


type: remap
inputs:
  - cache_logs_access
drop_on_abort: true
drop_on_error: true
file: /etc/vector/remap/cache_logs_access_parse.vrl

-------------------------------------------------------------------------------------------


### remap ( cache_logs_access.parse.vrl)


parsed, err = parse_grok(
  .message,
  "(?<datetimestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{SPACE}%{TIME})%{SPACE}%{GREEDYDATA:rest}"
)
if err != null { abort } else { . = parsed }

datetime_utc = format_timestamp!(parse_timestamp!(.datetimestamp, format: "%F %X"), format: "%F %X")
.message = join!([datetime_utc, .rest], separator: " ")


-------------------------------------------------------------------------------------------

### Sink (`opensearch_front.yaml`)


type: http
buffer:
  max_events: 1000
  when_full: block
tls:
  verify_certificate: false
inputs:
  - cache_logs_access_parse
encoding:
  codec: "raw_message"
uri: https://log.example.com:9998/ingest/edge/example

Version

vector 0.45.0

Debug Output

Example Data

✅ Expected (Normal) Collection Flow


Found new file to watch. file=/var/log/example/site.com/access.log
Stopped watching file. file=/var/log/example/site.com/access.log reached_eof="true"
Files checkpointed. count=418 duration_ms=1
HTTP POST https://log.example.com:9998/ingest/edge/example … status=200 OK

After log rotation, Vector immediately detect the new access.log for continued collection.

❌ Observed (Missing) Collection Flow

Resuming to watch file. file=/var/log/example/site.com/access.log file_position=104970000
Stopped watching file. file=/var/log/example/site.com/access.log reached_eof="true"
// No further “Found new file to watch” event for the new access.log → logs missing

After log rotation, Vector resumes watching the old access.log by referring to its checkpoint (inode and offset at 104,970,000) instead of picking up the newly created file.
it occasionally fails to detect the new access.log and misses subsequent log entries.

Additional Context

access_log list

References

No response

Source: vectordotdev/vector

View original on GitHub View discussion on GitHub