Should vector be maintaining open file handles to `ignore_older` files older than the cutoff?
Author: jszwedkoCreated Aug 25, 2020Updated Sep 9, 2026
Labelssource: file
I'm not sure if this is a bug or expected behavior, but it looks like, when using the ignore_older config for the file source, Vector still maintains an open file handle to the files with a modified time before the cutoff.
From gitter: https://gitter.im/timberio-vector/community?at=5f456c28c3aa024ef99e4907 . The user was trying to limit the number of open file handles, by using the ignore_older config.
If it is expected, we should probably call it out in the docs as I was not expecting that. It seemingly limits its usefulness in avoiding resource consumption.
Vector Version
vector 0.11.0 (g8b4ff32 x86_64-unknown-linux-gnu 2020-08-25)
Vector Configuration File
data_dir = "/tmp/vector"
[sources.in]
type = "file" # required
ignore_older = 10 # optional, no default, seconds
include = ["/tmp/log/*.log"] # required
[sinks.http]
type = "console"
inputs = ["in"]
encoding.codec = "json"
Debug Output
Aug 25 16:57:23.252 INFO vector: Log level "debug" is enabled.
Aug 25 16:57:23.256 INFO vector: Loading configs. path=["/tmp/test.toml"]
Aug 25 16:57:23.276 INFO vector::topology: Running healthchecks.
Aug 25 16:57:23.276 INFO vector::topology: Starting source "in"
Aug 25 16:57:23.277 INFO vector::topology::builder: Healthcheck: Passed.
Aug 25 16:57:23.277 INFO vector::topology: Starting sink "http"
Aug 25 16:57:23.277 INFO vector: Vector has started. version="0.11.0" git_version="v0.9.0-573-g8b4ff32" released="Tue, 25 Aug 2020 20:48:03 +0000" arch="x86_64"
Aug 25 16:57:23.277 INFO source{name=in type=file}: vector::sources::file: Starting file server. include=["/tmp/log/*.log"] exclude=[]
Aug 25 16:57:23.279 INFO source{name=in type=file}:file_server: vector::internal_events::file: found new file to watch. path="/tmp/log/a.log"
Aug 25 16:57:23.280 DEBUG source{name=in type=file}:file_server: vector::internal_events::file: files checkpointed. count=0
Aug 25 16:57:24.315 DEBUG source{name=in type=file}:file_server: vector::internal_events::file: files checkpointed. count=0
Aug 25 16:57:25.340 DEBUG source{name=in type=file}:file_server: vector::internal_events::file: files checkpointed. count=0
Aug 25 16:57:27.391 DEBUG source{name=in type=file}:file_server: vector::internal_events::file: files checkpointed. count=0
^CAug 25 16:57:28.368 INFO vector: Vector has stopped.
Aug 25 16:57:28.370 INFO vector::topology: Shutting down... Waiting on: in, http. 59 seconds left
Aug 25 16:57:28.370 DEBUG source{name=in type=file}: vector::topology::builder: Finished
Aug 25 16:57:28.370 DEBUG sink{name=http type=console}: vector::topology::builder: Finished
Expected Behavior
Vector does not open the file
Actual Behavior
Vector opens the file
Additional Context
$ lsof -p 27049 | grep a.log
vector 27049 CORP\jesse 15r REG 259,3 104581 178538005 /tmp/log/a.log
Source: vectordotdev/vector