Vector keeps reading from a log file that has been deleted when no space left on device

A note for the community

• Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When there is no remaining space on the device, I found that vector continues to read those deleted files. By checking the logs, I suspect it might be due to a failure in writing checkpoints. The issue persists even after the device frees up space until Vector is restarted.

2024-10-22T03:20:48.690801Z INFO source{component_kind="source" component_id=multiline-symlinks component_type=file}:file_server: vector::internal_events::file::source: Stopped watching file. file=/sysroot/var/log/zbs/netbouncer/[email protected] reached_eof="true" 2024-10-22T04:08:18.904447Z ERROR vector::internal_events::file::source: Failed writing checkpoints. error=No space left on device (os error 28) error_code="writing_checkpoints" error_type="writer_failed" stage="receiving" internal_log_rate_limit=true 2024-10-22T04:08:18.905785Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] is being suppressed to avoid flooding. 2024-10-22T04:08:28.955636Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] has been suppressed 46 times. 2024-10-22T04:08:28.955648Z ERROR vector::internal_events::file::source: Failed writing checkpoints. error=No space left on device (os error 28) error_code="writing_checkpoints" error_type="writer_failed" stage="receiving" internal_log_rate_limit=true 2024-10-22T04:08:29.032913Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] is being suppressed to avoid flooding. 2024-10-22T04:08:39.106843Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] has

Configuration

Version

0.38.0-alpine

Debug Output

Example Data

No response

Additional Context

No response

References

No response