filestream: support regex patterns in take_over.from_ids
Summary
The take_over.from_ids configuration for the filestream input currently only accepts exact input ID strings. When migrating files from multiple inputs whose IDs share a common pattern (e.g. filestream-app-v1-*), users must enumerate every ID explicitly. This is brittle and impractical when input IDs are generated dynamically.
Proposed Change
Add a new take_over.from_id_patterns configuration field that accepts a list of Go regular expressions. Each pattern is matched against the input ID segment of registry keys (i.e. the <inputID> part of filestream::<inputID>::<identityName>::<identityValue>).
filestream:
id: my-new-input
take_over:
enabled: true
from_id_patterns:
- "filestream-app-v[0-9]+-.*"
- "legacy-input-.*"
paths: [/var/log/*.log]from_ids (exact literal IDs) and from_id_patterns (regex) can be used together.
Implementation Notes
Key files:
filebeat/input/filestream/internal/input-logfile/manager.go—TakeOverConfigstruct,Unpack,SourceIdentifier/MatchesInputfilebeat/input/filestream/internal/input-logfile/store.go—identifiersToTakeOver,TakeOver,matchPreviousFilestreamIDs
Approach:
- Add
FromIDPatterns []string(config:from_id_patterns) toTakeOverConfig; compile to[]*regexp.RegexpinUnpack - Define an
InputMatcherinterface (MatchesInput(string) bool) implemented by both the existingSourceIdentifierand a newRegexInputMatcher - Change
identifiersToTakeOver []*SourceIdentifier→[]InputMatcherinsourceStore RegexInputMatcher.MatchesInputextracts the input ID segment from the registry key and tests it against the compiled regexpFromFilestream()returns true when eitherFromIDsorFromIDPatternsis non-empty
Patterns are compiled at config parse time (Unpack) so invalid regexes are rejected before the input starts.
Acceptance Criteria
-
from_id_patternsaccepts a list of Go regex strings - Invalid regex strings are rejected at config parse time with a clear error
- Patterns match against the input ID segment of registry keys only (not the full key)
-
from_ids(literal) andfrom_id_patterns(regex) work independently and together - Unit tests covering: valid patterns, invalid patterns, partial matches, combined literal+regex
- Integration test demonstrating takeover via pattern
Source: elastic/beats