#53096·beats

filestream: support regex patterns in take_over.from_ids

Author: blakerouseCreated Sep 8, 2026Updated Sep 17, 2026
LabelsenhancementTeam:Elastic-Agent-Data-Plane

Summary

The take_over.from_ids configuration for the filestream input currently only accepts exact input ID strings. When migrating files from multiple inputs whose IDs share a common pattern (e.g. filestream-app-v1-*), users must enumerate every ID explicitly. This is brittle and impractical when input IDs are generated dynamically.

Proposed Change

Add a new take_over.from_id_patterns configuration field that accepts a list of Go regular expressions. Each pattern is matched against the input ID segment of registry keys (i.e. the <inputID> part of filestream::<inputID>::<identityName>::<identityValue>).

yaml
filestream:
  id: my-new-input
  take_over:
    enabled: true
    from_id_patterns:
      - "filestream-app-v[0-9]+-.*"
      - "legacy-input-.*"
  paths: [/var/log/*.log]

from_ids (exact literal IDs) and from_id_patterns (regex) can be used together.

Implementation Notes

Key files:

  • filebeat/input/filestream/internal/input-logfile/manager.goTakeOverConfig struct, Unpack, SourceIdentifier/MatchesInput
  • filebeat/input/filestream/internal/input-logfile/store.goidentifiersToTakeOver, TakeOver, matchPreviousFilestreamIDs

Approach:

  1. Add FromIDPatterns []string (config: from_id_patterns) to TakeOverConfig; compile to []*regexp.Regexp in Unpack
  2. Define an InputMatcher interface (MatchesInput(string) bool) implemented by both the existing SourceIdentifier and a new RegexInputMatcher
  3. Change identifiersToTakeOver []*SourceIdentifier[]InputMatcher in sourceStore
  4. RegexInputMatcher.MatchesInput extracts the input ID segment from the registry key and tests it against the compiled regexp
  5. FromFilestream() returns true when either FromIDs or FromIDPatterns is non-empty

Patterns are compiled at config parse time (Unpack) so invalid regexes are rejected before the input starts.

Acceptance Criteria

  • from_id_patterns accepts a list of Go regex strings
  • Invalid regex strings are rejected at config parse time with a clear error
  • Patterns match against the input ID segment of registry keys only (not the full key)
  • from_ids (literal) and from_id_patterns (regex) work independently and together
  • Unit tests covering: valid patterns, invalid patterns, partial matches, combined literal+regex
  • Integration test demonstrating takeover via pattern