Persistent memory growth with Tail filesystem storage on Fluent Bit 5.1.2, significantly mitigated by `storage.type: memory`

Bug Report

Describe the bug We are observing persistent memory growth in Fluent Bit running as a DaemonSet on a high-volume OpenShift cluster.

The issue was initially observed on Fluent Bit 5.0.6 (https://github.com/fluent/fluent-bit/issues/11846) and has also been reproduced with Fluent Bit 5.1.2 using the official debug image.

The same general Fluent Bit configuration is deployed on another cluster with significantly lower log volume, where we do not observe the same level of memory growth.

The most relevant finding from our investigation is that changing Tail input chunk storage from:

storage.type: filesystem

to:

storage.type: memory

significantly changes the long-term memory behavior.

With filesystem-backed Tail storage, container memory continuously increases over multiple days.

With memory-backed Tail storage, the initial memory usage rises during warm-up, but the long-term curve becomes significantly flatter and the growth of cgroup reclaimable slab memory is greatly reduced.

We performed several A/B tests to isolate the behavior, described below.

To Reproduce

• Rubular link if applicable: N/A
• Example log message if applicable: N/A

We are unfortunately not able to provide our real application logs because they contain confidential/internal data.

We can provide the Fluent Bit configuration structure, Prometheus metrics, process memory statistics, cgroup memory statistics, and A/B test results.

We would appreciate it if the maintainers could try to reproduce the behavior using synthetic/generated Kubernetes container logs, ideally with:

• sustained log volume;
• periodic bursty log traffic;
• multiple Tail inputs;
• multiple Loki outputs;
• filesystem-backed Tail storage;
• normal log rotation behavior.

The issue is much easier to observe on our higher-throughput cluster than on an otherwise similarly configured lower-volume cluster.

Environment topology

• OpenShift 4.20
• 3 worker nodes
• Fluent Bit deployed as a DaemonSet
• One Fluent Bit pod per worker node
• Fluent Bit 5.1.2
• Latest investigation performed with Fluent Bit 5.1.2 debug image
• ConfigMap reloader: v0.14.0
• Fluent Bit metrics exposed through the built-in HTTP server
• Loki used as output
• Mutual TLS enabled for Loki outputs
• cgroup v2

Simplified configuration

Our real configuration is generated from multiple tenant configuration files and included into one main Fluent Bit configuration.

All internal hostnames, tenant IDs, application names, and other identifying information have been removed below.

Main service configuration:

service:
  daemon: off
  flush: 1
  log_level: info

  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  health_check: on

  storage.path: /var/log/fluent-bit-storage
  storage.sync: normal
  storage.checksum: on
  storage.max_chunks_up: 128
  storage.delete_irrecoverable_chunks: on

  scheduler.base: 5
  scheduler.cap: 600

We have multiple Tail inputs.

Representative Tail input:

pipeline:
  inputs:
    - name: tail
      tag: kube.application.*
      path: /var/log/containers/*_namespace_container-*.log

      refresh_interval: 2
      parser: cri

      skip_empty_lines: on
      read_from_head: false

      db: /var/log/fluent-bit-storage/application.db

      storage.type: filesystem

      threaded: true

Each Tail input now has its own SQLite DB file.

Some applications use multiline parsing:

multiline_parsers:
  - name: application_multiline
    type: regex
    flush_timeout: 1000
    rules:
      - state: start_state
        regex: /.../
        next_state: cont
      - state: cont
        regex: /.../
        next_state: cont

Some pipelines also use filesystem-backed emitters:

filters:
  - name: multiline
    match: kube.application.*
    multiline.key_content: message
    multiline.parser: application_multiline
    emitter_storage.type: filesystem

or equivalent rewrite/emitter processing.

Kubernetes enrichment is enabled:

filters:
  - name: kubernetes
    match: kube.application.*
    kube_tag_prefix: kube.application.var.log.containers.
    merge_log: off
    annotations: off
    labels: on
    buffer_size: 64k

For tenants that do not require Kubernetes labels, we changed:

labels: off

We currently have approximately 20+ Loki output instances per Fluent Bit pod because logs are routed to multiple tenants.

Representative Loki output:

outputs:
  - name: loki
    match: kube.application.*

    host: <internal-loki-endpoint>
    port: 443

    line_format: json

    tls: on
    tls.verify: on
    tls.debug: 0

    tls.ca_file: /fluent-bit/etc/certs/ca.crt
    tls.crt_file: /fluent-bit/etc/certs/tls.crt
    tls.key_file: /fluent-bit/etc/certs/tls.key

    storage.total_limit_size: 10G

    net.connect_timeout: 310
    net.keepalive_idle_timeout: 300
    net.dns.resolver: ASYNC

    retry_limit: no_limits

    workers: 1

Originally all Loki outputs used:

workers: 2

The A/B testing below explains why we changed this to workers: 1.

A/B test timeline

Baseline

Original relevant configuration:

Tail storage.type: filesystem
Loki workers: 2

Observed behavior:

• Fluent Bit memory continuously increased over time.
• The increase was much more visible on the higher-throughput cluster.
• The lower-volume cluster with similar configuration did not show the same degree of growth.

Test 1 - configuration cleanup

We first applied the following changes:

1. Disabled TLS debug logging:

   复制

   tls.debug: 0
   

2. Changed Tail database configuration so that every Tail input has its own SQLite DB file.

3. Disabled Kubernetes labels for tenants that do not require them:

   复制

   labels: off

These changes did not materially change the long-term memory growth behavior.

Test 2 - reduced Loki output workers

We changed every Loki output from:

workers: 2

to:

workers: 1

This had a very clear effect on baseline memory usage.

Before:

workers=2
Threads per Fluent Bit process: ~55

After:

workers=1
Threads per Fluent Bit process: ~34

The number of threads dropped by exactly 21, which matches the number of Loki output instances.

The cold-start anonymous memory usage also dropped by approximately 100-130 MiB per pod.

However, the long-term memory growth was still present.

With:

workers=1
Tail storage.type=filesystem

memory still increased over multiple days.

We also observed increasing reclaimable slab memory in the Fluent Bit container cgroup.

After roughly three days, representative slab_reclaimable values were approximately:

node 1: ~64 MiB
node 2: ~16 MiB
node 3: ~37 MiB

At the same time, Fluent Bit process anonymous memory remained significant.

This suggested that the container memory growth was not explained only by process RSS / anonymous memory.

Test 3 - changed only Tail chunk storage from filesystem to memory

We kept:

workers: 1

and changed only Tail input chunk storage from:

storage.type: filesystem

to:

storage.type: memory

We also added explicit Tail memory limits.

For normal Tail inputs:

mem_buf_limit: 16M

For the higher-throughput Airflow Tail input:

mem_buf_limit: 32M

Important: we intentionally left the existing emitter storage unchanged:

emitter_storage.type: filesystem

This was done to isolate the effect of Tail input chunk storage.

SQLite Tail databases also remained on disk.

Results after switching Tail storage to memory

The memory behavior changed significantly.

Instead of continuously climbing, the memory curve now shows:

1. initial warm-up;
2. some workload-related spikes;
3. then a much flatter long-running memory profile.

After approximately four days, representative cgroup values were:

Pod / node 1:
memory.current       ~325 MiB
anon                 ~234 MiB
slab_reclaimable     ~12 MiB

Pod / node 2:
memory.current       ~314 MiB
anon                 ~248 MiB
slab_reclaimable     ~4 MiB

Pod / node 3:
memory.current       ~308 MiB
anon                 ~227 MiB
slab_reclaimable     ~7 MiB

This is significantly different from the filesystem-backed Tail test, where reclaimable slab continued to increase.

The Kubernetes working-set memory curve also changed from a persistent upward trend to something much closer to a plateau around approximately:

280-310 MiB

depending on the node and workload.

There are still workload-related spikes.

One pod temporarily reached close to the 512 MiB cgroup memory limit, but memory returned to its previous range afterward.

For that pod:

memory.events:
max 41
oom 0
oom_kill 0

No OOM occurred.

Additional observations

During the tests:

Output retries: 0
Output errors: 0

We did not observe a sustained storage backlog.

Tail storage chunks occasionally increase during log bursts, but return to zero.

Busy chunk metrics remain at or close to zero during normal operation.

The Fluent Bit process also shows stable thread and file descriptor counts.

Typical values after setting workers: 1:

Threads: 34
FD count: ~640-670

No deleted-but-open file descriptors were observed:

Deleted FDs: None

The main Fluent Bit process memory is still mostly anonymous memory.

Representative process values:

VmRSS:          ~250 MiB
RssAnon:        ~230 MiB
Anonymous:      ~230 MiB
AnonHugePages:  ~150 MiB

The binary is built with jemalloc support.

cgroup memory comparison

With Tail filesystem storage, after several days:

node 1:
slab_reclaimable ~64 MiB

node 2:
slab_reclaimable ~16 MiB

node 3:
slab_reclaimable ~37 MiB

With Tail memory storage, after approximately four days:

node 1:
slab_reclaimable ~12 MiB

node 2:
slab_reclaimable ~4 MiB

node 3:
slab_reclaimable ~7 MiB

This is one of the strongest differences we observed during the A/B testing.

Relevant comparison with existing issues

This behavior appears related to, but not necessarily identical to:

• #11846 - Linear memory increase after update to 5.0.6
• #11958 - Persistent linear memory increase even after upgrading to 5.0.5

In #11846, maintainers requested that users still seeing memory growth open a new issue with a clean reproduction.

In #11958, another user reported that changing only the rewrite-tag emitter to memory did not help, while later changing Tail chunk storage from filesystem to memory appeared to stop the linear memory growth.

Our observations are similar.

However, our cgroup memory breakdown is not identical to #11958.

In our environment:

• Fluent Bit still has a significant anonymous-memory component.
• Tail filesystem storage appears to add an additional reclaimable-slab growth component.
• Reducing Loki workers lowers the baseline userspace memory substantially, but does not fully stop long-term growth.
• Changing Tail storage to memory significantly reduces the slab growth and changes the memory curve from persistent growth to a much flatter profile.

Expected behavior Fluent Bit Tail inputs using:

storage.type: filesystem

should not cause container memory usage to continue increasing over multiple days when:

• outputs are healthy;
• there are no output errors;
• there are no output retries;
• there is no sustained chunk backlog;
• filesystem chunks are being flushed normally.

We expect Fluent Bit memory usage to reach a stable steady state after normal allocator/cache warm-up.

Switching Tail chunk storage from filesystem to memory should not be required to achieve stable long-running memory behavior.

Screenshots

Your Environment

• Version used: Fluent Bit 5.1.2 and ConfigMap reloader v0.14.0. The latest A/B investigation was performed using the Fluent Bit 5.1.2 debug image. The issue was originally observed with Fluent Bit 5.1.1 as well.
• Configuration: See the simplified configuration above.
• Environment name and version (e.g. Kubernetes? What version?): Red Hat OpenShift 4.20 3 worker nodes Fluent Bit deployed as a DaemonSet
• Server type and version: Linux worker nodes managed by OpenShift
• Operating System and version: OpenShift-managed Linux nodes.
• Filters and plugins: Main components used: tail CRI parser custom JSON / regex parsers multiline parser/filter grep modify kubernetes rewrite/emitter processing loki output filesystem storage SQLite Tail DB HTTP metrics server

Additional context We would prefer to keep filesystem-backed buffering in production.

Our original reason for using:

storage.type: filesystem
retry_limit: no_limits

is to preserve queued log data during a temporary Loki outage and send the backlog once Loki becomes available again.

Changing Tail storage to memory improves the memory-growth behavior, but it changes the durability characteristics:

• buffered chunks are no longer persisted;
• a Fluent Bit restart can lose in-memory buffered data;
• prolonged downstream outages can cause Tail inputs to pause when mem_buf_limit is reached.

Therefore, storage.type: memory is useful for us as a diagnostic workaround, but we would prefer to identify and fix the underlying filesystem-storage behavior.

We cannot provide our real application log files due to confidentiality restrictions.