Node Exporter Metrics Timestamp not set properly in OpenTelemetry push mode

Bug Report

Describe the bug We are running fluent-bit is a combined log and metrics agent that pushes both to an opentelemetry collector service, from where it then gets split up to its respective target. Metrics in this case go to Prometheus.

What we observed was then that, after a certain amount of time - namely pretty much one hour - Prometheus stopped accepting metrics for being to old. Despite Fluent-Bit constantly sending and OtelCollector constantly forwarding and all systems involved being far from under load.

Now our investigation indicates that the reason is, that Fluent-Bit does not properly update the metrics timestamps of some metrics it collects, specifically as it only updates the values on value change. This leads to some metrics that don't change often to cause blockages, as Prometheus will drop all metrics wholesale if one of them has timestamps it considers outdated.

To Reproduce

• Steps to reproduce the problem: Set up Fluent-Bit with node exporter, specifically include metrics that do not update often, like Systemd Set up OpenTelemetry-Collector Set up Prometheus

Configure Fluent-Bit to forward metrics to the OtelCollectors metrics endpoint. Configure OtelCollector to forward metrics to Prometheus.

Wait.

Expected behavior Fluent-Bit should have a configurable flag for the node exporter settings to update metrics timestamps of "on Value change" metrics with every scrape (or rather, collect a new sample, with every scrape). Given that this surely will cause issues with existing setups this should be off by default, but it should be noted in documentation.

Your Environment

• Version used: fluent-bit 5.1.1, otel-collector 0.160, prometheus 3.11.3
• Configuration:

    - name: node_exporter_metrics
      tag: node_metrics
      scrape_interval: 30
      metrics: "cpu,meminfo,diskstats,filesystem,uname,stat,time,timex,loadavg,vmstat,netdev,netstat,sockstat,filefd,systemd"
      processors:
        metrics:
          - name: labels
            upsert: host ${HOSTNAME}

• Environment name and version (e.g. Kubernetes? What version?): LXC
• Server type and version:
• Operating System and version: Debian 12
• Filters and plugins:
• filters: modify
• input plugins: systemd, node_exporter_metrics
• output plugins: opentelemetry

Additional context We are currently trying to consolidate monitoring efforts and route everything through OpenTelemetry Collector as a single ingestion point