Elastic Agent on K8s - Improve logging on falling behind reading the event queue of K8s
Author: lucabellucciniCreated Sep 17, 2025Updated Sep 18, 2026
LabelsStalledTeam:Elastic-Agent-Data-Plane
Describe the enhancement:
When low on resources, Elastic Agent used to implement K8s observability use cases might log:
{"log.level":"error","@timestamp":"2025-09-15T00:31:18.803Z","message":"I0915 00:31:16.372977 43 trace.go:236] Trace[997072019]: \"DeltaFIFO Pop Process\" ID:kube-system/konnectivity-agent-b5787dd87-h4z4k,Depth:12,Reason:slow event handlers blocking the queue (15-Sep-2025 00:31:11.464) (total time: 236ms):","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-09-15T00:31:18.810Z","message":"Trace[997072019]: [236.621333ms] [236.621333ms] END","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}The message is quite obscure and reported in several posts online:
- https://discuss.elastic.co/t/metricbeat-crashing-reason-slow-event-handlers-blocking-the-queue/306648/3
- https://discuss.elastic.co/t/elastic-agent-not-shipping-all-logs-from-kubernetes-cluster-errors-in-logs/325620
Describe a specific use case for the enhancement or feature:
We should attempt to improve the experience of the user and make those errors actionable, such as:
- If possible, return the state of the unit as
degradeduntil it recovers. - Suggest to increase the resources of the host / container in the logs
FYI @ebeahan
Source: elastic/beats