[Bug]: Elasticsearch/OpenSearch metrics return incomplete call and error rate series when ratePer > 10m
What happened?
GET /api/metrics/calls and /api/metrics/errors accept a ratePer parameter. With the Prometheus metrics backend it sets the PromQL rate window, so any value works. With the Elasticsearch or OpenSearch metrics backend, a ratePer above 10 minutes leaves the first part of the returned series without values.
Both backends use internal/storage/metricstore/elasticsearch. There, calculateTimeRange (reader.go) always starts the Elasticsearch query 10 minutes before the requested start, whatever ratePer is:
extendedStartTime := startTime.Add(-10 * time.Minute)calcCallRate (processor.go) needs ceil(ratePer / step) preceding buckets for each point and returns NaN until it has them. With ratePer=30m or 1h, only 10 minutes of preceding buckets are fetched, so the earliest points in the requested range cannot be computed.
Error rates are computed from call rates (GetErrorRates → CalculateErrorRates), so they are affected the same way.
The Monitor tab in the Jaeger UI always sends ratePer=10m, so it is not affected today. Direct API callers are.
Steps to reproduce
- Start the SPM stack with the Elasticsearch metrics backend from
docker-compose/monitor:JAEGER_VERSION=2.21.0 docker compose -f docker-compose-elasticsearch.yml up -d - Let microsim generate traces for at least 30 minutes, so Elasticsearch holds data from before the queried window.
- Request call rates for the last 10 minutes with a 15-minute rate window:
curl -s "http://localhost:16686/api/metrics/calls?service=frontend&lookback=600000&step=60000&ratePer=900000" \ | jq -r '.metrics[0].metricPoints[] | "\(.timestamp) \(.gaugeValue.doubleValue)"' - Run the same request with a 10-minute rate window (
ratePer=600000) for comparison.
ratePer=900000 (15m): the earliest points in the window have no value.
ratePer=600000 (10m), same window: every point has a value.
Expected behavior
The query should reach back at least ratePer before the requested start, so every returned point in the requested range has a value, matching the Prometheus backend.
Storage backend
Elasticsearch, OpenSearch (metrics store)
Jaeger backend version
v2.8.0 through v2.21.0 (latest) and main. The hardcoded window was introduced in #7229.
Source: jaegertracing/jaeger