loki-mixin: log panels select on job and pod, and per_component_label is unused
Describe the bug
On main, 14 LogQL queries select streams on job or pod: the error log panels in Loki / Operational (6), Compactor Logs in Loki / Retention (1), Loki / Deletion (2, on pod) and the error panels in Loki / Bloom Build (5). A log pipeline that doesn't index those labels in Loki, for example an OpenTelemetry collector that indexes container, namespace and service_name, gets empty panels. Loki / Logs and Loki / Bloom Gateway already select on container and work.
There is no setting to change these selectors on their own: Operational's jobMatchers and jobMatcher also drive the PromQL job matchers, Deletion builds its matcher from per_instance_label, and Bloom Build hardcodes job in its JSON. #24757 adds job_prefix, which also applies to the Operational and Retention queries, but it only helps when job exists with a different prefix. config.libsonnet already defines per_component_label: 'container' (added in #17412), but nothing reads it.
To Reproduce
- Ship Loki's own logs to Loki with
cluster,namespaceandcontainerlabels, but nojoborpod. - Open Loki / Operational, Loki / Retention, Loki / Deletion or Loki / Bloom Build: the log panels show no data, while Loki / Logs does.
Expected behavior
Log panels select streams on labels the mixin configuration controls, so they work without relabeling.
Proposed fix
Build these LogQL selectors from per_namespace_label and per_component_label, for example {cluster=~"$cluster", namespace=~"$namespace", container=~"distributor"}, as Loki / Logs does. That selects the same streams wherever containers are named after components, as in the Loki Helm chart and production/ksonnet. Happy to open the PR.
Environment:
- Infrastructure: Kubernetes
- Deployment tool: jsonnet (loki-mixin), logs shipped by an OpenTelemetry collector
Source: grafana/loki