v0.92.0 generates invalid YAML for Prometheus v3: "field retention not found in type config.plain"

Author: davidquan1Created Jun 23, 2026Updated Sep 2, 2026

Labelskind/feature

Is there an existing issue for this?

I have searched the existing issues

What happened?

Description

In Prometheus Operator v0.92.0, a new feature was introduced via PR #8547 to migrate retention options from CLI flags to the config file for Prometheus >= v3.

However, the generated YAML structure is incompatible with Prometheus v3's actual configuration schema. The operator generates a nested storage.tsdb.retention.time block, while Prometheus v3 expects a flat storage.tsdb.retention_time field (or similar flat mapping). This causes Prometheus v3 to crash during configuration loading.

Steps to Reproduce

Use Prometheus Operator v0.92.0.
Deploy a Prometheus custom resource with spec.version set to a v3 version (e.g., v3.5.4).
Set spec.retention: 42d in the Prometheus CR.
Observe the generated prometheus.env.yaml in the ConfigReloader or the Secret.

Expected Result

Prometheus starts successfully. The generated YAML should match the Prometheus v3 configuration schema, or the operator should continue using CLI flags if the YAML schema is not yet stable/correctly mapped.

Actual Result

Prometheus fails to start with the following error:

level=ERROR source=main.go:638 msg="Error loading config" file=/etc/prometheus/config_out/prometheus.env.yaml err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: yaml: unmarshal errors:\n  line 548: field retention not found in type config.plain"

Generated YAML snippet (Incorrect):

storage:
  tsdb:
    retention:
      time: 42d

Evidence from generated config:

I inspected the generated prometheus.env.yaml inside the config-reloader container. At the end of the file, the operator appends the following block which is not recognized by Prometheus v3.5.4:

# Snippet from /etc/prometheus/config_out/prometheus.env.yaml
# This block is generated by the operator and causes the unmarshal error

storage:       <-- Line 548
  tsdb:
    retention:
      time: 24h  # <--- This nested structure is rejected by Prometheus v3

Bug Code path

pkg/prometheus/promcfg.go

	if cg.WithMinimumVersion("3.11.0").IsCompatible() {
		var retentionSlice yaml.MapSlice
		retentionTime := string(RetentionTimeOrDefault(retention, retentionSize))
		if retentionTime != "" {
			retentionSlice = append(retentionSlice, yaml.MapItem{Key: "time", Value: retentionTime})
		}
		if retentionSize != "" {
			retentionSlice = append(retentionSlice, yaml.MapItem{Key: "size", Value: string(retentionSize)})
		}
		tsdbSlice = append(tsdbSlice, yaml.MapItem{Key: "retention", Value: retentionSlice})
	}

Incompatibility with Prometheus v3.5.4(LTS version)

Incompatibility with Prometheus v3.5.4: The image you're actually running is v3.5.4. According to Prometheus release records, the YAML field storage.tsdb.retention was introduced only after v3.11.0. For v3.5.4, it does not recognize the nested field of retention at all, so it reports the error field retention not found.

Prometheus v3 Expected Field: storage.tsdb.retention_time: 42d

Environment

Prometheus Operator Version: v0.92.0
Prometheus Version: v3.5.4
Kubernetes Version: 1.36

Prometheus Operator Version

bash

# k describe deploy -n tanzu-system-monitoring prometheus-operator
Name:                   prometheus-operator
Namespace:              tanzu-system-monitoring
CreationTimestamp:      Tue, 23 Jun 2026 06:16:36 +0000
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/name=prometheus-operator
                        kapp.k14s.io/app=1782195387318082006
                        kapp.k14s.io/association=v1.06220ce8d93db23873ed6dcaa6afd72a
Annotations:            deployment.kubernetes.io/revision: 1
                        kapp.k14s.io/identity: v1;tanzu-system-monitoring/apps/Deployment/prometheus-operator;apps/v1
                        kapp.k14s.io/original:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"kbld.k14s.io/images":"- origins:\n  - resolved:\n      tag: v0.92....
                        kapp.k14s.io/original-diff-md5: 766788bac67afc206eb56059b2386ba0
                        kbld.k14s.io/images:
                          - origins:
                            - resolved:
                                tag: v0.92.0_vmware.1
                                url: localhost:5000/prometheus-operator:v0.92.0_vmware.1
                            - preresolved:

Kubernetes Version

yaml

# kubectl version -o yaml
clientVersion:
  buildDate: "2025-11-21T03:14:22Z"
  compiler: gc
  gitCommit: d916b27a7c10a1bdb27b924b701eb8a460274d72
  gitTreeState: clean
  gitVersion: v1.31.11+vmware.8-fips
  goVersion: go1.23.10 X:boringcrypto
  major: "1"
  minor: "31"
  platform: linux/amd64
kustomizeVersion: v5.4.2
serverVersion:
  buildDate: "2026-06-02T04:06:26Z"
  compiler: gc
  gitCommit: 52244655f2b1d34af0af74f1874ae920f1bf78ef
  gitTreeState: clean
  gitVersion: v1.36.1+vmware.4
  goVersion: go1.26.2-X:boringcrypto
  major: "1"
  minor: "36"
  platform: linux/amd64

WARNING: version difference between client (1.31) and server (1.36) exceeds the supported minor version skew of +/-1

Kubernetes Cluster Type

Other (please comment)

How did you deploy Prometheus-Operator?

Other (please comment)

Manifests

yaml

prometheus-operator log output

bash

1. operator log:

# k logs -n tanzu-system-monitoring prometheus-prometheus-0    
time=2026-06-23T06:39:09.409Z level=ERROR source=main.go:638 msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: yaml: unmarshal errors:\n  line 548: field retention not found in type config.plain"


2. config yaml

# kubectl exec -n tanzu-system-monitoring prometheus-prometheus-0 -c config-reloader -- cat /etc/prometheus/config_out/prometheus.env.yaml | sed -n '540,555p'

    target_label: kubernetes_namespace
    action: replace
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: kubernetes_pod_name
    action: replace
storage:
  tsdb:
    retention:
      time: 24h
alerting:
  alert_relabel_configs:
  - action: labeldrop
    regex: prometheus_replica
  alertmanagers:
  - scheme: https

Anything else?

No response

Source: prometheus-operator/prometheus-operator

View original on GitHub View discussion on GitHub