Invalid probes when using distroless images with `listenLocal` set to true

Author: mahendrapaipuriCreated Jun 3, 2026Updated Sep 6, 2026

Labelskind/feature

Is there an existing issue for this?

I have searched the existing issues

What happened?

Description

The operator uses sh and curl for setting up probes when listenLocal=true. As distroless images are used by default in kube-prometheus-stack chart, these probes fail when using with listenLocal: true due to lack of sh and curl in the images.

Here is the Prometheus container description:

yaml

Containers:
  prometheus:
    Container ID:  containerd://083d656184d794d4a57a1c0b337a7627cc08e0125999ba608901264cef2b0045
    Image:         quay.io/prometheus/prometheus:v3.11.3-distroless
    Image ID:      quay.io/prometheus/prometheus@sha256:cff72a3f49918f41c4b5c8a6174dd8433036bebf7878120da538b3720ba3fa0d
    Port:          <none>
    Host Port:     <none>
    Args:
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --web.enable-lifecycle
      --web.external-url=https://kube-ceems-kube-prometheus-prometheus.ceems:10090
      --web.route-prefix=/
      --web.listen-address=127.0.0.1:9090
      --storage.tsdb.retention.time=10d
      --storage.tsdb.path=/prometheus
      --storage.tsdb.wal-compression
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    State:          Running
      Started:      Sun, 31 May 2026 18:01:55 +0000
    Ready:          False
    Restart Count:  0
    Liveness:       exec [sh -c if [ -x "$(command -v curl)" ]; then exec curl --fail http://localhost:9090/-/healthy; elif [ -x "$(command -v wget)" ]; then exec wget -q -O /dev/null http://localhost:9090/-/healthy; else exit 1; fi] delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:      exec [sh -c if [ -x "$(command -v curl)" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x "$(command -v wget)" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi] delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:        exec [sh -c if [ -x "$(command -v curl)" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x "$(command -v wget)" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi] delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment:    <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-kube-ceems-kube-prometheus-prometheus-rulefiles-0 from prometheus-kube-ceems-kube-prometheus-prometheus-rulefiles-0 (ro)
      /etc/prometheus/rules/prometheus-kube-ceems-kube-prometheus-prometheus-rulefiles-1 from prometheus-kube-ceems-kube-prometheus-prometheus-rulefiles-1 (ro)
      /etc/prometheus/rules/prometheus-kube-ceems-kube-prometheus-prometheus-rulefiles-2 from prometheus-kube-ceems-kube-prometheus-prometheus-rulefiles-2 (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-kube-ceems-kube-prometheus-prometheus-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bbh2j (ro)

Here are Kubelet logs:

May 31 18:03:23 ceems-k8s microk8s.daemon-kubelite[161083]: E0531 18:03:23.778417  161083 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = failed to exec in container: failed to start exec \"3a54869167adc538e5213b1023b3289e188f4c247cfa1a8d9476998334fe666c\": OCI runtime exec failed: exec failed: unable to start container process: exec: \"sh\": executable file not found in $PATH" containerID="083d656184d794d4a57a1c0b337a7627cc08e0125999ba608901264cef2b0045" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 31 18:03:23 ceems-k8s microk8s.daemon-kubelite[161083]: E0531 18:03:23.833777  161083 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = failed to exec in container: failed to start exec \"c66bb41e2d405701a6fb40e03b69fff307610ddfb1a7bc7c93ad38423e9f9893\": OCI runtime exec failed: exec failed: unable to start container process: exec: \"sh\": executable file not found in $PATH" containerID="083d656184d794d4a57a1c0b337a7627cc08e0125999ba608901264cef2b0045" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 31 18:03:23 ceems-k8s microk8s.daemon-kubelite[161083]: E0531 18:03:23.884588  161083 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = failed to exec in container: failed to start exec \"4acd36ef723ef089e5b8e884258f3b9b402b1c99acafd4c89208de81dc1c33a9\": OCI runtime exec failed: exec failed: unable to start container process: exec: \"sh\": executable file not found in $PATH" containerID="083d656184d794d4a57a1c0b337a7627cc08e0125999ba608901264cef2b0045" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
May 31 18:03:23 ceems-k8s microk8s.daemon-kubelite[161083]: E0531 18:03:23.884709  161083 prober.go:106] "Probe errored" err="rpc error: code = Unknown desc = failed to exec in container: failed to start exec \"4acd36ef723ef089e5b8e884258f3b9b402b1c99acafd4c89208de81dc1c33a9\": OCI runtime exec failed: exec failed: unable to start container process: exec: \"sh\": executable file not found in $PATH" probeType="Startup" pod="ceems/prometheus-kube-ceems-kube-prometheus-prometheus-0" podUID="38d8d161-e3f6-4403-8eb6-0904b6321318" containerName="prometheus" probeResult="unknown"

Steps to Reproduce

Deploy kube-prometheus-stack chart with values as below:

yaml

prometheus:
    prometheusSpec:
      listenLocal: true

Expected Result

Probes work as normal and containers going to Ready state.

Actual Result

Containers do not achieve Ready state.

Prometheus Operator Version

bash

Name:                   kube-ceems-kube-prometheus-operator
Namespace:              ceems
CreationTimestamp:      Sun, 31 May 2026 21:31:51 +0000
Labels:                 app=kube-prometheus-stack-operator
                        app.kubernetes.io/component=prometheus-operator
                        app.kubernetes.io/instance=kube-ceems
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=kube-prometheus-stack-prometheus-operator
                        app.kubernetes.io/part-of=kube-prometheus-stack
                        app.kubernetes.io/version=86.0.2
                        chart=kube-prometheus-stack-86.0.2
                        heritage=Helm
                        release=kube-ceems
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: kube-ceems
                        meta.helm.sh/release-namespace: ceems
Selector:               app=kube-prometheus-stack-operator,release=kube-ceems
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=kube-prometheus-stack-operator
                    app.kubernetes.io/component=prometheus-operator
                    app.kubernetes.io/instance=kube-ceems
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=kube-prometheus-stack-prometheus-operator
                    app.kubernetes.io/part-of=kube-prometheus-stack
                    app.kubernetes.io/version=86.0.2
                    chart=kube-prometheus-stack-86.0.2
                    heritage=Helm
                    release=kube-ceems
  Service Account:  kube-ceems-kube-prometheus-operator
  Containers:
   kube-prometheus-stack:
    Image:      quay.io/prometheus-operator/prometheus-operator:v0.91.0
    Port:       10250/TCP (https)
    Host Port:  0/TCP (https)
    Args:
      --kubelet-endpoints=false
      --kubelet-endpointslice=false
      --namespaces=ceems
      --localhost=127.0.0.1
      --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.91.0
      --config-reloader-cpu-request=0
      --config-reloader-cpu-limit=0
      --config-reloader-memory-request=0
      --config-reloader-memory-limit=0
      --thanos-default-base-image=quay.io/thanos/thanos:v0.41.0
      --secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1
      --web.enable-tls=true
      --web.cert-file=/cert/cert
      --web.key-file=/cert/key
      --web.listen-address=:10250
      --web.tls-min-version=VersionTLS13
    Liveness:   http-get https://:https/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get https://:https/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      GOGC:  30
    Mounts:
      /cert from tls-secret (ro)
  Volumes:
   tls-secret:
    Type:          Secret (a volume populated by a Secret)
    SecretName:    kube-ceems-kube-prometheus-admission
    Optional:      false
  Node-Selectors:  <none>
  Tolerations:     <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kube-ceems-kube-prometheus-operator-69bbb56ffc (1/1 replicas created)
Events:          <none>

Kubernetes Version

yaml

clientVersion:
  buildDate: "2025-12-18T08:37:08Z"
  compiler: gc
  gitCommit: 66452049f3d692768c39c797b21b793dce80314e
  gitTreeState: clean
  gitVersion: v1.35.0
  goVersion: go1.25.5
  major: "1"
  minor: "35"
  platform: linux/amd64
kustomizeVersion: v5.7.1
serverVersion:
  buildDate: "2025-12-18T08:38:08Z"
  compiler: gc
  emulationMajor: "1"
  emulationMinor: "35"
  gitCommit: 66452049f3d692768c39c797b21b793dce80314e
  gitTreeState: clean
  gitVersion: v1.35.0
  goVersion: go1.25.5
  major: "1"
  minCompatibilityMajor: "1"
  minCompatibilityMinor: "34"
  minor: "35"
  platform: linux/amd64

Kubernetes Cluster Type

kind

How did you deploy Prometheus-Operator?

helm chart:prometheus-community/kube-prometheus-stack

Manifests

yaml

prometheus-operator log output

bash

N/A

Anything else?

More details in Prometheus Helm chart repo

Source: prometheus-operator/prometheus-operator

View original on GitHub View discussion on GitHub