Solver ingressTemplate annotations are not applied to existing Ingress resources when acme.cert-manager.io/http01-edit-in-place: 'true' is set
Describe the bug:
When using a solver which defines annotations via ingressTemplate, these annotations do not get applied to the Ingress resource when acme.cert-manager.io/http01-edit-in-place: 'true' is set.
Expected behaviour:
The annotation ingress.cilium.io/force-https: disabled should be applied to the Ingress while the acme challenge is active and cleared when it has compleded.
Steps to reproduce the bug: Set up an Issuer:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: test-issuer
spec:
acme:
caBundle: ...
privateKeySecretRef:
name: issuer-acme-account-key
server: https://redacted
solvers:
- http01:
ingress:
ingressClassName: cilium
ingressTemplate:
metadata:
annotations:
ingress.cilium.io/force-https: disabled
serviceType: ClusterIPSet up an Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
acme.cert-manager.io/http01-edit-in-place: 'true'
cert-manager.io/cluster-issuer: test-issuer
name: acme-ingress
namespace: testing
spec:
ingressClassName: cilium
rules:
- host: hello-k8s.redacted
http:
paths:
- backend:
service:
name: test-backend
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- hello-k8s.redacted
secretName: acme-ingress-tls-certObserve the ingress resource as the acme challenge progresses.
Anything else we need to know?: The missing annotation is a problem, because cilium enforces redirects to https by default, once a tls secret is defined on the ingress resource. So cert-manager cannot reach the acme challenge path over http which can be seen in hubble:
Mar 3 17:48:41.195: cert-manager/cert-manager-dc95d9f8c-2lppz:53290 (ingress) -> kube-system/cilium-ingress:80 (world) http-request FORWARDED (HTTP/1.1 GET http://hello-k8s.redacted/.well-known/acme-challenge/KEsqBOGPjM5X7bF88fgGkxQ1UakWQLMH)
Mar 3 17:48:41.195: cert-manager/cert-manager-dc95d9f8c-2lppz:53290 (ingress) <- kube-system/cilium-ingress:80 (world) http-response FORWARDED (HTTP/1.1 301 0ms (GET http://hello-k8s.redacted/.well-known/acme-challenge/KEsqBOGPjM5X7bF88fgGkxQ1UakWQLMH))
Mar 3 17:48:41.211: cert-manager/cert-manager-dc95d9f8c-2lppz:57964 (ingress) -> hello-k8s/cm-acme-http-solver-9jvdq:8089 (ID:24673) http-request FORWARDED (HTTP/1.1 GET https://hello-k8s.redacted/.well-known/acme-challenge/KEsqBOGPjM5X7bF88fgGkxQ1UakWQLMH)
Mar 3 17:48:41.212: cert-manager/cert-manager-dc95d9f8c-2lppz:57964 (ingress) <- hello-k8s/cm-acme-http-solver-9jvdq:8089 (ID:24673) http-response FORWARDED (HTTP/1.1 200 1ms (GET https://hello-k8s.redacted/.well-known/acme-challenge/KEsqBOGPjM5X7bF88fgGkxQ1UakWQLMH))The challenge gets stuck in this state for a while until it aborts and reports:
"Error accepting authorization: acme: authorization error for hello-k8s.redacted: 0 de:mtg:params:acme:error:failedChallengeValidation: Challenge could not be validated; subproblems:\n\turn:ietf:params:acme:error:connection: URL 'http://hello-k8s.redacted/.well-known/acme-challenge/KEsqBOGPjM5X7bF88fgGkxQ1UakWQLMH' was not reachableWhen I set ingress.cilium.io/force-https: disabled manually while the challenge is still active, it succeeds immediately.
Environment details:
- Kubernetes version: 1.33.7
- Cilium 1.18.4
- Cloud-provider/provisioner: rke2
- cert-manager version: 1.19.4
- Install method: helm
/kind bug
Source: cert-manager/cert-manager