cloudflare DNS01 - Client.Timeout exceeded while awaiting headers
Describe the bug:
When I try to use DNS01 challenge with cloudflare it throws error: Client.Timeout exceeded while awaiting headers.
I know token works (I've tried to use it with certbot and acme.sh and it works, below I even have a curl request that shows it works). There is no network policies, proxies, firewalls or anything else applied to the cluster. I'm using k3d for local development, but I've tried to use plain k3s inside a VM and got the same error.
Expected behaviour:
I expected it to successfully create DNS TXT records.
Steps to reproduce the bug:
- Start a k3d cluster (or k3s inside a VM)
- bootstrap a basic fluxcd repo
- Add HelmRelease and cert-manager configuration to the repo
- Wait it to reconcile and spawn challenges
- See infinite "pending state" with
Client.Timeout exceeded while awaiting headerserrors.
Anything else we need to know?:
K3S setup (disabled: helm-controller, metric-server, traefik)
# k3d
k3d cluster create \
--k3s-arg "--disable=metrics-server,traefik@server:*" \
--k3s-arg "--disable-helm-controller@server:*" \
--port "80:80@loadbalancer" \
--port "443:443@loadbalancer" \
--image rancher/k3s:v1.35.0-k3s1
# k3s
curl -sfL https://get.k3s.io | \
INSTALL_K3S_VERSION=v1.35.0+k3s1 \
INSTALL_K3S_EXEC="server --disable=metrics-server,traefik --disable-helm-controller" \
sh -
flux bootstrap gitlab \
--hostname=host \
--owner=username \
--repository=repostiory \
--branch=main \
--path=fluxcd \
--components-extra=image-reflector-controller,image-automation-controller \
--read-write-key \
--personal \
--token-auth
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: cert-manager
namespace: flux-system
spec:
interval: 1h
url: oci://quay.io/jetstack/charts/cert-manager
ref:
semver: "1.19.2"
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: cert-manager
namespace: flux-system
spec:
interval: 10m
chartRef:
kind: OCIRepository
name: cert-manager
install:
strategy:
name: RetryOnFailure
retryInterval: 2m
upgrade:
strategy:
name: RetryOnFailure
retryInterval: 2m
values:
crds:
enabled: true
keep: false
dns01RecursiveNameservers: 1.1.1.1:53,9.9.9.9:53
dns01RecursiveNameserversOnly: true
podDnsConfig:
nameservers:
- 1.1.1.1
- 9.9.9.9
podDnsPolicy: None
prometheus:
enabled: false
---
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
---
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token
namespace: cert-manager
data:
api-token: my-token
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-staging-key
solvers:
- dns01:
cloudflare:
apiTokenSecretRef:
name: cloudflare-api-token
key: api-token
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: tls-certificate
namespace: cert-manager
spec:
secretName: tls-certificate
issuerRef:
name: letsencrypt-staging
kind: Issuer
dnsNames:
- "my-domain.ru"
- "*.my-domain.ru"
Full error from challenge description:
Warning PresentError 2m2s (x8 over 22m) cert-manager-challenges Error presenting challenge: while attempting to find Zones for domain _acme-challenge.my-domain.ru.
while querying the Cloudflare API for GET "/zones?name=_acme-challenge.my-domain.ru": Get "https://api.cloudflare.com/client/v4/zones?name=_acme-challenge.my-domain.ru": context deadline exceeded (Client.Timeout exceeded while awaiting headers)Token configuration: All zones - Zone:Read, DNS:Read, DNS:Edit.
Connection and token test from inside a cluster:
kubectl run debug --rm -it \
--image=curlimages/curl \
--restart=Never \
-n cert-manager \
-- time curl "https://api.cloudflare.com/client/v4/user/tokens/verify" -H "Authorization: Bearer $(kubectl get secret cloudflare-api-token -n cert-manager -o jsonpath='{.data.api-token}' | base64 -d)"
{"result":{"id":"a41662e724f57bbd43820f54bf41071e","status":"active"},"success":true,"errors":[],"messages":[{"code":10000,"message":"This API Token is valid and active","type":null}]}
real 0m 1.03s
user 0m 0.03s
sys 0m 0.02sEnvironment details:
- Kubernetes version: v1.35.0+k3s1
- Cloud-provider/provisioner: k3d/k3s
- cert-manager version: 1.19.3
- Install method: e.g., helm/static manifests: helm (fluxcd OCIRepository + HelmRelease)
/kind bug
Source: cert-manager/cert-manager