Cloudflare custom hostname applied to every headless-service record, not just the annotated hostname

What happened:

For a headless (clusterIP: None) Service in front of a StatefulSet, annotated with both external-dns.kubernetes.io/hostname and external-dns.kubernetes.io/cloudflare-custom-hostname (--cloudflare-custom-hostnames enabled), external-dns applies the Cloudflare custom hostname to every DNS record it derives from that one Service — the aggregate record matching the hostname annotation, and one additional record per StatefulSet pod (<pod>.<hostname>, generated because those pods have pod.Spec.Hostname set).

A Cloudflare custom hostname is a 1:1 mapping (one custom hostname → one origin), so only the first of these records to sync wins. Every other record sharing the same custom-hostname value gets rejected by the Cloudflare API with a 409, and this repeats on every reconciliation instead of resolving:

level=error msg="failed to add custom hostname \"<custom-hostname>.example.com\": POST \"https://api.cloudflare.com/client/v4/zones/<zone-id>/custom_hostnames\": 409 Conflict {
  \"result\": null,
  \"success\": false,
  \"errors\": [{\"code\": 1406, \"message\": \"Duplicate custom hostname found.\"}],
  \"messages\": []
}"

Reproduced live against a StatefulSet with 2 replicas: 3 competing A records were generated for the one custom hostname (the aggregate record + one per pod), and only the first to sync succeeded.

What you expected to happen:

A cloudflare-custom-hostname annotation on a headless Service should only be applied to the record matching the literal external-dns.kubernetes.io/hostname value. The additional per-pod records that a headless Service in front of a StatefulSet also produces are a different DNS name serving a different purpose (per-pod discovery) and shouldn't inherit the same external-facing 1:1 alias.

How to reproduce it (as minimally and precisely as possible):

1. Run external-dns with the Cloudflare provider and --cloudflare-custom-hostnames enabled (real startup args from the deployment used to reproduce this, zone IDs redacted):

--source=service
--source=ingress
--zone-id-filter=<zone-id-1>
--zone-id-filter=<zone-id-2>
--provider=cloudflare
--annotation-filter=external-dns.kubernetes.io/exclude notin (true)
--policy=sync
--txt-owner-id=external-dns
--events
--listen-endpoint-events
--cloudflare-custom-hostnames
--cloudflare-custom-hostnames-certificate-authority=none

image: registry.k8s.io/external-dns/external-dns:v0.22.0

1. Live Service object (real object from the reproduction, names/domains redacted, all fields as returned by the API server):

apiVersion: v1
kind: Service
metadata:
  name: example-external-dns
  namespace: example-ns
  annotations:
    external-dns.kubernetes.io/cloudflare-custom-hostname: <custom-hostname>.example.com
    external-dns.kubernetes.io/cloudflare-proxied: "true"
    external-dns.kubernetes.io/endpoints-type: HostIP
    external-dns.kubernetes.io/hostname: example-app.internal.example.com
  labels:
    app: example
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: "9200"
    port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    app: example
    role: data
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

1. Point a StatefulSet with serviceName: example-external-dns and 2 replicas at this Service (labels/selector matching above). Real EndpointSlice produced for it (names redacted, both replicas land on the same node in this repro but that is incidental — the bug is not IP-collision dependent, see below):

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: example-external-dns-h6j6b
  namespace: example-ns
  labels:
    kubernetes.io/service-name: example-external-dns
    service.kubernetes.io/headless: ""
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
addressType: IPv4
ports:
- name: "9200"
  port: 9200
  protocol: TCP
endpoints:
- addresses: ["10.0.0.1"]
  conditions: {ready: true, serving: true, terminating: false}
  nodeName: node-1
  targetRef: {kind: Pod, name: example-0, namespace: example-ns}
- addresses: ["10.0.0.2"]
  conditions: {ready: true, serving: true, terminating: false}
  nodeName: node-1
  targetRef: {kind: Pod, name: example-1, namespace: example-ns}

1. Observe external-dns logs (real excerpt, names/domains/zone redacted, chronological, unedited otherwise):

level=info msg="Changing record." action=CREATE record=example-0.example-app.internal.example.com ttl=1 type=A zone=<zone-id>
level=info msg="Changing record." action=CREATE record=example-app.internal.example.com ttl=1 type=A zone=<zone-id>
level=info msg="Changing record." action=CREATE record=example-1.example-app.internal.example.com ttl=1 type=A zone=<zone-id>
level=info msg="Changing record." action=CREATE record=a-example-0.example-app.internal.example.com ttl=1 type=TXT zone=<zone-id>
level=info msg="Changing record." action=CREATE record=a-example-app.internal.example.com ttl=1 type=TXT zone=<zone-id>
level=info msg="Changing record." action=CREATE record=a-example-1.example-app.internal.example.com ttl=1 type=TXT zone=<zone-id>
level=info msg="Creating custom hostname \"<custom-hostname>.example.com\"" action=CREATE record=example-0.example-app.internal.example.com ttl=1 type=A zone=<zone-id>
level=info msg="Creating custom hostname \"<custom-hostname>.example.com\"" action=CREATE record=example-app.internal.example.com ttl=1 type=A zone=<zone-id>
level=error msg="failed to create custom hostname \"<custom-hostname>.example.com\": POST \"https://api.cloudflare.com/client/v4/zones/<zone-id>/custom_hostnames\": 409 Conflict {\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 1406,\n      \"message\": \"Duplicate custom hostname found.\"\n    }\n  ],\n  \"messages\": []\n}\n" action=CREATE record=example-app.internal.example.com ttl=1 type=A zone=<zone-id>
level=info msg="Creating custom hostname \"<custom-hostname>.example.com\"" action=CREATE record=example-1.example-app.internal.example.com ttl=1 type=A zone=<zone-id>
level=error msg="failed to create custom hostname \"<custom-hostname>.example.com\": POST \"https://api.cloudflare.com/client/v4/zones/<zone-id>/custom_hostnames\": 409 Conflict {\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 1406,\n      \"message\": \"Duplicate custom hostname found.\"\n    }\n  ],\n  \"messages\": []\n}\n" action=CREATE record=example-1.example-app.internal.example.com ttl=1 type=A zone=<zone-id>

Note: logging was at the deployment's normal (info) level, not --log-level=debug — I didn't want to change a shared production controller's log level to reproduce this. Happy to re-run with debug logging and attach the fuller trace if a maintainer needs it; the info-level excerpt above already shows the full failing sequence for one reconciliation.

1. Resulting DNS records (from the same reconciliation; only one of the three A records competing for the custom hostname wins):

Anything else we need to know?:

Traced this to source, not just observed behavior:

• extractHeadlessEndpoints intentionally fans a single headless Service out into an aggregate record (keyed by the bare hostname) plus one record per pod that has pod.Spec.Hostname set (true for any StatefulSet pod addressed via serviceName): https://github.com/kubernetes-sigs/external-dns/blob/v0.22.0/source/service.go#L231-L235

  go复制

  headlessDomains := []string{hostname}
  if pod.Spec.Hostname != "" {
      headlessDomains = append(headlessDomains,
          fmt.Sprintf("%s.%s", pod.Spec.Hostname, hostname))
  }

• generateEndpoints then copies ProviderSpecific (which carries the Cloudflare custom-hostname key) onto every endpoint that came back from extractHeadlessEndpoints, with no check that en.DNSName actually equals the hostname being processed: https://github.com/kubernetes-sigs/external-dns/blob/v0.22.0/source/service.go#L435-L438

  go复制

  for _, en := range endpoints {
      en.ProviderSpecific = providerSpecific
      en.SetIdentifier = setIdentifier
  }

• The Cloudflare provider doesn't filter by DNSName either — it treats any endpoint carrying CloudflareCustomHostnameKey as a legitimate origin for that custom hostname: https://github.com/kubernetes-sigs/external-dns/blob/v0.22.0/provider/cloudflare/cloudflare.go#L624-L631 and https://github.com/kubernetes-sigs/external-dns/blob/v0.22.0/provider/cloudflare/cloudflare.go#L666-L669

  go复制

  for _, v := range getEndpointCustomHostnames(ep) {
      newCustomHostnames[v] = p.newCustomHostname(v, ep.DNSName)
  }

This is a different bug from #4282 / its fix in #5624: that was about duplicate target IP values within a single record (Route53 rejecting a repeated IP in one record's value list), already handled today by the deduppedTargets set in buildHeadlessEndpoints. This issue is about the same provider-specific property being copied onto multiple different records (different DNSNames) derived from one annotated Service — reproduces regardless of whether the pods share a node/IP or not, since the aggregate record alone is already a second claimant against any one per-pod record.

Suggested fix — only propagate provider-specific properties to the endpoint whose DNSName equals hostname:

for _, en := range endpoints {
    if en.DNSName == hostname {
        en.ProviderSpecific = providerSpecific
        en.SetIdentifier = setIdentifier
    }
}

(Per-pod records may still need SetIdentifier for other reasons — happy to help narrow down intended behavior for headless per-pod records if a maintainer can weigh in.)

Environment:

• External-DNS version: v0.22.0 (registry.k8s.io/external-dns/external-dns:v0.22.0, current latest release)
• DNS provider: Cloudflare, with --cloudflare-custom-hostnames
• Others: headless (clusterIP: None) Service with endpoints-type: HostIP, in front of a StatefulSet

Checklist

• I have searched existing issues and tried to find a fix myself — closest is #4282 / #5624, which is a different bug (see above)
• I am using the latest release (v0.22.0)
• I have provided the actual process flags (not Helm values)
• I have provided kubectl get <resource> -o yaml output including status (Service has no status.loadBalancer content since it's ClusterIP: None/headless, not a LoadBalancer-type Service)
• I have provided full external-dns debug logs — info-level only (see note above); can provide debug logs on request
• I have described what DNS records exist and what I expected