Baike.dev
Log in
Back to tool/Back to issues
#6733·external-dns

RFC2136 Provider Fails to Update DNS When First Nameserver is Down

Author: YasminkundurCreated Sep 16, 2026Updated Sep 17, 2026
Labelskind/bug

What happened:

When using RFC2136 provider with multiple nameservers configured in round-robin mode, and the primary nameserver becomes unreachable, ExternalDNS correctly attempts failover to the secondary nameserver. However, the RFC2136 UPDATE request sent to the failover nameserver is rejected with FORMERR (format error), causing the DNS record delta (adds/deletes) to fail and not be applied during that reconciliation cycle, even though the fallback nameserver is reachable and operational.

Example error sequence:

  • First nameserver 192.168.101.200:53 is down (connection refused)
  • ExternalDNS fails over to 192.168.101.201:53 (reachable)
  • Second nameserver returns status: FORMERR to the UPDATE request
  • Records are not applied despite failover occurring in same reconcile

What you expected to happen:

When the primary nameserver is down, ExternalDNS should:

  1. Detect the connection failure on the primary nameserver
  2. Retry the same RFC2136 UPDATE request to the next configured nameserver in the same reconciliation cycle
  3. The secondary nameserver should accept and apply the UPDATE
  4. DNS records should be updated on the reachable nameserver

How to reproduce it (as minimally and precisely as possible):

  1. Deploy ExternalDNS with RFC2136 provider and multiple nameservers:
--rfc2136-host=192.168.101.200,192.168.101.201
--rfc2136-port=53
--rfc2136-zone=mbl.int.internal
--rfc2136-tsig-keyname=externaldns-key
--rfc2136-tsig-secret=<base64-encoded-secret>
--rfc2136-load-balancing-strategy=round-robin

  1. Verify both nameservers are reachable and have the zone configured with dynamic updates enabled

  2. Shutdown or make unreachable the first nameserver (192.168.101.200)

  3. Trigger a pod restart or service endpoint change that requires DNS record update

  4. Observe ExternalDNS logs for the update failure:

time="2026-09-15T12:20:47Z" level=warning msg="warn in dns.Client.Exchange: dial tcp 192.168.101.200:53: connect: connection refused"
time="2026-09-15T12:20:47Z" level=info msg="Adding RR: cb-vz-1-0694.vzone.mbl.int.internal 60 A 192.168.100.110"
time="2026-09-15T12:20:47Z" level=info msg="Bad dns.Client.Exchange response: ;; opcode: UPDATE, status: FORMERR"
time="2026-09-15T12:20:47Z" level=error msg="RFC2136 create record failed: soft error\nbad return code: FORMERR"

Root Cause:

In provider/rfc2136/rfc2136.go, the SendMessage() function reuses and mutates the same DNS message across nameserver retry attempts. When retrying after the first nameserver fails, the message carries over mutated state from previous attempts (specifically TSIG header modifications via msg.SetTsig()), causing the second nameserver to reject the UPDATE as malformed.

Anything else we need to know?:

  • Running with 2 ExternalDNS instances across 2 clusters, both configured to update the same RFC2136 nameservers, increases the likelihood of concurrent retries and timing-related FORMERR responses.
  • Bind9 logs show successful zone updates on later reconciliation attempts, indicating the issue is transient per-attempt but blocks delivery during that specific cycle.
  • TSIG key is correctly configured on both nameservers and works for AXFR (zone transfers work consistently).

Environment:

  • External-DNS version: v0.21.0 (deployed), v0.22.0-3-g006527d5 (local)
  • DNS provider: RFC2136 (Bind9 9.21)
  • Configuration:
    • RFC2136 with 2 nameservers (192.168.101.200, 192.168.101.201)
    • Load balancing strategy: round-robin
    • TSIG authentication: hmac-sha256
    • Zone: mbl.int.internal
  • Kubernetes: OpenShift 4.x
  • Other: 2-cluster setup, both running ExternalDNS controllers

LOGS

time="2026-09-15T12:17:01Z" level=info msg="GitCommitShort=unknown, GoVersion=go1.26.1, Platform=linux/amd64, UserAgent=ExternalDNS/v0.21.0-rfc2136-fix2"
time="2026-09-15T12:17:01Z" level=info msg="Created Kubernetes client https://172.30.0.1:443"
time="2026-09-15T12:17:01Z" level=info msg="Configured RFC2136 with zones '[mbl.int.internal]' and nameservers '[192.168.101.200 192.168.101.201]'"
time="2026-09-15T12:17:01Z" level=info msg="All records are already up to date"
time="2026-09-15T12:17:21Z" level=info msg="All records are already up to date"
time="2026-09-15T12:17:42Z" level=info msg="All records are already up to date"
time="2026-09-15T12:18:02Z" level=info msg="All records are already up to date"
time="2026-09-15T12:18:22Z" level=info msg="All records are already up to date"
time="2026-09-15T12:18:43Z" level=info msg="All records are already up to date"
time="2026-09-15T12:19:04Z" level=info msg="All records are already up to date"
time="2026-09-15T12:19:25Z" level=info msg="All records are already up to date"
time="2026-09-15T12:19:45Z" level=info msg="Removing RR: cb-vz-1-0693.vzone.mbl.int.internal 60 A 192.168.100.110"
time="2026-09-15T12:19:45Z" level=info msg="Removing RR: external-dnsa-cb-vz-1-0693.vzone.mbl.int.internal 0 TXT \"heritage=external-dns,external-dns/owner=ocp1\""
time="2026-09-15T12:20:06Z" level=info msg="Removing RR: cb-vz-1-0693.vzone.mbl.int.internal 60 A 192.168.100.110"
time="2026-09-15T12:20:06Z" level=info msg="Removing RR: external-dnsa-cb-vz-1-0693.vzone.mbl.int.internal 0 TXT \"heritage=external-dns,external-dns/owner=ocp1\""
time="2026-09-15T12:20:27Z" level=warning msg="Last operation failed for nameserver 192.168.101.201:53"
time="2026-09-15T12:20:27Z" level=warning msg="Last operation error message: failed to fetch records via AXFR: failed to connect for transfer: dial tcp 192.168.101.200:53: connect: connection refused"
time="2026-09-15T12:20:27Z" level=info msg="All records are already up to date"
time="2026-09-15T12:20:47Z" level=warning msg="Last operation failed for nameserver 192.168.101.201:53"
time="2026-09-15T12:20:47Z" level=warning msg="Last operation error message: failed to fetch records via AXFR: failed to connect for transfer: dial tcp 192.168.101.200:53: connect: connection refused"
time="2026-09-15T12:20:47Z" level=info msg="Adding RR: cb-vz-1-0694.vzone.mbl.int.internal 60 A 192.168.100.110"
time="2026-09-15T12:20:47Z" level=info msg="Adding RR: external-dnsa-cb-vz-1-0694.vzone.mbl.int.internal 60 TXT \"heritage=external-dns,external-dns/owner=ocp1\""
time="2026-09-15T12:20:47Z" level=warning msg="warn in dns.Client.Exchange: dial tcp 192.168.101.200:53: connect: connection refused"
time="2026-09-15T12:20:47Z" level=warning msg="Last operation failed for nameserver 192.168.101.201:53"
time="2026-09-15T12:20:47Z" level=warning msg="Last operation error message: dial tcp 192.168.101.200:53: connect: connection refused"
time="2026-09-15T12:20:47Z" level=info msg="Bad dns.Client.Exchange response: ;; opcode: UPDATE, status: FORMERR, id: 44649\n;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0\n\n;; ZONE SECTION:\n;mbl.int.internal.\tIN\t SOA\n"
time="2026-09-15T12:20:47Z" level=error msg="RFC2136 create record failed: soft error\nbad return code: FORMERR"
time="2026-09-15T12:20:47Z" level=error msg="Failed to do run once: soft error\nRFC2136 had errors in one or more of its batches: [soft error\nbad return code: FORMERR] (consecutive soft errors: 1)"

Checklist

  • I have searched existing issues and tried to find a fix myself
  • I am using the latest release, or have checked the staging image to confirm the bug is still reproducible
  • I have provided the actual process flags (not Helm values)
  • I have provided full external-dns debug logs
  • I have described what DNS records exist and what I expected
  • Root cause identified: DNS message reuse across retry attempts causes TSIG mutation

Source: kubernetes-sigs/external-dns

View original on GitHubView discussion on GitHub