CoreDNS cannot handle DoT upstream queries when network returns fragmentation needed
I have a bizarre edge case that took me a while to debug.
My setup:
I am trying to set up an instance of CoreDNS to perform upstream DNS queries over a macvlan interface (this is part of a more complicated setup and in fact, necessary).
The interface is set up via Multus (using the reference macvlan CNI binary), and uses the master interface's MTU (1500). Additionally, through external (to the cluster) policy-based routing, all of the macvlan traffic is routed over a site-to-site VPN. This reduces the effective MTU for non-local IP addresses to 1420 (due to the VPN overhead).
Visually, this looks something like this:
CoreDNS pod ---macvlan---> Router ---VPN---> remote site ---> internet Because of this setup, the MTU for the macvlan LAN network is 1500, but the MTU for all other addresses is 1420.
Normally, a situation like this is handled transparently by the kernel either through a "proactive" PMTU discovery (i.e. something like ping -M want -c 3 -s 1500 9.9.9.9) or when a "Fragmentation Needed" ICMP error is returned for an outgoing connection.
What happened:
I noticed all CoreDNS DoT queries were failing, but only while the site-to-site policy-based routing rule is enabled on my router.
After some investigation, I managed to narrow down the problem to the "fragmentation needed" message being processed and applied to the CoreDNS connection, but the connection not succeeding anyway.
Visually, this looks something like this:
- Fragmentation needed message returned to the macvlan interface
$ tcpdump -ni any 'icmp[0]=3 and icmp[1]=4'
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
18:03:50.787383 macv-out0 In IP 10.1.10.1 > 10.1.10.50: ICMP 9.9.9.9 unreachable - need to frag (mtu 1420), length 556- The "learned" PMTU is applied
ss -itn dst 9.9.9.9
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 10.1.10.50:42636 9.9.9.9:853
bbr wscale:11,10 rto:272 rtt:69.213/26.137 mss:1368 pmtu:1420 rcvmss:536 advmss:1448 cwnd:12 bytes_sent:3092 bytes_retrans:1546 bytes_acked:1547 segs_out:7 segs_in:3 data_segs_out:4 data_segs_in:1 bbr:(bw:314408bps,mrtt:69.101,pacing_gain:2.88672,cwnd_gain:2.88672) send 1897447bps lastsnd:3016 lastrcv:3020 lastack:2948 pacing_rate 4789032bps delivery_rate 314568bps delivered:3 app_limited busy:72ms retrans:0/2 rcv_space:14480 rcv_ssthresh:42242 minrtt:69.101 rcv_ooopack:1 snd_wnd:43008 rcv_wnd:43008 tcp-ulp-tls rxconf: none txconf: noneYet the connection does not succeed. Ever.
At this point, I am not sure if this is a kernel bug (using Talos 1.12.2, kernel 6.18.8) or if it is just not possible to retransmit the dropped part of the packet because it was already offloaded to the hardware to deal with. (This is all speculation on my part)
What I do know for sure is that if I simulate this with openssl I get the same result (being stuck indefinitely) on the first try, but succeeding on the second try, due to the PMTU now being in the route cache:
$ ip route get 9.9.9.9
9.9.9.9 via 10.1.10.1 dev macv-out0 table 100 src 10.1.10.50 uid 0
cache # No cache
$ openssl s_client -connect 9.9.9.9:853 -servername dns.quad9.net
CONNECTED(00000003)
Connecting to 9.9.9.9
^C # attempt 1 fails
$ ip route get 9.9.9.9
9.9.9.9 via 10.1.10.1 dev macv-out0 table 100 src 10.1.10.50 uid 0
cache expires 590sec mtu 1420 # PMTU now in cache
$ openssl s_client -connect 9.9.9.9:853 -servername dns.quad9.net
Connecting to 9.9.9.9
CONNECTED(00000003)
depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G3
verify return:1
depth=1 C=US, O=DigiCert Inc, CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1
verify return:1
---
Certificate chain
0 s:C=CH, ST=Zurich, L=Zurich, O=Quad9, CN=dns.quad9.net
i:C=US, O=DigiCert Inc, CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1
... # SuccessWhat you expected to happen:
As far as I can tell CoreDNS tries to reuse the same connection for the lifetime of the pod. This is great, however, from my testing, it seems the only way to "learn" the effective MTU of the path is to first fail and then start a new connection with the PMTU in the route cache.
It would be great if CoreDNS could do one of 3 things:
- be smart enough to know when to start a new connection due to the path MTU changing
- Start a new connection after N query failures, just in case it's due to the path MTU changing
- Make the health probes fail after N query failures, so it can be restarted (this would need to be combined with a "proactive" PMTU discovery before CoreDNS starts)
How to reproduce it (as minimally and precisely as possible):
- Have a network that has a smaller effective MTU for non-local IPs than it does for LAN IPs
- Make a TLS connection (or some other type of connection that sets the "do not fragment" flag) that exceeds the non-local IP MTU
Anything else we need to know?:
Environment:
- the version of CoreDNS: CoreDNS-1.12.3
- Corefile:
servers:
- zones:
- zone: .
scheme: dns://
use_tcp: true
port: 53
plugins:
- name: errors
- name: health
configBlock: lameduck 5s
- name: ready
parameters: 0.0.0.0:8181
- name: log
- name: prometheus
parameters: 0.0.0.0:9153
- name: reload
- name: loop
- name: loadbalance
- name: bind
parameters: 0.0.0.0
- name: acl
configBlock: |-
allow net 192.168.254.0/24
drop
- name: whoami
- name: kubernetes
parameters: cluster.local in-addr.arpa
configBlock: |-
pods verified
fallthrough in-addr.arpa
- name: k8s_external
parameters: in-addr.arpa
configBlock: |-
fallthrough in-addr.arpa
- name: rewrite
parameters: stop
configBlock: |-
name suffix .in-addr.arpa. .in-addr.arpa.
answer name auto
answer value (.*)\.cluster\.local\. {1}.cluster.local
- name: cache
parameters: 30
- name: forward
parameters: . tls://9.9.9.9
configBlock: |-
tls_servername dns.quad9.net
max_fails 3
expire 10s
health_check 5s
failfast_all_unhealthy_upstreams- logs, if applicable:
[INFO] 192.168.254.3:58193 - 718 "A IN google.com. udp 39 false 1232" - - 0 30.000455613s
[ERROR] plugin/errors: 2 google.com. A: context deadline exceeded- OS (e.g:
cat /etc/os-release): Talos 1.12.2, kernel 6.18.8 - Others:
As a workaround, I am currently manually lowering the macvlan MTU to 1420 (even though the LAN network can handle 1500) via the macvlan CNI config. However, ideally I would like to have this resolved, as currently this will not work if CoreDNS tries to connect to something that goes through a route that has an even lower effective MTU (for whatever reason)
Source: coredns/coredns