DNS-over-QUIC: the retry only covers two errors, and Quad9 returns neither
I forward to Quad9 over QUIC and got intermittent SERVFAILs for months. I switched the same server to DoT, to the same two Quad9 addresses, and they stopped completely. I think the difference is one line in the QUIC client's retry check.
My setup
- Technitium DNS Server 15.4 on .NET 10.0.11, Linux
- Forwarders:
dns.quad9.net:853 (9.9.9.9)anddns.quad9.net:853 (149.112.112.112) forwarderProtocol: Quic,forwarderConcurrency: 2,forwarderRetries: 3
What I measured
Over seven days on QUIC, my server logged 2,306 failed forwarder queries. Every one was a
QuicException, and there were only two messages between them: Stream aborted by peer (5).
(1,470) and Operation aborted. (836).
I then switched only forwarderProtocol to Tls — same addresses, same box, nothing else changed.
In the 7.5 hours since, there have been zero. At the rate I had been seeing, that window
should have produced somewhere between 50 and 100.
What I think is happening
In TechnitiumLibrary.Net/Dns/ClientConnection/QuicClientConnection.cs
(QueryAsync → QuicQueryAsync), a failed query throws the connection away and retries — but only
for two specific errors:
if (((ex.QuicError == QuicError.ConnectionIdle) || (ex.QuicError == QuicError.ConnectionAborted)) && (retry == 1))
{
retry = 0;
continue;
}
throw;Neither of my two errors is on that list, so the retry never ran. All 2,306 failures gave up on the first try. It does recover by itself — the next query builds a fresh connection and works — so this is one failed lookup each time, not a stuck connection pool.
My guess is that Quad9 closes connections that have gone quiet, and closes them in a way that shows up as a stream abort rather than an idle timeout. I can't see Quad9's side, so that part is a guess. Any server that behaves the same way would hit this.
I read the open library PR and I don't think it covers this
https://github.com/TechnitiumSoftware/TechnitiumLibrary/pull/62 — "Fix DNS-over-QUIC stream lifetime under cancellation". It changes how connections and streams are created and cleaned up, but leaves the retry check unchanged, so both of my errors still fail on the first try with it applied.
The change I'd suggest
The connection has already been thrown away by the time this check runs, so a retry gets a fresh one. Retrying once on any error would cover it:
if (retry == 1)
{
retry = 0;
continue;
}
throw;The TCP client, which the TLS client inherits from, already retries on IOException generally
rather than on a list of specific errors.
Two things I could not work out, in case they matter
- I have
forwarderRetriesset to 3, and it did not hide this. I don't know why those retries didn't cover it, and I couldn't work it out from the source. - I don't know how long each connection had been idle before it failed. My logs record the error but not the gap before it.
I haven't tested the patch — I moved to DoT instead, so the fix is a suggestion rather than something I've proven. Happy to send raw log lines if they'd help.
Source: TechnitiumSoftware/DnsServer