Advanced Forwarding app / Conditional Forwarder Zones intermittently discard valid upstream responses under concurrent multi-forwarder queries

Summary

The Advanced Forwarding app (v5.1) intermittently causes DNS resolution to fail with "All name servers failed to answer the request... RCODE=Refused/ServerFailure" even when every configured forwarder returns a correct, valid answer within milliseconds. Packet capture shows the correct response arriving and matching the outstanding query's transaction ID, yet the client-facing resolution still fails. Disabling the app (falling back to the DNS Server's own core forwarders setting for the same upstream list) reduced failures from 85-160 per 10-minute window down to zero observed in 27 minutes of clean measurement, under otherwise identical real-world load.

Environment

• Technitium DNS Server 15.4.0 (.NET 10 runtime)
• OS: Debian 13, deployed in an LXC container (Proxmox), not Docker
• Advanced Forwarding app v5.1, enableForwarding: true, appPreference: 200
• Forwarders: 4 addresses in one named profile (forwarderProtocol: Udp, dnssecValidation: true), matched to the wildcard * domain rule for the everyone client group
• concurrentForwarding: true, forwarderConcurrency: 10, forwarderRetries: 2, forwarderTimeout: 2000
• Real household DNS load: mixed A/AAAA/HTTPS queries from ~15-20 client devices, bursty (multiple distinct domains queried within the same second, e.g. by browsers issuing parallel HTTPS/A/AAAA lookups per page load)

Evidence

1. Valid response received, resolution still reported as failed

Packet capture on the container's network interface, correlated against Technitium's own systemd journal, at the exact timestamp of a logged failure:

Journal: [2026-09-15 13:38:00 UTC] DNS Server failed to resolve the request
         'github.com. A IN' using forwarders: 1.1.1.1, 1.0.0.1, 8.8.8.8, 8.8.4.4.

Wire (tcpdump, same second):
13:38:00.533467  192.168.1.242 -> 1.1.1.1:53   A? github.com  (txn 12063)
13:38:00.533467  192.168.1.242 -> 1.0.0.1:53   A? github.com  (txn 46360)
13:38:00.533467  192.168.1.242 -> 8.8.8.8:53   A? github.com  (txn 17019)
13:38:00.533507  192.168.1.242 -> 8.8.4.4:53   A? github.com  (txn 24860)

13:38:00.544656  1.1.1.1:53 -> 192.168.1.242   txn 12063  1/0/1  A 20.26.156.215
13:38:00.547412  1.0.0.1:53 -> 192.168.1.242   txn 46360  1/0/1  A 20.26.156.215
13:38:00.550127  8.8.8.8:53 -> 192.168.1.242   txn 17019  1/0/1  A 20.26.156.215
13:38:00.550137  8.8.4.4:53 -> 192.168.1.242   txn 24860  1/0/1  A 20.26.156.215

All four forwarders answered correctly within 17ms, transaction IDs matched, valid NOERROR A record. Technitium reported total failure for this exact query at this exact timestamp regardless.

A second, independent instance (different domain, different time, socket-pool setting toggled off in between to rule that out as a factor) showed the same pattern: valid matched responses on the wire, logged as a failure with no corresponding query packet at the reported failure timestamp — consistent with a stale/incorrect result being surfaced from an in-flight-request or cache path rather than a fresh forwarding attempt.

2. Ruled out before concluding this is app-specific

• Network path: 85/85 pings to the forwarders, sampled at the exact moment of 85 separate logged failures, showed 0% loss and normal latency (9-18ms).
• OS resources: thread count, open file descriptors, and socket totals were flat/identical across all 85 samples — no resource exhaustion correlating with failures.
• forwarderConcurrency: reducing from 10 to 1 made the failure rate worse, not better (12/10min -> 28/10min), ruling out a per-forwarder-concurrency fan-out theory.
• enableUdpSocketPool: toggled off, failure rate and the "valid response, still failed" pattern were unchanged.
• Comparative baseline: Pi-hole, configured with the identical 4 forwarders, under a higher synthetic query load (4941 queries/10min) against the same problem domains, showed a 0.10% failure rate (5 failures) — consistent with ordinary internet/anycast-routing background noise, not a systemic defect.

3. A/B result: disabling the Advanced Forwarding app

With the app's enableForwarding set to false (both at the app level and the client-group level) and the DNS Server's own core forwarders setting populated with the identical 4 addresses:

Note on measurement methodology: my first pass at this used journalctl -u dns -f in a loop to catch failures as they happened. I later realized journalctl -f replays a tail of recent journal history on process startup before following live output, which can double-count or misattribute a small number of stale entries to a fresh measurement window — this is what produced an initial (incorrect) "3 failures in 10 min" reading for the disabled condition, which was actually re-displaying two old entries from before the app was disabled. Re-measuring with a plain bounded --since/--until query (no follow mode, no replay possible) gave the true result: zero failures in a continuous 27-minute window. I re-ran the enabled condition with the same bounded method afterward (160/10min) to confirm the earlier follow-mode counts weren't themselves inflated by the same artifact — they were not (if anything the bounded count was higher), so the enabled-state numbers stand, and only the disabled-state "3" needed correcting (to 0).

Expected behavior

A DNS query that receives a correct, timely NOERROR response from every configured forwarder should resolve successfully and consistently, regardless of which forwarding mechanism (core forwarders setting vs. Advanced Forwarding app conditionalForwarders) handles it.

Actual behavior

Under concurrent, bursty query load (multiple distinct domains queried by different client devices within the same second — a normal home-network pattern, e.g. browser page loads triggering parallel A/AAAA/HTTPS lookups), the Advanced Forwarding app intermittently reports total resolution failure for queries whose correct answer is independently verifiable as having arrived from every forwarder within milliseconds.

Candidate mechanism (code-level, not yet proven with instrumentation)

I checked the Advanced Forwarding app's own ProcessRequestAsync (App.cs) — it is a synchronous, stateless, side-effect-free lookup (network-group match -> domain-pattern match -> return a list of FWD records). It doesn't touch sockets, has no shared mutable state, no caching. It only decides which forwarders to hand to the core server; it does not participate in the actual querying/arbitration. So the divergence isn't "buggy app code" — it's that supplying a non-null/non-empty forwarder list (from this app, or equally from Technitium's own native Conditional Forwarder Zones) routes the query through a different core code path than the plain forwarders setting does, and that path looks architecturally riskier:

DefaultRecursiveResolveAsync (used when no app/zone supplies a forwarder list; core forwarders setting only):

DnsClient dnsClient = new DnsClient(forwarders); // one DnsClient, given the whole list
dnsClient.Concurrency = _forwarderConcurrency;
return await dnsClient.ResolveAsync(question, cancellationToken);

Multi-server racing is delegated to a single call into DnsClient.ResolveAsync with the full server list — presumably the most heavily-exercised path in the codebase, since it's what a default "set your forwarders" configuration uses.

ConcurrentConditionalForwarderResolveAsync (used whenever more than one same-priority forwarder is supplied via conditionalForwarders -- i.e. whenever the Advanced Forwarding app, or native Conditional Forwarder Zones, are in play):

foreach (DnsResourceRecord conditionalForwarder in conditionalForwarders)
{
    tasks.Add(Task.Factory.StartNew(delegate () {
        return ConditionalForwarderResolveAsync(...); // separate DnsClient per forwarder
    }, CancellationToken.None, TaskCreationOptions.DenyChildAttach, TaskScheduler.Current).Unwrap());
}
// then a hand-rolled: while (tasks.Count > 0) { await Task.WhenAny(tasks); ... }

This hand-rolls the same multi-server racing that DefaultRecursiveResolveAsync gets for free from DnsClient, spawning one task and one separate DnsClient instance per forwarder, then arbitrating manually. Worth noting specifically: it schedules with TaskScheduler.Current rather than TaskScheduler.Default, which inherits whatever ambient scheduler is active at that point in the async chain rather than the plain thread pool -- a documented source of surprising behavior under nested async continuations.

I have not instrumented the code to prove this specific mechanism causes the observed symptom (that would need a debug build or added logging on your end) -- this is offered as a concrete starting point for investigation, not a confirmed root cause.

Notes / caveats

• I have not tested whether this reproduces on a non-LXC deployment (Docker/bare metal), so I can't rule out a container-networking interaction, though the packet-capture evidence (valid response demonstrably received by the process) argues against a pure network-layer cause.
• This same divergent-code-path pattern (ConcurrentConditionalForwarderResolveAsync / PriorityConditionalForwarderResolveAsync) would also be exercised by the DNS Server's native Conditional Forwarder Zones feature, not just this app -- worth checking whether that feature shows the same symptom.

Happy to provide the full pcap, systemd journal, or Advanced Forwarding app config on request.