DNS issues with VPN usage
httpx version:
v1.10.0
Current Behavior:
When using a VPN on Linux, httpx becomes unreliable due to intermittent DNS failures. Running the same command a few times will yield different results since the DNS resolution will fail, seemingly at random.
Expected Behavior:
A somewhat deterministic output every time I run the command on a given data set.
Steps To Reproduce:
- Connect to VPN (tested with Private Internet Access)
- Run httpx with a handful of hosts
- Observe failures
Anything else:
Here are a few executions with public servers, showing the DNS failures:
$ echo "https://perdu.com\nhttps://lapresse.ca/\nhttps://ledevoir.com" | httpx -v 2>&1 | grep -e http -e DBG
[INF] Current httpx version v1.10.0 (latest)
https://ledevoir.com
[DBG] Failed 'https://lapresse.ca/': Get "https://lapresse.ca/": cause="no address found for host"
https://perdu.com
$ echo "https://perdu.com\nhttps://lapresse.ca/\nhttps://ledevoir.com" | httpx -v 2>&1 | grep -e http -e DBG
[INF] Current httpx version v1.10.0 (latest)
https://perdu.com
[DBG] Failed 'https://lapresse.ca/': Get "https://lapresse.ca/": cause="no address found for host"
https://ledevoir.com
$ echo "https://perdu.com\nhttps://lapresse.ca/\nhttps://ledevoir.com" | httpx -v 2>&1 | grep -e http -e DBG
[INF] Current httpx version v1.10.0 (latest)
https://lapresse.ca/
[DBG] Failed 'https://ledevoir.com': Get "https://ledevoir.com": cause="no address found for host"
https://perdu.comNone of these have weird DNS, in fact they all resolve properly:
$ dig +short perdu.com
172.64.80.1
$ dig +short lapresse.ca
52.70.7.144
32.195.53.148
3.82.167.114
$ dig +short ledevoir.com
151.101.66.132
151.101.2.132
151.101.130.132
151.101.194.132Something strange I found is that if I specify a public resolver in httpx, the DNS resolution works fine every time, even if its the same DNS server that I have configured in /etc/resolv.conf:
$ cat /etc/resolv.conf
nameserver 1.1.1.1
$ echo "https://perdu.com\nhttps://lapresse.ca/\nhttps://ledevoir.com" | httpx -v 2>&1 | grep -e http -e DBG
[INF] Current httpx version v1.10.0 (latest)
[DBG] Failed 'https://ledevoir.com': Get "https://ledevoir.com": cause="no address found for host"
https://lapresse.ca/
https://perdu.com
$ echo "https://perdu.com\nhttps://lapresse.ca/\nhttps://ledevoir.com" | httpx -v -r 1.1.1.1 2>&1 | grep -e http -e DBG
[INF] Current httpx version v1.10.0 (latest)
[DBG] Using resolvers: 1.1.1.1
https://ledevoir.com
https://lapresse.ca/
https://perdu.comThis leads me to believe that DNS resolution is broken for whatever reason when using a VPN.
I checked with dnsx and it also fails to do DNS lookups when on VPN:
$ dnsx -v
_ __ __
__| | _ __ ___ \ \/ /
/ _' || '_ \ / __| \ /
| (_| || | | |\__ \ / \
\__,_||_| |_||___//_/\_\
projectdiscovery.io
[INF] Current dnsx version 1.2.3 (outdated)
$ echo "perdu.com\nlapresse.ca\nledevoir.com" | dnsx -v 2>&1 | grep -e DBG -e write
[DBG] query error for lapresse.ca: could not resolve, max retries exceeded
write udp 10.7.18.114:58914->149.112.112.112:53: write: operation not permitted
[DBG] query error for perdu.com: could not resolve, max retries exceeded
write udp 10.7.18.114:43895->9.9.9.9:53: write: operation not permitted
[DBG] query error for ledevoir.com: could not resolve, max retries exceeded
write udp 10.7.18.114:39107->208.67.222.222:53: write: operation not permitted
[DBG] incomplete response for perdu.com (all retries exhausted)
[DBG] incomplete response for ledevoir.com (all retries exhausted)
[DBG] incomplete response for lapresse.ca (all retries exhausted)But it works with a hardcoded DNS server flag.
[INF] Current dnsx version 1.2.3 (outdated)
ledevoir.com
lapresse.ca
perdu.comSource: projectdiscovery/httpx