sshuttle doesn't cooperate with Cloudflare Warp Zero Trust
We're testing out CloudFlare Warp, which is... vaguely VPN-shaped, but really more about getting corporate traffic onto CloudFlare networks to do deep inspection. Unfortunately, it uses pf to do its magic and it isn't playing nicely with sshuttle.
(It also does SSL MITM, but I don't think that's relevant here, but mentioning it just in case.)
I'm also going to ask CloudFlare for assistance on this, but I figure I'd approach it from both ends, especially with all the experience the sshuttle team has on figuring out these conflicts.
I did look at #563 and some other VPN-related tickets, but neither the --dns option nor passing ssh the -b ipaddr option helped.
When I run nc 10.17.186.217 22 it times out trying to connect. If Warp is disabled, sshuttle works as expected and nc connects to that internal host's ssh service.
Here is all the details I think you might care about:
sshuttle logCommand: sshuttle -vvv -r bastion 10.17.0.0/16
Starting sshuttle proxy (version 1.1.2).
c : which() could not find 'doas' in /Users/rlafferty/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/Users/rlafferty/.bash-my-aws/bin:/Users/rlafferty/gbin:/Users/rlafferty/bin:/Users/rlafferty/.local/bin:/Users/rlafferty/.rd/bin:/Users/rlafferty/code/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/System/Cryptexes/App/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Ghostty.app/Contents/MacOS
c : which() found 'sudo' at /usr/bin/sudo
c : Starting firewall manager with command: ['/usr/bin/sudo', '-p', '[local sudo] Password: ', '/usr/bin/env', 'PYTHONPATH=/opt/homebrew/Cellar/sshuttle/1.1.2/libexec/lib/python3.12/site-packages', '/opt/homebrew/Cellar/sshuttle/1.1.2/libexec/bin/python', '/opt/homebrew/bin/sshuttle', '-v', '-v', '-v', '--method', 'auto', '--firewall']
[local sudo] Password:
fw: Starting firewall with Python version 3.12.7
fw: Selecting a method automatically...
fw: which() could not find 'iptables' in /Users/rlafferty/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/Users/rlafferty/.bash-my-aws/bin:/Users/rlafferty/gbin:/Users/rlafferty/bin:/Users/rlafferty/.local/bin:/Users/rlafferty/.rd/bin:/Users/rlafferty/code/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/System/Cryptexes/App/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Ghostty.app/Contents/MacOS
fw: nat method not supported because 'iptables' command is missing.
fw: which() could not find 'nft' in /Users/rlafferty/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/Users/rlafferty/.bash-my-aws/bin:/Users/rlafferty/gbin:/Users/rlafferty/bin:/Users/rlafferty/.local/bin:/Users/rlafferty/.rd/bin:/Users/rlafferty/code/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/System/Cryptexes/App/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Ghostty.app/Contents/MacOS
fw: nft method not supported because 'nft' command is missing.
fw: which() found 'pfctl' at /sbin/pfctl
fw: Method 'pf' was automatically selected.
fw: which() found 'pfctl' at /sbin/pfctl
fw: ready method name pf.
c : IPv6 enabled: Using default IPv6 listen address ::1
c : Method: pf
c : IPv4: on
c : IPv6: on
c : UDP : off (not available with pf method)
c : DNS : off (available)
c : User: off (not available with pf method)
c : Subnets to forward through remote host (type, IP, cidr mask width, startPort, endPort):
c : (<AddressFamily.AF_INET: 2>, '10.17.0.0', 16, 0, 0)
c : Subnets to exclude from forwarding:
c : (<AddressFamily.AF_INET: 2>, '127.0.0.1', 32, 0, 0)
c : (<AddressFamily.AF_INET6: 30>, '::1', 128, 0, 0)
c : Trying to bind redirector on port 12300
c : TCP redirector listening on ('::1', 12300, 0, 0).
c : TCP redirector listening with <socket.socket fd=5, family=30, type=1, proto=0, laddr=('::1', 12300, 0, 0)>.
c : TCP redirector listening on ('127.0.0.1', 12300).
c : TCP redirector listening with <socket.socket fd=7, family=2, type=1, proto=0, laddr=('127.0.0.1', 12300)>.
c : Starting client with Python version 3.12.7
c : Connecting to server...
c : which() found 'ssh' at /usr/bin/ssh
c : executing: ['/usr/bin/ssh', 'BASTION', '--', '/bin/sh -c \'P=python3; $P -V 2>/dev/null || P=python; exec "$P" -c \'"\'"\'import sys, os; verbosity=3; sys.stdin = os.fdopen(0, "rb"); exec(compile(sys.stdin.read(1702), "assembler.py", "exec")); sys.exit(98);\'"\'"\'; exit 97\'']
c : > channel=0 cmd=PING len=7 (fullness=0)
s: Running server on remote host with /usr/bin/python3 (version 3.8.10)
s: assembling 'sshuttle' (88 bytes)
s: assembling 'sshuttle.cmdline_options' (85 bytes)
s: assembling 'sshuttle.helpers' (2655 bytes)
s: assembling 'sshuttle.ssnet' (5704 bytes)
s: assembling 'sshuttle.hostwatch' (2517 bytes)
s: assembling 'sshuttle.server' (3780 bytes)
s: latency control setting = True
s: > channel=0 cmd=PING len=7 (fullness=0)
s: auto-nets:False
s: > channel=0 cmd=ROUTES len=0 (fullness=7)
s: Waiting: 1 r=[0] w=[1] x=[] (fullness=7/0)
s: Ready: 1 r=[] w=[1] x=[]
s: mux wrote: 15/15
s: Waiting: 1 r=[0] w=[1] x=[] (fullness=7/0)
s: Ready: 1 r=[] w=[1] x=[]
s: mux wrote: 8/8
s: Waiting: 1 r=[0] w=[] x=[] (fullness=7/0)
c : Connected to server.
c : Waiting: 2 r=[5, 7, 9] w=[9] x=[] (fullness=7/0)
c : Ready: 2 r=[9] w=[9] x=[]
c : < channel=0 cmd=PING len=7
c : > channel=0 cmd=PONG len=7 (fullness=7)
c : < channel=0 cmd=ROUTES len=0
fw: Got subnets: [(2, 16, False, '10.17.0.0', 0, 0), (2, 32, True, '127.0.0.1', 0, 0), (30, 128, True, '::1', 0, 0)]
fw: Got nslist: []
fw: Got ports: 12300,12300,0,0
fw: Got udp: False, user: None, group: None, tmark: 0x01, sshuttle_pid: 40490
fw: setting up.
fw: setting up IPv6.
fw: >> pfctl -s Interfaces -i lo -v
fw: >> pfctl -s all
fw: rules:
pass out inet6 proto tcp to ::1/128
fw: >> pfctl -a sshuttle6-12300 -f /dev/stdin
fw: >> pfctl -E
fw: setting up IPv4.
fw: >> pfctl -s Interfaces -i lo -v
fw: >> pfctl -s all
fw: rules:
rdr pass on lo0 inet proto tcp from ! 127.0.0.1 to 10.17.0.0/16 -> 127.0.0.1 port 12300
pass out route-to lo0 inet proto tcp to 10.17.0.0/16 keep state
pass out inet proto tcp to 127.0.0.1/32
fw: >> pfctl -a sshuttle-12300 -f /dev/stdin
fw: >> pfctl -E
fw: which() could not find 'resolvectl' in /Users/rlafferty/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/Users/rlafferty/.bash-my-aws/bin:/Users/rlafferty/gbin:/Users/rlafferty/bin:/Users/rlafferty/.local/bin:/Users/rlafferty/.rd/bin:/Users/rlafferty/code/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/System/Cryptexes/App/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Ghostty.app/Contents/MacOS
fw: which() could not find 'systemd-resolve' in /Users/rlafferty/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/Users/rlafferty/.bash-my-aws/bin:/Users/rlafferty/gbin:/Users/rlafferty/bin:/Users/rlafferty/.local/bin:/Users/rlafferty/.rd/bin:/Users/rlafferty/code/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/System/Cryptexes/App/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Ghostty.app/Contents/MacOS
c : mux wrote: 15/15
c : mux wrote: 15/15
c : Waiting: 2 r=[5, 7, 9] w=[] x=[] (fullness=14/0)
s: Ready: 1 r=[0] w=[] x=[]
s: < channel=0 cmd=PING len=7
s: > channel=0 cmd=PONG len=7 (fullness=7)
s: < channel=0 cmd=PONG len=7
s: received PING response
s: mux wrote: 15/15
s: Waiting: 1 r=[0] w=[] x=[] (fullness=0/0)
c : Ready: 2 r=[9] w=[] x=[]
c : < channel=0 cmd=PONG len=7
c : received PING response
c : Waiting: 2 r=[5, 7, 9] w=[] x=[] (fullness=0/0)
^Cfw: undoing changes.
fw: undoing IPv6 changes.
fw: >> pfctl -a sshuttle6-12300 -F all
fw: >> pfctl -X 3273491464797269955
fw: undoing IPv4 changes.
fw: >> pfctl -a sshuttle-12300 -F all
fw: >> pfctl -X 12663568868717924929
fw: which() could not find 'resolvectl' in /Users/rlafferty/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/Users/rlafferty/.bash-my-aws/bin:/Users/rlafferty/gbin:/Users/rlafferty/bin:/Users/rlafferty/.local/bin:/Users/rlafferty/.rd/bin:/Users/rlafferty/code/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/System/Cryptexes/App/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Ghostty.app/Contents/MacOS
fw: which() could not find 'systemd-resolve' in /Users/rlafferty/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/Users/rlafferty/.bash-my-aws/bin:/Users/rlafferty/gbin:/Users/rlafferty/bin:/Users/rlafferty/.local/bin:/Users/rlafferty/.rd/bin:/Users/rlafferty/code/go/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/System/Cryptexes/App/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Ghostty.app/Contents/MacOS
c :
c : Keyboard interrupt: exiting.(note that utun6 is Warp's tunnel interface, and I limited this to ipv4 to make the length manageable because of all of the ipv6 routes Warp adds)
$ netstat -rn -f inet
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.55.10.254 UGScg en0
default link#32 UCSIg utun6
0.0.0.1/32 utun6 Uc utun6
0.0.0.2/31 utun6 Uc utun6
0.0.0.4/30 utun6 Uc utun6
0.0.0.8/29 utun6 Uc utun6
0.0.0.16/28 utun6 Uc utun6
0.0.0.32/27 utun6 Uc utun6
0.0.0.64/26 utun6 Uc utun6
0.0.0.128/25 utun6 Uc utun6
0.0.1/24 utun6 Uc utun6
0.0.2/23 utun6 Uc utun6
0.0.4/22 utun6 Uc utun6
0.0.8/21 utun6 Uc utun6
0.0.16/20 utun6 Uc utun6
0.0.32/19 utun6 Uc utun6
0.0.64/18 utun6 Uc utun6
0.0.128/17 utun6 Uc utun6
0.1/16 utun6 Uc utun6
0.2/15 utun6 Uc utun6
0.4/14 utun6 Uc utun6
0.8/13 utun6 Uc utun6
0.16/12 utun6 Uc utun6
0.32/11 utun6 Uc utun6
0.64/10 utun6 Uc utun6
0.128/9 utun6 Uc utun6
1 utun6 Uc utun6
2/7 utun6 Uc utun6
4/6 utun6 Uc utun6
8/7 utun6 Uc utun6
10.55.10/24 link#16 UCS en0 !
10.55.10.1 0:1:2e:48:6f:7e UHLWI en0 381
10.55.10.10 0:e:58:c3:71:b4 UHLWI en0 !
10.55.10.27 54:2a:1b:df:93:e0 UHLWI en0 !
10.55.10.30/32 link#16 UCS en0 !
10.55.10.30 bc:d0:74:28:3b:9a UHLWI lo0
10.55.10.65 68:54:fd:4f:11:62 UHLWI en0 816
10.55.10.89 1c:30:8:61:97:a3 UHLWI en0 !
10.55.10.102 f8:36:9b:78:dc:dd UHLWI en0 1101
10.55.10.107 ea:fe:71:20:57:a8 UHLWI en0 1163
10.55.10.108 6a:ca:45:a6:cc:9e UHLWI en0 !
10.55.10.114 a0:a3:b3:9c:bd:e4 UHLWI en0 !
10.55.10.132 70:2c:9:69:17:b1 UHLWI en0 535
10.55.10.137 34:e1:d1:81:3f:7f UHLWI en0 315
10.55.10.138 d8:e7:43:53:be:1f UHLWI en0 1190
10.55.10.139 42:7c:3f:45:b6:a0 UHLWI en0 !
10.55.10.188 8a:b9:2e:59:cd:f2 UHLWI en0 !
10.55.10.218 38:42:b:30:5f:60 UHLWI en0 !
10.55.10.251 4:bc:6d:e8:cd:81 UHLWIi en0 1060
10.55.10.254/32 link#16 UCS en0 !
10.55.10.254 3c:7c:3f:45:b6:98 UHLWIir en0 1169
10.55.10.255 ff:ff:ff:ff:ff:ff UHLWbI en0 !
11 utun6 Uc utun6
12 utun6 Uc utun6
13/10 utun6 Uc utun6
13.64/11 utun6 Uc utun6
13.96/13 utun6 Uc utun6
13.104/15 utun6 Uc utun6
13.106/16 utun6 Uc utun6
13.107/22 utun6 Uc utun6
13.107.4/23 utun6 Uc utun6
13.107.6/25 utun6 Uc utun6
13.107.6.128/28 utun6 Uc utun6
13.107.6.144/29 utun6 Uc utun6
13.107.6.154/31 utun6 Uc utun6
13.107.6.156/30 utun6 Uc utun6
13.107.6.160/29 utun6 Uc utun6
13.107.6.168/31 utun6 Uc utun6
13.107.6.170/32 utun6 Uc utun6
13.107.6.172/30 utun6 Uc utun6
13.107.6.176/28 utun6 Uc utun6
13.107.6.193/32 utun6 Uc utun6
13.107.6.194/31 utun6 Uc utun6
13.107.6.196/30 utun6 Uc utun6
13.107.6.200/29 utun6 Uc utun6
13.107.6.208/28 utun6 Uc utun6
13.107.6.224/27 utun6 Uc utun6
13.107.7/24 utun6 Uc utun6
13.107.8/24 utun6 Uc utun6
13.107.9/25 utun6 Uc utun6
13.107.9.128/26 utun6 Uc utun6
13.107.9.193/32 utun6 Uc utun6
13.107.9.194/31 utun6 Uc utun6
13.107.9.196/30 utun6 Uc utun6
13.107.9.200/29 utun6 Uc utun6
13.107.9.208/28 utun6 Uc utun6
13.107.9.224/27 utun6 Uc utun6
13.107.10/23 utun6 Uc utun6
13.107.12/22 utun6 Uc utun6
13.107.16/23 utun6 Uc utun6
13.107.18/29 utun6 Uc utun6
13.107.18.8/31 utun6 Uc utun6
13.107.18.12/31 utun6 Uc utun6
13.107.18.14/32 utun6 Uc utun6
13.107.18.16/28 utun6 Uc utun6
13.107.18.32/27 utun6 Uc utun6
13.107.18.64/26 utun6 Uc utun6
13.107.18.128/25 utun6 Uc utun6
13.107.19/24 utun6 Uc utun6
13.107.20/22 utun6 Uc utun6
13.107.24/21 utun6 Uc utun6
13.107.32/19 utun6 Uc utun6
13.107.64/18 utun6 Uc utun6
13.107.132/22 utun6 Uc utun6
13.107.140/30 utun6 Uc utun6
13.107.140.4/31 utun6 Uc utun6
13.107.140.7/32 utun6 Uc utun6
13.107.140.8/29 utun6 Uc utun6
13.107.140.16/28 utun6 Uc utun6
13.107.140.32/27 utun6 Uc utun6
13.107.140.64/26 utun6 Uc utun6
13.107.140.128/25 utun6 Uc utun6
13.107.141/24 utun6 Uc utun6
13.107.142/23 utun6 Uc utun6
13.107.144/20 utun6 Uc utun6
13.107.160/19 utun6 Uc utun6
13.107.192/18 utun6 Uc utun6
13.108/14 utun6 Uc utun6
13.112/12 utun6 Uc utun6
13.128/9 utun6 Uc utun6
14/7 utun6 Uc utun6
16/6 utun6 Uc utun6
20/12 utun6 Uc utun6
20.16/14 utun6 Uc utun6
20.20/19 utun6 Uc utun6
20.20.64/18 utun6 Uc utun6
20.20.128/17 utun6 Uc utun6
20.21/16 utun6 Uc utun6
20.22/15 utun6 Uc utun6
20.Source: sshuttle/sshuttle