mitmproxy's TLS fingerprint doesn't match typical web browsers

Cloudflare detects MITM via TLS fingerprinting [1][2]. mitmproxy's traffic is flagged as bot traffic, since the TLS fingerprint doesn't match the User-Agent's expected one. Cloudflare's "bot fight mode" [3] responds with a 403 error response, attempts to run JS and show captchas in response to this.

In cases where the request is a cross-site request (AJAX, etc), there's no sane way to work around this (since you don't even see the captcha in the browser). This makes mitmproxy unusable for sites which load content via cross-site AJAX requests to domains with Cloudflare bot detection enabled.

The ideal fix would be to mirror the client's TLS configuration (cipher suites, etc) in the outgoing connections. Another less perfect (but possibly easier) approach would be hardcoding a specific browser's (e.g., Chrome on Windows) TLS settings, and overriding the User-Agent header to match. This would make traffic indistinguishable from a normal browser.

This may also be an issue with AWS WAF's bot detection [4] and other similar services.

[1] https://malcolm.cloudflare.com/ [2] https://github.com/cloudflare/mitmengine [3] https://blog.cloudflare.com/super-bot-fight-mode/ [4] https://aws.amazon.com/waf/features/bot-control/