#7500·scrapy

HTTPS proxy certificate checking

Author: wRARCreated May 8, 2026Updated Aug 27, 2026
Labelsdiscusshttps

We have DOWNLOAD_VERIFY_CERTIFICATES for HTTPS connections to websites, but we can also have HTTPS connections to proxies. We even support this in HTTP11DownloadHandler for HTTP destinations (which are very rare nowadays, #5828 covers adding full support), and non-Twisted download handlers should support them as easily as HTTP proxies. HTTP11DownloadHandler currently always checks the certificates but #7496 makes that depend on DOWNLOAD_VERIFY_CERTIFICATES.

On one hand, tying proxy certificate verification to DOWNLOAD_VERIFY_CERTIFICATES (especially as it's off by default) sounds like a security problem that could lead to proxy credentials being stolen. On the other hand, always verifying them means not being able to write tests without monkey-patching or maybe doing some handler-specific things with trust stores, which is a big downside. We could add a separate setting for this, set it on by default and turn it off for tests, that could work.