#18093·selenium

[ Feature]: Remove dependencies on public URLs from tests

Author: titusfortnerCreated Sep 29, 2026Updated Sep 29, 2026
LabelsI-enhancement

Description

Several tests navigate to or connect to real public sites. These tests fail or flake when the site changes, is slow, is blocked by CI networking, or when the browser changes behavior on untrusted networks (we already have xfail_chrome markers for HTTP to HTTPS auto-upgrade on http://example.com). They also prevent running the suites with network access disabled.

A larger set of tests uses real registrable domains (example.com, google.com, test.com, etc.) purely as string data. Those never hit the network, but they are inconsistent and make it hard to audit which tests actually need network access.

This is breaking trunk CI now

Python / Browser Tests (chrome-bidi, windows) failed on trunk (4732af16c5). The only failures in the job are the two copies of one test, each failing all 3 attempts, so this is deterministic and not a flake:

  • //py:test/selenium/webdriver/common/bidi/browser_tests-chrome-bidi
  • //py:test/selenium/webdriver/common/_bidi/browser_tests-chrome-bidi
FAILED py\test\selenium\webdriver\common\bidi\browser_tests.py::test_create_user_context_with_direct_proxy[chrome]
AssertionError: assert 'example domain' in 'this domain is for use in documentation examples without needing permission. this is not a service, avoid relying on it for testing and monitoring purposes.\nlearn more'

Underlying cause: nothing changed in Selenium, the browser, or the driver. test_create_user_context_with_direct_proxy loads http://example.com/ and asserts the body text contains "example domain". IANA changed the page (Last-Modified: Mon, 28 Sep 2026 16:19:23 GMT, the day before the failing run) and removed the <h1>Example Domain</h1>. The body is now:

xml
<body><p>This domain is for use in documentation examples without needing permission. This is not a service, avoid relying on it for testing and monitoring purposes.</p><a href=https://iana.org/help/example-domains>Learn more</a><script src=/s.js></script></body>

"Example Domain" survives only in <title>, which is not part of the body text. The new page text itself says not to rely on it for testing.

The immediate fix is to point this test at a page on the local test server. Matching the new text or the title would only reset the clock until the next change. The rest of this issue covers the other tests with the same exposure.

Tests that reach the public network

Binding Location Host Notes
Python py/test/selenium/webdriver/common/bidi/browser_tests.py (test_create_user_context_with_accept_insecure_certs, ..._with_direct_proxy, ..._with_proxy_and_accept_insecure_certs) self-signed.badssl.com, example.com Asserts on third-party page content ("example domain", badssl h1)
Python py/test/selenium/webdriver/common/_bidi/browser_tests.py same Near-duplicate of the file above; confirm whether both directories should exist
Python py/test/selenium/webdriver/common/bidi/browser_tests.py (..._manual_proxy_all_params) example.com Goes through the local fake proxy, but already xfail on Chrome because of HTTPS upgrade
Python py/test/selenium/webdriver/remote/remote_hub_connection_tests.py wrong.host.badssl.com Asserts on the TLS error text from a third-party host
Python py/test/selenium/webdriver/common/example2.py www.google.com Searches Google and asserts on results; nothing references it, candidate for deletion
.NET dotnet/test/webdriver/NetworkInterceptionTests.cs (TransformNetworkResponse) www.selenium.dev Response is replaced by the handler, so a local page works equally well
.NET dotnet/test/webdriver/BiDi/Input/CombinedInputActionsTests.cs (Paint) kleki.com [Test] is commented out; delete or port to a local page
Ruby rb/spec/integration/selenium/webdriver/manager_spec.rb saucelabs.com subdomains, watir.com, selenium.dev Guarded with skip_unless: {driver: :none} so they never run; they need subdomains or HTTPS
Rust rust/tests/mirror_tests.rs registry.npmmirror.com Third-party mirror; Selenium Manager tests need network by design, but this adds a dependency on a host we do not control
Shared pages common/src/web/mouse_interaction.html selenium.dev Loads jQuery and jQuery UI CSS/JS from selenium.dev instead of the local copies
Java java/test/org/openqa/selenium/PageLoadingTest.java www.thisurldoesnotexist.comx Not a reserved TLD; currently @NotYetImplemented for all browsers
Java java/test/org/openqa/selenium/grid/distributor/DistributorNodeAvailabilityTest.java does.not.exist Not a reserved TLD

Real domains used only as data (no network access)

Not failures today, but worth normalizing:

  • google.com: Java EndToEndTest and GridSupplier.main (spoofed session, never loads), JS test/lib/http_test.js, webdriver_test.js, until_test.js, Ruby IE options and Firefox profile specs, Python example2.py
  • test.com, some.host.com, cheese.com, my.domain.com, selenium2.ru, idp.com, foo.com, boo.com, yes.com, froogle.com, another.server.com, get.com, navigate-to.com, http_proxy.com, https_proxy.com, tracking.com, other.com, proxy.org, pac.url, a.base.url: mocks, proxy config, and parsing tests across all bindings
  • example.com / example.org in unit tests: reserved by RFC 2606 and acceptable as data, as long as nothing resolves or loads it
  • HttpClientTestBase.shouldAllowUrlsWithSchemesToBeUsed creates a client for http://example.com (the code comment already calls it a terrible choice); the request itself goes to the local server

Already correct, no change needed: *.test, *.example, and *.invalid hosts (not_a_valid_url.test, invalid-domain-that-does-not-exist.test, doesnotexist.invalid, rp.example, idp.example, grid.example), and URLs in comments, skip reasons, and error message assertions.

Proposed changes

  1. Serve everything from the local test web server.
    • Insecure cert tests: use the test server's existing self-signed HTTPS endpoint instead of badssl.com.
    • Proxy tests: navigate to a reserved hostname (for example http://proxied.test/) that only resolves via the fake proxy, and use a local page for the direct-proxy case.
    • Cookie domain and secure cookie tests: add subdomain and HTTPS support to the test environment (for example *.localhost aliases) so the Ruby specs can run instead of being permanently skipped.
    • TransformNetworkResponse: use a local page.
    • mouse_interaction.html: reference the jQuery files already in common/src/web.
  2. Replace unresolvable-host URLs with reserved names (.invalid or .test).
  3. Replace real domains used as data with reserved ones (example.com, *.example, *.test).
  4. Delete dead code: example2.py (and google_one_box if unused), the disabled Paint test.
  5. Decide on the Rust mirror test: keep with an explicit note that it depends on a third party, or point it at a mirror we control.
  6. Add a lint check that fails on new non-reserved hosts in test directories, with an allowlist for comments, bug tracker links, and Selenium Manager tests.
  7. Once clean, audit requires-network tags so only targets that truly need it (browser and driver downloads, Selenium Manager) keep it.

Have you considered any alternatives or workarounds?

Marking the affected tests as flaky or skipping them in CI hides the problem and loses coverage. Keeping public URLs but retrying does not help with content changes or blocked networks.