Proposal: optional native parser backend with pure-Python fallback

python-dotenv optional native parser backend

Problem

python-dotenv parsing is on the configuration path of real consumers such as pydantic-settings. We measured complete consumer calls, not parser-only throughput, on the same macOS arm64 / CPython 3.14 environment.

Proposal

• Keep the current pure-Python parser and pip install python-dotenv default.
• Add an optional native extra whose backend-only dependency is fast-dotenv-rs-backend.
• Keep the Rust package separate from the dotenv namespace and console script; the adapter only converts lossless parser records into the existing Binding and Original objects.
• If the extra/backend is absent, the existing pure-Python path remains in use. If an explicitly selected backend violates its versioned contract or raises, surface that failure rather than silently masking it.

Runtime evidence

• Upstream base: a00cb2eed0704cd6d2071b2004c37e95ccc86ee5.
• Backend implementation: fast-dotenv-rs commits 7cbdca0 and 95ea5c1 in the Master integration clone; shared Rust core and backend-only wheel are ready for review.
• Upstream adapter patch: python-dotenv commits 0c1d17b9878ae2b390616fbd8b64f605f9fd9d67 and f5856435e229485040e0e11530a191b0f51007df.
• Exact Binding differential: 1,088 records, including 500 generated valid and 500 generated malformed records; native OFF, native ON, and backend direct paths each retained 0 mismatches.
• Upstream suite: 264 passed, 1 skipped in both OFF and ON runs, using a portable fixture for the host-specific printenv --version test.
• Native call proof: fast_dotenv_rs_backend.parse_bindings was observed on the same parse_stream path; backend and official package coexist without a top-level dotenv collision.
• Real consumer benchmark: dotenv_values, load_dotenv, pydantic-settings service/worker settings, repeated loads, and cold startup across small/medium/large fixtures. Semantic hashes matched; 30 p50 metrics classified as 21 win / 8 neutral / 1 loss under a 2% neutral band. The only loss was small cold CLI startup (-1.6675 ms); representative warm savings were 0.1266 ms for small dotenv_values, 0.7477 ms for medium, and 4.9392 ms for large.

Maintainer questions

1. Is an optional backend extra acceptable, with backend wheels maintained outside the default python-dotenv release path?
2. Should activation use the native extra, another name, or a different boundary?
3. Which platform, PyPy/free-threaded, supply-chain, and rollback guarantees are required before merge?

This is a concrete proposal with a merge-ready local patch. The backend is not yet published to PyPI, and the current wheel evidence is macOS arm64 only; those are intentionally open release gates, not hidden claims.