Complete native request backend parity
Author: maurosoriaCreated Aug 13, 2026Updated Sep 11, 2026
The native request backend now has executable CI coverage plus GET scanning, request headers/static cookies, retries and timeouts, advanced filters, HTTP/HTTPS proxies with credentials, replay proxy support, redirect following, bounded normal-response buffering, and CPU-based Tokio worker sizing.
Merged foundation:
- #1584 native option guardrails and real native CI
- #1585 bounded response memory and runtime worker separation
- #1586 HTTP/HTTPS proxy support
- #1587 redirect following
P0: scan control and resource bounds
- Return native results incrementally instead of materializing every task and response for a batch.
- Add cooperative cancellation from Python into Rust.
- Make pause, skip, quit,
--max-time, and--target-max-timeinterrupt an active native batch. - Keep the Tokio runtime and reqwest clients alive across batches instead of recreating them.
- Bound the raw HTTP fallback reader before
read_to_end. - Decode chunked raw responses correctly and preserve measured vs declared response length.
- Add tests for cancellation latency, timeout enforcement, and bounded memory with large/chunked bodies.
P1: request feature parity
- Support non-GET methods and request bodies (
--data,--data-file). - Support Basic, Bearer/JWT, Digest, and NTLM authentication; reject only unsupported auth types.
- Support URL-embedded Basic credentials.
- Support SOCKS4/SOCKS5 and Tor proxies.
- Support client certificates and keys.
- Support
--random-agent,--interface, and--ip. - Support
--max-rateand--delaywithout blocking the runtime. - Match Python cookie/session behavior, not only a static Cookie header.
- Enable and test gzip, deflate, Brotli, and zstd response decoding.
- Define HTTPS behavior for malformed/raw paths; the current raw fallback is HTTP-only.
- Apply retries consistently to raw-path requests.
P2: CLI and scheduling
- Keep
--threadsas HTTP in-flight concurrency and document that runtime workers follow CPU count. - Benchmark whether an expert
--native-workersoverride is useful before exposing it. - Add a bounded native batch/in-flight control only if incremental streaming still needs one.
- Avoid scheduling every path in a chunk up front; maintain a bounded task set.
- Emit results in completion order while preserving path/result association.
- Benchmark the Python dictionary claim/release boundary in full scans. If it is material, add batch claim/commit operations for native chunks and an O(1) ordered claim ledger while preserving dynamic additions and at-least-once session recovery; do not remove claim tracking or move the cursor into Rust without serializable unfinished-work state.
P3: packaging and platform coverage
- Lower the abi3 baseline from Python 3.13 and validate native mode on every Python version supported by dirsearch, or document why the native minimum must differ.
- Run real native integration tests on Windows and macOS in addition to Linux.
- Add
cargo clippyand dependency/security auditing to CI. - Exercise native wheels and standalone artifacts, not only source builds.
- Resolve the missing v0.5.0 tag tracked by #1550.
Acceptance criteria
- Native mode never silently ignores a CLI option.
- Unsupported combinations fail during option parsing with an actionable message.
- Python and native backends share local-server contract tests for request target, method/body, headers, proxy, redirects, auth, cookies, compression, retries, filters, cancellation, and time limits.
- Large scans have bounded task count and bounded per-response memory.
- Benchmarks report runtime workers and HTTP concurrency separately.
Source: maurosoria/dirsearch