#1588·dirsearch

Complete native request backend parity

Author: maurosoriaCreated Aug 13, 2026Updated Sep 11, 2026

The native request backend now has executable CI coverage plus GET scanning, request headers/static cookies, retries and timeouts, advanced filters, HTTP/HTTPS proxies with credentials, replay proxy support, redirect following, bounded normal-response buffering, and CPU-based Tokio worker sizing.

Merged foundation:

  • #1584 native option guardrails and real native CI
  • #1585 bounded response memory and runtime worker separation
  • #1586 HTTP/HTTPS proxy support
  • #1587 redirect following

P0: scan control and resource bounds

  • Return native results incrementally instead of materializing every task and response for a batch.
  • Add cooperative cancellation from Python into Rust.
  • Make pause, skip, quit, --max-time, and --target-max-time interrupt an active native batch.
  • Keep the Tokio runtime and reqwest clients alive across batches instead of recreating them.
  • Bound the raw HTTP fallback reader before read_to_end.
  • Decode chunked raw responses correctly and preserve measured vs declared response length.
  • Add tests for cancellation latency, timeout enforcement, and bounded memory with large/chunked bodies.

P1: request feature parity

  • Support non-GET methods and request bodies (--data, --data-file).
  • Support Basic, Bearer/JWT, Digest, and NTLM authentication; reject only unsupported auth types.
  • Support URL-embedded Basic credentials.
  • Support SOCKS4/SOCKS5 and Tor proxies.
  • Support client certificates and keys.
  • Support --random-agent, --interface, and --ip.
  • Support --max-rate and --delay without blocking the runtime.
  • Match Python cookie/session behavior, not only a static Cookie header.
  • Enable and test gzip, deflate, Brotli, and zstd response decoding.
  • Define HTTPS behavior for malformed/raw paths; the current raw fallback is HTTP-only.
  • Apply retries consistently to raw-path requests.

P2: CLI and scheduling

  • Keep --threads as HTTP in-flight concurrency and document that runtime workers follow CPU count.
  • Benchmark whether an expert --native-workers override is useful before exposing it.
  • Add a bounded native batch/in-flight control only if incremental streaming still needs one.
  • Avoid scheduling every path in a chunk up front; maintain a bounded task set.
  • Emit results in completion order while preserving path/result association.
  • Benchmark the Python dictionary claim/release boundary in full scans. If it is material, add batch claim/commit operations for native chunks and an O(1) ordered claim ledger while preserving dynamic additions and at-least-once session recovery; do not remove claim tracking or move the cursor into Rust without serializable unfinished-work state.

P3: packaging and platform coverage

  • Lower the abi3 baseline from Python 3.13 and validate native mode on every Python version supported by dirsearch, or document why the native minimum must differ.
  • Run real native integration tests on Windows and macOS in addition to Linux.
  • Add cargo clippy and dependency/security auditing to CI.
  • Exercise native wheels and standalone artifacts, not only source builds.
  • Resolve the missing v0.5.0 tag tracked by #1550.

Acceptance criteria

  • Native mode never silently ignores a CLI option.
  • Unsupported combinations fail during option parsing with an actionable message.
  • Python and native backends share local-server contract tests for request target, method/body, headers, proxy, redirects, auth, cookies, compression, retries, filters, cancellation, and time limits.
  • Large scans have bounded task count and bounded per-response memory.
  • Benchmarks report runtime workers and HTTP concurrency separately.