Plan a persistent Rust-owned native scan session

Context

This is a companion to #1588, not a replacement for it.

• #1588 tracks native request/transport feature parity and platform coverage.
• This issue tracks the architectural migration from a Python-driven NativeFuzzer batch bridge to a persistent Rust-owned scan session.
• #1675 keeps the current Python claim ledger efficient. It is an interim improvement, not the target architecture.

A persistent native wordlist alone is insufficient. Today Rust generates a Vec<String> that is converted into a Python list; Python then owns dictionary cursors and claims, constructs chunks, invokes Rust for each batch, wraps returned responses, applies important scan policy, runs callbacks, adds crawl/backup/recursion candidates, and serializes the unfinished work. Keeping only the wordlist in Rust would remove one allocation/copy boundary while retaining most Python/Rust crossings and lifecycle complexity.

This is a planning issue. Do not implement it as one rewrite or begin the cutover until the Phase 0 contracts and performance gates are accepted.

Target boundary

Python control plane remains responsible for

• CLI/config parsing, validation, and target enumeration;
• selecting threaded, async, or native execution;
• terminal presentation, report writers, and response-artifact file I/O;
• durable session-file I/O around a versioned native snapshot;
• compatibility engines and public Python integration points.

A per-target Rust NativeScanSession should own

• native wordlist generation, the complete template grammar, encoding/newline handling, ordered deduplication, and exclusions;
• the candidate store and its pending/in-flight/completed/dynamic states;
• Tokio runtime, clients, bounded scheduling, retries, rate/delay/deadlines, and response capture limits;
• wildcard probes, auto-calibration, dynamic-content normalization, fingerprints, and the complete match/filter pipeline;
• crawl, backup, and recursion feedback into the same candidate store;
• progress/error counters and skip-on-status/stop policy that depend on individual requests;
• pause, resume, cancel, target shutdown, and recoverable snapshots.

The steady-state boundary should be small and typed

Illustrative API (exact PyO3 shape is a design decision):

NativeScanSession(config, wordlist_sources)
start()
poll_events(max_events, timeout_ms) -> list[NativeEvent]
pause(deadline_ms) -> NativeSnapshot
resume()
snapshot(deadline_ms) -> NativeSnapshot
cancel(deadline_ms)
close(deadline_ms)

Python should receive bounded structured events such as match, error, progress, discovered directory, response artifact, and lifecycle state. It should not claim paths, construct scan chunks, receive every filtered response, or wrap every Rust response merely to make an internal scan decision.

Events need candidate/session IDs plus monotonic sequence, config_revision, and generation fields so late effects from cancellation, resume, or target transitions can be rejected. Rust must never invoke Python callbacks while holding an engine lock; network work and waits must not hold the GIL.

Migration plan

Phase 0 — freeze contracts and set decision gates

• Build one deterministic local fixture suite exercised by threaded, async, and native engines: normal/raw request targets, redirects, compression/charset, cookies/auth/proxy, retries, timeouts, cancellation, wildcard responses, filters, crawl, backup discovery, recursion, and session resume.
• Record which differences are intentional capabilities and which are bugs. Define a stable error taxonomy rather than matching backend-specific exception text.
• Specify the session state machine, event schema, filter/regex semantics, snapshot versioning, and report-delivery guarantees before moving ownership.
• Benchmark the current full scan, including CPU, peak memory, Python/Rust crossings, time-to-first-result, pause/cancel latency, and end-to-end throughput. Use docs/native-backend-benchmarks.md and #1675 as baselines, while keeping direct-client and full-dirsearch results separate.
• Agree on a meaningful end-to-end improvement threshold. Stop the migration if the prototype only improves isolated Rust microbenchmarks while increasing complexity or regressing scan correctness.

Phase 1 — separate the Rust core from adapters

• Split dependency-light engine/state types from the PyO3 adapter so the scan lifecycle is testable directly in Rust.
• Replace loosely related constructor parameters with an immutable typed configuration and explicit capability negotiation.
• Give errors stable codes and structured context for Python presentation.
• Retain current generate_wordlist()/scan() entry points as compatibility shims during migration.

Phase 2 — persistent native candidate store

• Keep the ordered deduplication structure in Rust instead of consuming it into a Python list.
• Port all wordlist behavior, not only %EXT%: prefixes/suffixes, categories, date/subject/CRUD tokens, case transforms, encoding detection/conversion, CRLF/LF handling, comments, exclusions, invalid-line behavior, and malformed-input errors.
• Model base items, dynamic additions, cursor/order, claims, attempts, and completion explicitly.
• Preserve deterministic ordering and O(1) membership/claim operations.
• Define snapshot/restore v2 before removing Python claims. Legacy Python sessions must remain loadable through a documented migration path.

Phase 3 — bounded streaming scheduler and lifecycle

• Replace scan(Vec<String>) -> Vec<Result> batches with a long-lived bounded producer/consumer loop.
• Keep both in-flight request tasks and the Python-facing event queue bounded; apply an explicit backpressure policy.
• Emit results in completion order with stable candidate IDs. Aggregate progress rather than crossing into Python for every filtered miss.
• Distinguish pause, cancel, skip target, deadline, and fatal shutdown. Define whether in-flight requests drain or abort for each transition.
• Fence late completions by generation. A cancelled batch must not silently discard already completed, undelivered results.
• Ensure close() releases clients, proxy transports, tasks, channels, and runtime-owned resources even if one cleanup step fails.

Phase 4 — move calibration and filtering

• Port wildcard canary setup and response-profile creation.
• Port dynamic-content parsing, reflected-path/redirect normalization, similarity/content-type/length checks, extra calibration samples, threshold fingerprints, blacklists, and distinct-content safeguards.
• Port every legacy and advanced match/filter combination with the same precedence and evidence/reason output.
• Make a deliberate regex-dialect compatibility decision and reject unsupported expressions at startup rather than changing their meaning silently.
• Native mode must no longer use the Python requester to calibrate a target once the native session starts.

Phase 5 — close the dynamic discovery loop

• Feed crawl, backup, and recursion candidates directly into the Rust candidate store.
• Preserve first-effective-base and same-origin URL resolution, srcset, query/fragment/backslash behavior, extension filtering, excluded subdirectories, maximum depth, already-visited directories, and redirect recursion rules.
• Move initial root crawling and per-match crawling behind the same contract so native behavior does not depend on a separate Python requester.
• Treat replay-proxy delivery and response-store/reporting as explicit output side effects. Decide whether Rust performs them or emits a durable command/event for Python, and lock the intended semantics with tests.
• Move consecutive-error, skip-on-status, maximum-time, and target-transition decisions that depend on the request stream into the session state machine.

Phase 6 — complete request parity via #1588

Use #1588 as the required capability checklist: methods/bodies, authentication, embedded credentials, cookies, HTTP/HTTPS/SOCKS/Tor proxies, client certificates, interface/IP selection, user-agent behavior, rate/delay, redirect history, retries, raw targets, decompression, charset handling, and platform/package coverage.

Do not claim the Rust-owned engine is interchangeable while a supported CLI option is silently ignored. Until parity is complete, unsupported combinations must fail during startup with an actionable error.

Phase 7 — durable sessions and result delivery

• A native snapshot must include configuration identity, target/base path, ordered candidate state, dynamic additions, attempts/in-flight retries, completed and delivered-result IDs, recursion frontier, calibration profiles/counters, progress, event sequence, and generation.
• Capture either after a bounded pause/drain or through a defined consistent-snapshot protocol. Do not hold Rust engine locks while Python writes files.
• Have SessionStore atomically persist the opaque versioned snapshot plus a small human-readable envelope.
• Preserve at-least-once work recovery with no unprocessed path loss. Bound and deduplicate replayed result events by stable IDs after crashes.
• Specify forward-version rejection, legacy import, and downgrade behavior. Track native/Python package version compatibility with #1640.
• Reporting can remain in Python initially, but writer failure or a full event queue must not deadlock the engine or lose acknowledged results.

Phase 8 — cutover, rollback, and cleanup

• Introduce the new session behind an experimental/internal switch while the current bridge remains available.
• Run differential CI and a canary period before making it the native default.
• Keep rollback possible without converting an in-progress v2 snapshot into corrupt legacy state.
• Only then remove NativeFuzzer chunk claim/release, native-mode Python wordlist materialization, per-response wrapper work used solely for filtering, duplicate filter pushdown configuration, and compatibility shims.
• Do not change threaded or async ownership as part of this migration.

Phase 9 — platform, packaging, and operations

• Test the native lifecycle on Linux, Windows, and macOS; cover wheels, source installs, PyInstaller, and the native-rust Docker stack.
• Add Rust format, lint, unit/integration, dependency, and artifact smoke checks appropriate to release builds.
• Validate both supported Python ABI/version behavior and cross-version session rejection/migration.
• Document runtime-worker versus HTTP-concurrency tuning and expose another knob only after benchmarks justify it.

Required concurrency and persistence invariants

• Every candidate is Pending -> InFlight -> Completed, or is atomically requeued on a recoverable stop.
• A candidate generation produces at most one acknowledged completion; late events from an old generation cannot mutate the current scan.
• A pause reports success only after a recoverable snapshot exists.
• All task sets, response bodies, raw readers, event queues, and retained evidence are bounded.
• All deadlines use a monotonic clock and every shutdown path has a bounded wait.
• Cancellation, snapshot, result delivery, and dynamic insertion races have deterministic ownership and tests.
• No Python callback runs while Rust locks are held, and Rust network execution does not hold the GIL.
• Reporter failure cannot block transport cleanup; backpressure and event acknowledgement are explicit.

Verification matrix

• Rust unit/property tests: ordered dedupe, candidate transitions, dynamic insertion, snapshot round trips, invalid state transitions, filter/calibration fixtures, and encoding/newline/template cases.
• Cross-engine contracts: identical local-server observations for supported behavior across threaded, async, and native.
• Race/fault tests: barriers at queued/connect/read/filter/event-delivery/snapshot states; cancel/pause/target-switch and injected channel/report/session failures.
• Resume tests: legacy import, v2 round trip, crash at every persistence boundary, no path loss, bounded replay, and incompatible-version diagnostics.
• Performance gates: median full-scan throughput/CPU/memory, crossings per candidate, time-to-first-result, cancellation latency, and large/dynamic wordlists. Direct Rust client numbers remain secondary evidence.
• Release tests: Linux/Windows/macOS plus wheel, standalone, and Docker smoke tests.

Acceptance criteria

• Native scans do not materialize the complete generated wordlist in Python.
• Python performs no per-path or per-chunk claim/release and receives no per-miss response solely for filtering.
• After NativeScanSession.start(), no Python HTTP requester is used for calibration, crawling, backup discovery, recursion, or request-policy decisions.
• Supported match/filter/calibration and dynamic-discovery results satisfy the shared deterministic contracts.
• Pause/cancel/target transitions are bounded and cannot lose unprocessed candidates or acknowledged matches.
• Resume provides at-least-once work recovery with stable deduplication of delivered results.
• #1588 is complete for the advertised native feature set, or unsupported options fail before scanning.
• The measured end-to-end benefit meets the Phase 0 gate without an unacceptable memory, latency, correctness, or maintenance regression.
• The old native bridge is not removed until rollback, legacy-session, packaging, and platform gates pass.

Suggested reviewable PR sequence

1. Characterization fixtures, event/state specification, and benchmark counters; no behavior change.
2. Rust candidate store plus property tests, unused by production.
3. Versioned native snapshot/restore plus legacy import tests.
4. Bounded event streaming behind an experimental switch.
5. Pause/cancel/skip/deadline state machine and race tests.
6. Calibration and filter pipeline, landed in independently reviewable slices.
7. Crawl/backup/recursion feedback loop, also sliced by feature.
8. Remaining transport capabilities as focused PRs linked to #1588.
9. Durable report/session acknowledgement integration.
10. Cross-platform cutover, canary, rollback validation, and only then bridge cleanup.

Each PR must be independently testable and reversible. This issue should stay not ready until Phase 0 is approved; it is a migration roadmap, not authorization for a large implementation PR.