FileWatcher idle-eviction leaks watcher threads under high repo churn — thread-cap exhaustion with EAGAIN panics (0.22.1)

UPSTREAM ISSUE DRAFT — DO NOT FILE WITHOUT OPERATOR APPROVAL

Repo: codesearch (v1.0.213+1, macOS arm64 pinned artifact) Title: serve: watcher-thread leak across idle-evict/reopen cycles exhausts macOS per-task thread cap (WouldBlock panic storm)

Summary

Long-running codesearch serve hubs accumulate OS threads monotonically under the idle reaper's evict→reopen churn until they hit kern.num_taskthreads (6144 on macOS). Every subsequent thread spawn fails with Os { code: 35, WouldBlock } and panics the calling tokio worker at tantivy-0.22.1/src/directory/file_watcher.rs:65 (.expect("Failed to spawn meta file watcher thread")). The process keeps its port open, so external liveness probes see a healthy server while searches intermittently panic.

Measured evidence (fleet of 3 hubs, one machine, 2026-09-13)

• 18-repo hub, 21h15m uptime: 6,074 threads (98.9% of cap); later found dead with repeated restart lines in its log.
• 61-repo hub: 6,145 threads at census (prior session) with 1,382 WouldBlock panics in one log file; fresh restart of the same hub: 544 threads at 24 min (~9/repo steady state).
• Control hub, 3h19m: 1,773 threads, 0 panics.
• Reaper churn census (machine-wide log): 475 evictions on Sep 12, 195 by 12:00 Sep 13; default idle timeout ≈30 min (open→evict deltas 30.3/30.5 min).
• fd limit ruled out: job carries NumberOfFiles=65536.

Mechanism (source-verified against tantivy 0.22.1)

1. FileWatcher::spawn() creates one thread-tantivy-meta-file-watcher OS thread per MmapDirectory (guarded one-per-instance), spawned on first watch().
2. The thread exits ONLY when the FileWatcher is dropped (Drop sets state=2; loop condition state == 1).
3. serve's eviction path logs (FSW stopped, DB closed) and its notify-based watcher stops cleanly — but thread counts prove the tantivy watcher thread (and/or notify fsevents thread) survives eviction: ~337 threads/repo after 21h vs ~9/repo fresh. The MmapDirectory (or something holding it) is not dropped on evict.
4. Each lazy reopen after eviction spawns a NEW watcher thread → leak is per evict→reopen cycle, multiplied by query-driven churn.

Knob observations

• CODESEARCH_REPO_IDLE_TIMEOUT_SECS appears in the binary and the reaper is active, but a controlled A/B (single-repo scratch registry, 4 arms, 45-180s windows, reaper ticks ~60s) showed no eviction at value=5 while the default-timeout instance evicts freely — the env value appears not honored (or clamped) at serve runtime. Please document semantics.
• serve --help exposes no watch-disable or watcher-budget flag.

Asks

1. Fix the leak: drop the tantivy MmapDirectory/FileWatcher (and notify watcher) on idle eviction; verify the thread-tantivy-meta-file-watcher thread exits (it polls at POLLING_INTERVAL, so exit is ≤1 tick after drop).
2. Honor + document CODESEARCH_REPO_IDLE_TIMEOUT_SECS (or remove the dead env read).
3. Degrade gracefully at spawn failure: replace .expect() with a logged fallback (unwatched index + poll-on-query) so a thread-cap incident is a warning, not a panic storm on tokio workers.
4. Consider a bounded/shared watcher: one FSEvents watcher (notify crate supports recursive multi-path) or a poll-multiplexer instead of per-index threads; alternatively expose a --max-watched-repos budget with LRU watch rotation.

Environment

macOS (darwin, arm64), codesearch v1.0.213+1, launchd-managed serve hubs per workspace, 18-61 registered repos per hub, LMDB + tantivy FTS stores, heavy intermittent MCP query traffic.