Root stuck in MustScanSubDirs KernelDropped recrawl loop never backs off (14,459 recrawls, done_initial never true)

Summary

A root that enters a MustScanSubDirs KernelDropped recrawl loop recrawls forever. There is no backoff, no rate limit, no retry cap, no give-up threshold, and no error surfaced to anyone — it just consumes a core indefinitely.

One root on my machine recrawled 14,459 times in ~19 minutes (~13/sec) with "done_initial": false the entire time. It never once completed its initial crawl. Three such roots pinned all 10 cores of an M1 Max at load average 254.

Environment

Roots were ordinary TypeScript project directories, ~7,700 files each. Not monorepos.

Evidence

watchman debug-status:

{
  "path": "/Users/<me>/<project>",
  "uptime": 1133,
  "recrawl_info": {
    "count": 14460,
    "should-recrawl": true,
    "completed": null,
    "started": -59,
    "reason": "MustScanSubDirs KernelDropped"
  },
  "watcher": "fsevents",
  "done_initial": false,
  "queries": [],
  "cancelled": false,
  "crawl-status": "re-crawling for 59 ms"
}

count: 14460 against uptime: 1133 seconds. done_initial: false. queries: [] — no client was even connected.

The log is this, flat, forever:

[notify <root>] MustScanSubDirs KernelDropped: scheduling a tree recrawl
[io <root>] PERF: {... "description": "full-crawl"}
[io <root>] recrawl complete
[notify <root>] MustScanSubDirs KernelDropped: scheduling a tree recrawl

The interval never grows. Recrawl 14,000 is scheduled exactly as eagerly as recrawl 1.

While this ran: fseventsd 164%, watchman 103%, filecoordinationd 53%, plus unrelated software that merely consumes file events (a password manager, an endpoint-security extension) dragged to 100%+. Load average 254, 122 runnable processes. After watch-del-all && shutdown-server: load 11.7, 6 runnable.

The Homebrew LaunchAgent ships Nice=-5, so this is scheduled above normal user processes.

The Bug

To be clear which layer I am reporting: the kernel dropping events is not a watchman bug. KernelDropped means the events were lost before watchman saw them, and recrawling to resync is the correct response.

The defect is that the recovery mechanism has no limiting behaviour of any kind:

• No backoff. Flat interval across 14,459 consecutive failures.
• No cap. Nothing stops at 10, 100, or 10,000 attempts.
• No health state. A root failing every recrawl is treated identically to a healthy one.
• No signal. The warning string in debug-status is the only indication, and you have to know to go looking. Nothing is logged at error level, no query fails, nothing tells the user their machine is being eaten (apart from heat and the fans taking off like a jumbo jet).

A root that has attempted 14,459 recrawls without once completing its initial crawl is doing zero useful work while consuming a full core. That state is detectable in a single comparison — recrawl_count high, done_initial still false — and nothing acts on it.

The troubleshooting docs call frequent recrawls "undesirable" and stop there. No circuit breaker exists.

Suggested Directions

1. Exponential backoff on consecutive failed recrawls — if N recrawls complete without done_initial becoming true, grow the delay instead of immediately rescheduling.
2. A give-up threshold — mark the root unhealthy, stop recrawling until a client explicitly re-establishes the watch, and return the error in query responses.
3. Log at error level once a root crosses some recrawl count, so this is discoverable without running debug-status.
4. Distinguish KernelDropped from UserDropped in the warning text. Different causes, different remedies — but the current message sends everyone to fsevents_latency, which cannot help KernelDropped.

Reproduction

I do not have a minimal reproduction; the kernel-side drop depends on machine-wide FSEvents pressure I have not isolated.

Most promising lead: facebook/metro#814 ("Metro seems to leave orphaned watchman watches frequently"). That issue is about clients failing to release watches and says nothing about KernelDropped — I am not claiming it causes the drops. But orphaned roots plausibly supply the precondition: client-less streams nobody monitors, restored at every boot from the statefile. Accumulate several via normal jest/metro use, apply FSEvents pressure, watch recrawl_count.

Note that regardless of what triggers the first drop, the unbounded retry is independently fixable.

Happy to run diagnostics or supply full logs. This has recurred on three unrelated projects on this machine, so I can likely reproduce on request.

Possibly Related

• #1157, #981 — same loop shape, but UserDropped rather than KernelDropped
• #1306 — different failure (process explosion), also macOS
• facebook/metro#814 — orphaned watches; see Reproduction

Thank you!