windows: double CloseHandle on a torn-down watch aborts the Go runtime (ERROR_INVALID_HANDLE)
Describe the bug
On Windows, readEvents can close the same directory handle twice. Because Windows reissues a closed handle's numeric value to the next caller — and in a Go program the most frequent caller is the runtime itself — the second close destroys a handle the runtime owns, and the process later dies with a fatal error whose stack points nowhere near fsnotify:
runtime: waitforsingleobject wait_failed; errno=6
fatal error: runtime.semasleep wait_failederrno=6 is ERROR_INVALID_HANDLE. I have also seen runtime: setevent failed; errno=6 / fatal error: runtime.semawakeup, and runtime: netpoll failed — three different runtime primitives, one cause: the M-parking event handles the runtime allocates in semacreate (CreateEvent), and the netpoll port, are exactly the objects that land on a handle value fsnotify just freed.
This may explain part of the existing unexplained Windows cluster — #704, #669, #502 — though I have only proven the mechanism below, not that it is behind each of those.
The mechanism
readEvents re-arms the watch unconditionally at the end of the event-processing loop:
if err := w.startRead(watch); err != nil {
w.sendError(err)
}but watch is recovered from the completion packet's OVERLAPPED pointer:
watch := (*watch)(unsafe.Pointer(ov))and that packet may have been queued before a remWatch tore the watch down. remWatch sets watch.mask = 0 and calls startRead, which takes the mask == 0 branch — closing watch.ino.handle and deleting the watch from w.watches. A read that had already completed with data is still sitting in the completion queue (CancelIo does not retract packets that already completed). When it is dequeued, the tail re-arm calls startRead on that dead watch: CancelIo fails on the closed handle, deleteWatch zeroes the mask, and CloseHandle(watch.ino.handle) runs a second time.
There is a second, causally linked bug
sendError receives Close's handshake token from w.done and drops it, where sendEvent puts it back:
select {
case <-w.done: // consumed, never returned
return falsew.done has capacity 1 and carries the reply channel that is Close's only handshake with the reader. If an error is reported while a Close is in flight, the token is lost: the reader falls through to default, parks in GetQueuedCompletionStatus having already consumed the wakeup, and Close blocks forever on <-ch. Its reader thread, watches and handles leak.
The two are linked: the double close is what makes CancelIo fail, which is what calls sendError, which is what loses the token. So the first bug is a frequent cause of the second.
Evidence
Reproduced on a Windows arm64 host (Go 1.26.5), driving repeated Add/Remove of one directory that a second goroutine writes to every 1 ms — the shape of a watcher that subscribes and unsubscribes under live file activity.
Against a build that routes every CloseHandle in backend_windows.go through a tracker, a reported double close carries the same watcher and the same *watch in both stacks:
=== DOUBLE CLOSE of handle 0x1a8 (CloseHandle err=The handle is invalid.) ===
--- first close was here ---
fsnotify.trackedClose(0x1a8)
fsnotify.(*readDirChangesW).startRead(0x62b2f7814100, 0x62b2f7926500) backend_windows.go:450
fsnotify.(*readDirChangesW).remWatch(0x62b2f7814100, ...) backend_windows.go:419
fsnotify.(*readDirChangesW).readEvents(0x62b2f7814100) backend_windows.go:524
--- second close is here ---
fsnotify.trackedClose(0x1a8)
fsnotify.(*readDirChangesW).startRead(0x62b2f7814100, 0x62b2f7926500) backend_windows.go:450
fsnotify.(*readDirChangesW).readEvents(0x62b2f7814100) backend_windows.go:642Measured over 150 rounds of that churn:
| before | after the patch | |
|---|---|---|
| double closes | 2404 | 0 |
hung Closes per 200 watcher rounds |
4 | 0 |
| process handle count | slow upward drift | flat |
And in the application that found this (a GUI toolkit whose file-watching layer does exactly this Add/Remove churn), the Go runtime fatal reproduced within 3 test iterations before the patch and not once in 160 runs after.
Note that the handle count barely moves — this is not handle exhaustion. ERROR_INVALID_HANDLE on a handle the runtime already owns is a stray close.
Versions
- fsnotify v1.9.0, and the code is unchanged on
mainat v1.10.1 — both bugs are still present there. Verified by readingbackend_windows.goonmain; the reproduction ran against v1.9.0. - Go 1.26.5, windows/arm64. Nothing about the mechanism is arm64-specific; that is just where the timing made it frequent.
Fix
PR follows: drop completion packets whose watch is no longer registered, and put the w.done token back in sendError.
Source: fsnotify/fsnotify