Slow read on a file blocks opening the same file until the read completes

Bug Report

A slow non-cached READ on a file blocks all subsequent OPENs of the same file for the entire read duration (FileNode Main resource held shared across the user-mode round-trip)

On a WinFsp volume, while a single non-cached READ on a file is in flight in the user-mode file system, any second CreateFile/open of the same file blocks until that read completes. For a file system whose backend is slow (network/cloud storage, on-demand decompression, etc.), a single multi-second read can freeze every open of that file for multiple seconds.

The blocking is per-FileNode (only the file being slowly read is affected; other files open normally), and the operation that is actually blocked is the OPEN, not other reads.

Root cause appears to be that FspFsvolReadNonCached acquires the FileNode Main resource shared and transfers ownership of that lock to the asynchronous Request (FspFileNodeSetOwner), so the shared lock is held across the entire user-mode read round-trip. Meanwhile the create/open completion path (FspFsvolCreateTryOpen) needs the same Main resource exclusive, which cannot be granted while the read holds it shared.

I could not find a prior issue for this specific read-vs-open case. It is a natural follow-on to #291 (which addressed read-vs-read serialization); I want to check whether this read-vs-open serialization is intended, or whether the Main-shared hold across the round-trip (or the exclusive acquire on the open-completion side) could be relaxed.

How to Reproduce

1. Stock memfs, no custom code

memfs already ships the slow-IO options used in #291, so this needs no custom file system.

1. Mount memfs with a large per-read delay so a read stays in flight long enough to race an open. -M is the maximum slow-IO delay in ms:

   复制

   memfs-x64 -M 10000 -F FOO -m X:

2. Create a file large enough that reads go down the non-cached path (probe with FILE_FLAG_NO_BUFFERING, as in #291, to avoid the reads being satisfied from the system cache):

   复制

   X:\foo.raw

3. From one thread: open foo.raw with FILE_FLAG_NO_BUFFERING and issue a read at some offset. Because of -M 10000 this read sits in the user-mode FS for up to ~10s.

4. While that read is in flight, from a second thread: call CreateFile("X:\foo.raw", ...) (a plain open) and time it.

2. Minimal pure-C reproduction (no .NET / no language binding)

I also built a minimal pure-C WinFsp file system (official SDK only, zero binding layer) to rule out any user-mode binding as the cause. It mounts read-only over a UNC prefix (no drive letter), serves a video.bin, and makes a tail read sleep for a fixed duration; a separate probe opens the same file and times the open vs. the read separately.

Representative numbers (tail read delay = 3000ms; probe starts ~200ms into the read):

Takeaways from the pure-C repro:

• It reproduces with zero binding code, so the serialization is in the kernel FSD, not any user-mode wrapper.
• The blocked operation is specifically the OPEN (~2806ms). Once open returns, the read on that handle is ~0.1ms.
• It is per-FileNode: a different file opens in ~1.2ms during the same window.
• STATUS_PENDING on the read callback does not help — the kernel holds the FileNode lock for the IRP lifetime regardless of whether the user-mode read is synchronous or pending.
• Turning on the real kernel cache (FileInfoTimeout = FspTimeoutInfinity32) does not help the cold open case: the first read of an offset is necessarily a cache miss, holds the lock, and the concurrent open still blocks. (A subsequent cached read of an already-returned offset does go down the in-function-release fast path and does not hold across a round-trip — but that does not cover the first read of any offset.)

Full repro sources (single-file pure-C FS + a small .NET probe, no drive letter — mounts over a UNC prefix): https://github.com/hooyao/ZipDrive/tree/e81c7c6c8eba985a7ff860e68d3937fb7d0c2f71/diag/winfsp-c-repro

• repro.c — minimal read-only WinFsp FS; a tail read of video.bin (offset ≥ 32 MB) sleeps for a fixed duration.
• probe.cs — starts the slow tail read, then times a concurrent same-file OPEN vs. read and an other-file OPEN.
• build.cmd — build (delay-loads winfsp-x64.dll via FspLoad).

static NTSTATUS Read(FSP_FILE_SYSTEM *FileSystem,
    PVOID FileContext, PVOID Buffer, UINT64 Offset, ULONG Length, PULONG PBytesTransferred)
{
    REPRO *r = (REPRO *)FileSystem->UserContext;

    if (Offset >= FILE_SIZE)
        return STATUS_END_OF_FILE;

    BOOLEAN isVideo = (FileContext == CTX_VIDEO);
    BOOLEAN isTail = (Offset >= TAIL_START);
    BOOLEAN slow = isVideo && (isTail || r->SlowAll) && r->TailDelayMs > 0;

    if (slow)
    {
        if (r->UsePending)
        {
            /* Async model: return STATUS_PENDING, complete from a worker thread.
             * The FileNode lock is STILL held by the FSD until SendResponse — this
             * tests whether pending vs blocking changes the same-file serialization. */
            PENDING_READ *pr = (PENDING_READ *)malloc(sizeof *pr);
            if (0 != pr)
            {
                pr->FileSystem = FileSystem;
                pr->Buffer = Buffer; pr->Offset = Offset; pr->Length = Length;
                pr->Hint = FspFileSystemGetOperationContext()->Request->Hint;
                pr->DelayMs = r->TailDelayMs;
                HANDLE h = CreateThread(0, 0, PendingReadThread, pr, 0, 0);
                if (0 != h) { CloseHandle(h); return STATUS_PENDING; }
                free(pr); /* fall through to blocking on failure */
            }
        }
        /* Blocking model. */
        Sleep(r->TailDelayMs);
    }

    UINT64 endOff = Offset + Length;
    if (endOff > FILE_SIZE) endOff = FILE_SIZE;
    ULONG xfer = (ULONG)(endOff - Offset);
    DoFill(Buffer, Offset, xfer);
    *PBytesTransferred = xfer;
    return STATUS_SUCCESS;
}

The Sleep / pending-thread delay stands in for any slow backend (network fetch, on-demand decompression). What matters to the FSD is only that the read callback does not return promptly, so the FileNode Main lock stays held across the round-trip.

Behaviors

Expected: a slow read of a file should not block an open (CreateFile) of the same file. Reads and opens on the same FileNode should be able to proceed concurrently (as reads already do among themselves since #291).

Actual: the second CreateFile blocks for ~ the remaining read delay (seconds). An open of a different file returns in ~1ms during the same window. Once the slow read returns, the blocked open completes immediately (~sub-ms), and a read on the freshly opened handle is also immediate.

Where the lock lifetimes come from (code references, master @ bdab233e)

Read side — Main (+ PagingIo) acquired shared, ownership transferred to the async Request, released only in the request-fini after the user-mode read returns:

src/sys/read.c, FspFsvolReadNonCached:

356:  Success = DEBUGTEST(90) &&
357:      FspFileNodeTryAcquireSharedF(FileNode, FspFileNodeAcquireFull, CanWait);  // Main + PagingIo, shared
...
444:  FspFileNodeSetOwner(FileNode, Full, Request);   // transfer lock ownership to the async Request
445:  FspIopRequestContext(Request, RequestIrp) = Irp;
...
461:  return FSP_STATUS_IOQ_POST;                     // IRP posted to the user-mode queue

Released only when the request finishes (i.e. after the user-mode read returns):

// src/sys/read.c  FspFsvolReadNonCachedRequestFini
671:  FspFileNodeReleaseOwner(FileNode, Full, Request);

• FspFileNodeAcquireFull = Main (1) + PagingIo (2) — src/sys/driver.h:1632-1634.
• FspFileNodeTryAcquireSharedF uses ExAcquireResourceSharedLite on both — src/sys/file.c:432,440.
• FspFileNodeSetOwnerF uses ExSetResourceOwnerPointer, so the lock survives past the dispatch thread — src/sys/file.c:544-549.

Open side — create completion needs Main exclusive (non-wait try, then repost/retry until it succeeds):

src/sys/create.c, FspFsvolCreateTryOpen:

1325:  Success = DEBUGTEST(90) &&
1326:      FspFileNodeTryAcquireExclusive(FileNode, Main) &&
1327:      FspFsvolCreateOpenOrOverwriteOplock(Irp, Response, &Result);
1328:  if (!Success)
1329:  {
...
1338:      FspIopRetryCompleteIrp(Irp, Response, &Result);   // repost to the Retried queue; retried on each transact
1339:      return Result;

FspFileNodeTryAcquireExclusive(N, Main) expands with Wait = FALSE (src/sys/driver.h:1906), so the open-completion does not block on the resource directly — it fails the try and reposts the IRP to the Retried queue (FspIopRetryCompleteIrp → FspIoqRetryCompleteIrp, src/sys/ioq.c:693), which is re-driven on each FSP_FSCTL_TRANSACT. The net effect from the application's perspective is that the CreateFile stays pending for the whole read duration.

Since Main shared (held by the read) and Main exclusive (wanted by the open) are incompatible, the open cannot make progress until the read releases at read.c:671.

Why this matters (impact)

For a RAM-backed FS the read returns instantly and the window is invisible. But for file systems whose reads are genuinely slow, the window is the full read latency, and it is very visible:

• Cloud/remote backends (e.g. an FS backed by Amazon S3 or similar): a single large object read of a few seconds blocks every open of that object.
• A concrete, easy-to-hit case: a folder of images and videos on such an FS, browsed with the Windows Photos viewer. Opening an image triggers thumbnail generation for the nearby video, which issues a slow read of that video. While that read is in flight, the app's own CreateFile on the same video (to display/preview it) blocks — and the whole viewer UI freezes for as long as the read takes.

Because the stall is on the open of a file that is already open by the slow reader, and because FspFileNodeTrySetFileInfoAndSecurityOnOpen early-exits when OpenCount > 1 (src/sys/file.c:1916), in this particular scenario the exclusive Main that the open waits seconds for is acquired only to perform a metadata update that it then skips.

Questions / discussion

1. Is this serialization intended? This is a natural follow-on to #291 (where FspFsvolReadNonCached was intentionally changed from exclusive to shared so concurrent reads could proceed). That change addressed read-vs-read; this report is about read-vs-open on the same file.

2. Does the read need to hold Main across the whole user-mode round-trip? The Main shared hold seems to protect the file's size/metadata view against a concurrent writer/SetInfo (which takes Main exclusive). Would it be correct to hold Main only long enough to snapshot what is needed and post the request, and rely on PagingIo (also held) — or another mechanism — for the duration of the round-trip? (In #291 you sketched ExAcquireSharedWaitForExclusive / release-and-reacquire to give exclusive waiters priority; a similar idea might apply to letting an open's exclusive acquire preempt an in-flight read that only holds Main to guard a request it has already posted.)

3. Could the open-completion side avoid needing Main exclusive when it will not mutate FileNode metadata (e.g. the OpenCount > 1 early-exit case), so a second open of an already-open file is not gated on the in-flight read?

I'm not attached to any particular fix — I mainly want to understand whether the read holding Main across the round-trip is a hard requirement, and whether same-file open latency during a slow read is something you'd consider addressing. Thanks for WinFsp.

Environment

• OS version and build: Windows 11 (also observed on Windows 10)
• WinFsp version and build: master @ bdab233e (v2.2B1-4-gbdab233e); also reproduced on the 2.x release line
• Guard strategy: default FINE (unchanged)