hf-mount

hf-mount

Mount Hugging Face Buckets and repos as local filesystems. No download, no copy, no waiting.

hf-mount start bucket myuser/my-bucket /tmp/data

Also works with any model or dataset repo (read-only):

hf-mount start repo openai/gpt-oss-20b /tmp/gpt-oss

Commands will pick up your HF_TOKEN from the environment, or you can pass it explicitly with --hf-token.

Then use your local folders as usual:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("/tmp/gpt-oss")  # reads on demand, no download step

hf-mount exposes Hugging Face Buckets and Hub repos as a local filesystem via FUSE or NFS. Files are fetched lazily on first read, so only the bytes your code actually touches ever hit the network.

Two backends are available:

• NFS (recommended) -- works everywhere, no root, no kernel extension
• FUSE -- tighter kernel integration, requires root or macFUSE on macOS

Agentic storage: Agents don't require complex APIs or SDKs, they thrive on the filesystem: ls, cat, find, grep, and the power of composable UNIX pipelines.

Install

Homebrew (macOS, Linux)

brew install hf-mount

On macOS, this installs the NFS backend only (hf-mount, hf-mount-nfs). For the FUSE backend on macOS, download the binary manually or build from source — macFUSE is closed-source and not distributable through homebrew-core.

Manual download

Binaries are available on GitHub Releases:

Platform	Daemon	NFS	FUSE
Linux x86_64	hf-mount-x86_64-linux	hf-mount-nfs-x86_64-linux	hf-mount-fuse-x86_64-linux
Linux aarch64	hf-mount-aarch64-linux	hf-mount-nfs-aarch64-linux	hf-mount-fuse-aarch64-linux
macOS Apple Silicon	hf-mount-arm64-apple-darwin	hf-mount-nfs-arm64-apple-darwin	hf-mount-fuse-arm64-apple-darwin

System dependencies (FUSE only)

The NFS backend has no system dependencies. For FUSE:

Linux: sudo apt-get install -y fuse3 (pre-built binaries only need the runtime; building from source also requires libfuse3-dev)

macOS: install macFUSE (brew install macfuse, requires reboot on first install)

Build from source

Requires Rust 1.89+.

# NFS only (no system deps, works everywhere)
cargo build --release --features nfs

# FUSE (requires macFUSE on macOS, fuse3 on Linux)
cargo build --release --features fuse

# All backends
cargo build --release --features fuse,nfs

Binaries: target/release/hf-mount, target/release/hf-mount-nfs, target/release/hf-mount-fuse

Best for / Not for

Best for:

• Loading models and datasets without downloading the full repo
• Browsing repo contents (ls, cat, find) without cloning
• Read-heavy ML workloads (training, inference, evaluation)
• Environments where disk space is limited

Not for:

• General-purpose networked filesystem (no multi-writer support, no cross-node file locking)
• Latency-sensitive random I/O (first reads require network round-trips)
• Workloads that need strong consistency (files can be stale for up to 10 s)
• Heavy concurrent writes from multiple mounts (last writer wins, no conflict detection)
• Editing files with text editors in default (streaming) mode (use --advanced-writes)

Advisory file locks (flock, fcntl POSIX record locks) are supported locally on a single mount on both backends — enough for Python filelock, huggingface_hub, datasets, and similar cache-coordination use cases within one machine. They are not coordinated across multiple clients.

See Consistency model for details.

Usage

Mount a repo (read-only)

# Public model (no token needed)
hf-mount start repo openai/gpt-oss-20b /tmp/model

# Private model
hf-mount start --hf-token $HF_TOKEN repo myorg/my-private-model /tmp/model

# Dataset
hf-mount start repo datasets/open-index/hacker-news /tmp/hn

# Specific revision
hf-mount start repo openai-community/gpt2 /tmp/gpt2 --revision v1.0

# Subfolder only
hf-mount start repo openai-community/gpt2/onnx /tmp/onnx

Mount a Bucket (read-write)

Buckets are S3-like object storage on the Hub, designed for large-scale mutable data (training checkpoints, logs, artifacts) without git version control.

hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket /tmp/data

# Read-only
hf-mount start --hf-token $HF_TOKEN --read-only bucket myuser/my-bucket /tmp/data

# Subfolder only
hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket/checkpoints /tmp/ckpts

Manage mounts

hf-mount status                  # list running mounts
hf-mount stop /tmp/data          # stop and unmount

Logs are written to ~/.hf-mount/logs/. PID files are stored in ~/.hf-mount/pids/.

FUSE backend

By default, hf-mount uses NFS. Pass --fuse for tighter kernel integration (page cache invalidation, per-file metadata revalidation). Requires fuse3 on Linux or macFUSE on macOS.

hf-mount start --fuse --hf-token $HF_TOKEN bucket myuser/my-bucket /mnt/data

Foreground mode

For scripts, containers, or debugging, use the backend binaries directly (they run in the foreground):

hf-mount-nfs repo gpt2 /tmp/gpt2
hf-mount-fuse --hf-token $HF_TOKEN bucket myuser/my-bucket /mnt/data

macOS: launch as a daemon with launchd

To have hf-mount start automatically on login, create a LaunchAgent:

label=co.huggingface.hf-mount

mkdir -p ~/Library/LaunchAgents

cat > ~/Library/LaunchAgents/$label.plist 

    Label
    $label
    ProgramArguments
    
        $HOME/.local/bin/hf-mount-nfs
        repo
        openai/gpt-oss-20b
        /tmp/gpt-oss
    
    RunAtLoad
    
    KeepAlive
    
    StandardOutPath
    /tmp/hf-mount.log
    StandardErrorPath
    /tmp/hf-mount.log

EOF

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/$label.plist

To stop: launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/$label.plist

Unmount

umount /tmp/data                 # NFS or FUSE (macOS)
fusermount -u /tmp/data          # FUSE (Linux)
hf-mount stop /tmp/data          # daemon mounts

Graceful shutdown (SIGTERM / CSI sidecar)

On SIGTERM the sidecar bounds the dirty-data flush (--flush-shutdown-timeout-ms) and disarms the kernel-cache invalidators before draining. This is deliberate: a FUSE_NOTIFY_INVAL_INODE writev issued during teardown can block uninterruptibly in the kernel (waiting on a folio under writeback the exiting daemon can no longer complete), leaving a D-state thread that even exit_group can't reap — an unkillable pod. Disarming the invalidator avoids creating that wedge; the CSI driver aborting the FUSE connection on NodeUnpublishVolume is the kernel-level backstop for any already-in-flight notify.

The same FUSE_NOTIFY_INVAL_INODE writev can also wedge at runtime (not just on shutdown): when the poll loop detects a remote change to a file the app currently has open, a full page-cache invalidation blocks in-kernel on a folio lock held by the app's in-flight read(), which is itself waiting for the daemon — a deadlock. Two guards prevent this: invalidations targeting an inode with open handles drop attributes only (a negative-offset notify the kernel never lets touch pages), and the blocking writev runs on the runtime's blocking pool rather than a core worker, so it can never starve FUSE request servicing. The trade-off: a file with a long-lived open handle won't see remote content updates refreshed in its page cache until the handle closes. On close-and-reopen the kernel revalidates the (attr-only-invalidated) attributes and, via the negotiated FUSE_AUTO_INVAL_DATA, drops the stale pages itself when it sees the new mtime/size — so a stale read only persists if a remote content change preserves both mtime and size on a file that was open at the moment of the change.

Options

Flag	Default	Description
--hf-token	$HF_TOKEN	HF API token (required for private repos/buckets)
--hub-endpoint	https://huggingface.co	Hub API endpoint
--cache-dir	/tmp/hf-mount-cache	Local cache directory
--cache-size	10000000000 (~10 GB)	Max on-disk chunk cache size in bytes
--cache-mode	chunk	Disk cache layer: chunk (xet-core xorb-range cache) or file (whole-file cache keyed by xet hash, avoids chunk-range fragmentation on warm reloads). Mutually exclusive; file disables the chunk cache.
--max-staging-size	0 (unlimited)	Max bytes for advanced-writes staging files before flushed files are garbage-collected (LRU by last-touched). 0 disables GC, so staging files persist as a read-after-write cache. Does not yet cover the HTTP download cache for non-Xet repo files.
--read-only	false	Mount read-only (always on for repos)
--advanced-writes	false	Enable staging files + async flush (random writes, seek, overwrite)
--poll-interval-secs	30	Remote change polling interval (0 to disable)
--poll-listing-concurrency	4	Max concurrent tree-listing requests per poll round. Main knob to throttle load on the Hub /api endpoint; lower it in shared environments where many mounts poll in parallel.
--max-threads	16	Maximum FUSE worker threads (Linux only)
--metadata-ttl-ms	10000	How long file metadata is cached before re-checking (ms)
--metadata-ttl-minimal	false	Re-check on every access (maximum freshness, lower throughput)
--negative-ttl-ms	1000	How long a lookup miss (ENOENT) is remembered before re-probing the Hub. Caps HEAD traffic for missing paths; also the longest a remotely-added file stays hidden from a client that probed it before it existed.
--flush-debounce-ms	2000	Advanced writes: flush debounce delay (ms)
--flush-max-batch-window-ms	30000	Advanced writes: max flush batch window (ms)
--flush-shutdown-timeout-ms	45000	Advanced writes: max time the SIGTERM flush drain may run before abandoning unflushed data to guarantee exit. Must be 0`.
--overlay	false	Treat the mount point as a writable local layer over the remote source. Local files persist on disk; writes are never pushed to the remote. See "Overlay mode" below.

Bounding inode memory

Under workloads that enumerate large trees (a find, a documentation scraper, du -sh), the in-memory inode table can grow without bound: every path the kernel ever looked up stays resident. With --inode-soft-limit N set, two evictors cooperate to keep the table near N:

1. Insert-time evictor (synchronous): before adding a new entry when len() >= N + 256, drop the oldest-touched file/symlink/leaf-directory entries.
   • Polite mode: only entries the kernel has already released (forget-ed). Safe, no FUSE races.
   • Force mode: when above 2 × N and polite found nothing, drop entries even if the kernel still caches the dentry. A racing kernel op sees ENOENT and re-looks up. Dirty files, locally-created dirs/symlinks, and inodes with live file handles are never dropped — the force path preserves all user data.
2. Background LRU sweep (every --lru-sweep-interval-ms): for inodes the kernel has cached but our table doesn't want, send FUSE_NOTIFY_INVAL_ENTRY so the kernel drops its dentry and sends us forget. Bounded to 1024 invalidations per sweep with EAGAIN backoff so we don't flood the notify channel.

Tuning: pick N below what a full-tree enumeration of your bucket would produce. For hf-doc-build/doc-dev with ~20k files, `--