[Feature Request] Any plan for official GDR support to enable zero-copy GPU IO ?
As we deploy 3FS in production to accelerate both training and inference, currently, data loading involves a CPU bounce buffer, which consumes PCIe bandwidth and CPU cycles. Enabling GDR support would allow for direct DMA transfers between the storage backend and GPU memory, significantly improving throughput and reducing latency (also reduce cpu cycles).
Before move forward, I'd ask is there an official roadmap or ongoing internal effort to support GDR/GDS in 3FS? I have already developed a working PoC that implements end-to-end GDR support. The implementation touches the following key components:
usrbio: Extended the iov/ior structures to support device pointers and memory registration awareness, also introduced a new create API (and/or extended flag support) to facilitate proper GPU memory handle registration during file operations.- backend impl: Added the necessary logic in both the FUSE and Storage layers to handle the RDMA operations directly to GPU addresses.
If the community are open to this proposal, I would submit a design doc and break down implementations into individual PRs.
Looking forward to your feedback.
Source: deepseek-ai/3FS