Hadoop SDK: positioned reads on one stream are serialized
What happened:
read(long pos, byte[] b, int off, int len) on a JuiceFS Hadoop stream runs one call at a time per stream. Two locks cause it:
- Java: both positioned read methods of
FileInputStreamaresynchronized(sdk/java/src/main/java/io/juicefs/JuiceFileSystemImpl.java:1132). They only readfdandpath; the buffered-read state (buf,position) is untouched, so the lock protects nothing but theclose()ordering. - Go:
File.Preadholds thefs.Filemutex acrossrdata.Read, i.e. across the object storage wait (pkg/fs/fs.go:1334-1343).vfs.FileReader.Readis already safe for concurrent callers and the FUSE path calls it concurrently for one handle (pkg/vfs/vfs.go:783-791).
Random preads do not trigger readahead, so an application that shares one FSDataInputStream across threads (HBase does this per HFile) gets at most one in-flight object storage request per open file when the block cache misses.
What you expected to happen:
Positioned reads on one stream overlap, as PositionedReadable intends and as the FUSE path already does. HDFS DFSInputStream preads are lock-free.
How to reproduce it (as minimally and precisely as possible):
Go test in pkg/fs, no external services:
- Build a
FileSystemon anobject.ObjectStoragewrapper whoseGetcounts in-flight calls and blocks until 4 have arrived or 2 s pass.CacheSize: 0,Readahead: 0. - Write a 6-block file, reopen with
vfs.MODE_MASK_R, callf.Preadfrom 4 goroutines at 4 different block offsets (not block 0, not the last 32 KiB). - Expected max in-flight
Get: 4. Actual: 1.
Java: hold synchronized (in.getWrappedStream()) in one thread and call in.read(pos, buf, 0, len) from another. The read blocks until the monitor is released.
Anything else we need to know?
Environment:
- JuiceFS version (use
juicefs --version) or Hadoop Java SDK version: - Cloud provider or hardware configuration running JuiceFS:
- OS (e.g
cat /etc/os-release): - Kernel (e.g.
uname -a): - Object storage (cloud provider and region, or self maintained):
- Metadata engine info (version, cloud provider managed or self maintained):
- Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
- Others:
Source: juicedata/juicefs