RustFS 1.0.0: sustained multipart PUT memory growth causes host OOM with only 5 UploadPart requests in flight

Supported storage confirmation

• I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.

Describe the bug

During sustained S3 multipart PUT workloads on RustFS 1.0.0, the RustFS process memory usage continuously increases until the host exhausts both RAM and swap.

The issue is reproducible with only 5 concurrent UploadPart requests (5 Warp clients, each using --concurrent=1 and --part.concurrent=1, with 10 MiB parts).

On an 8 GiB VM, RustFS grows from a few hundred MiB to approximately 6.8 GiB RSS, consumes the entire 3.8 GiB swap, and is eventually killed by the Linux global OOM killer. Docker then restarts the container with exit code 137.

A separate test on a 16 GiB VM also resulted in the Linux OOM killer terminating RustFS at approximately 15 GiB anonymous RSS.

Short 2-minute multipart tests complete normally, while sustained workloads show progressive memory growth and eventually lead to OOM.

Steps to reproduce

Reproduction Steps: RustFS 1.0.0 Memory Leak Under Sustained Multipart PUT Traffic

1. Environment Setup Run RustFS 1.0.0 via Docker Compose on a single RHEL VM using four XFS-backed data directories.

2. Client Preparation Deploy and connect five distributed MinIO Warp clients.

3. Execute Multipart PUT Workload Run the following workload:

   bash复制

   warp multipart-put \
     --parts=100 \
     --part.size=10MiB \
     --concurrent=1 \
     --part.concurrent=1 \
     --duration=5m \
     --warp-client=<client1>:7761,<client2>:7761,<client3>:7761,<client4>:7761,<client5>:7761 \
     --host=<rustfs-host>:9000 \
     --bucket=warp-benchmark-bucket \
     --access-key=<ACCESS_KEY> \
     --secret-key=<SECRET_KEY> \
     --tls

With five Warp clients, this produces a reported PUTPART concurrency of 5. Each multipart upload consists of:

• 100 parts
• 10 MiB per part
• approximately 1 GiB per completed multipart object

1. Monitor System and Container Metrics Monitor RustFS memory usage:

   bash复制

   watch -n 2 'sudo docker stats --no-stream rustfs'

   and host memory / swap allocations:

   bash复制

   watch -n 2 'free -h'

2. Observe that RustFS memory usage continuously increases during sustained multipart traffic. In my test, RustFS grew from a few hundred MiB to approximately 6.8 GiB RSS. The host then consumed all ~3.8 GiB of swap and entered global OOM. The Linux kernel killed the RustFS process and Docker automatically restarted the container. A 2-minute run with the same multipart parameters completes normally, so the failure appears after sustained traffic rather than immediately.

Expected behavior

RustFS memory usage should remain bounded during sustained multipart uploads.

If the server reaches its safe write capacity, RustFS should apply backpressure or return an appropriate S3 error such as SlowDown, rather than continuing to increase process memory usage until the host exhausts RAM and swap.

A sustained multipart workload with only 5 concurrent UploadPart requests should not cause a host-level OOM and restart the RustFS process.

RustFS version

1.0.0

Deployment method

Docker / Docker Compose

Data directory filesystems and mounts

RustFS data paths:

• /data/rustfs0
• /data/rustfs1
• /data/rustfs2
• /data/rustfs3

All four RustFS data directories are backed by separate block devices presented directly to the RHEL VM and formatted with XFS.

No NFS, SMB/CIFS, or other remote/shared filesystem is used for the RustFS data directories.

Mount information:

===== /data/rustfs0 =====
TARGET        SOURCE   FSTYPE OPTIONS
/data/rustfs0 /dev/sdb xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

===== /data/rustfs1 =====
TARGET        SOURCE   FSTYPE OPTIONS
/data/rustfs1 /dev/sdc xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

===== /data/rustfs2 =====
TARGET        SOURCE   FSTYPE OPTIONS
/data/rustfs2 /dev/sdd xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

===== /data/rustfs3 =====
TARGET        SOURCE   FSTYPE OPTIONS
/data/rustfs3 /dev/sde xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

Docker uses host bind mounts for these paths:

• /data/rustfs0:/data/rustfs0
• /data/rustfs1:/data/rustfs1
• /data/rustfs2:/data/rustfs2
• /data/rustfs3:/data/rustfs3

Server operating system and architecture

Red Hat Enterprise Linux 9.8, x86_64

Client environment

MinIO Warp 1.3.0 (5e1fbce) in distributed mode.

Benchmark setup:

• 5 Warp client VMs
• each client runs warp client 0.0.0.0:7761
• TLS enabled
• distributed benchmark executed using --warp-client
• multipart workload uses 5 total PUTPART concurrency for the low-concurrency reproduction

Relevant logs

### Linux OOM killer

The kernel confirms a host-level/global OOM and explicitly selects the RustFS process:


Sep 30 11:11:32 kernel: dockerd invoked oom-killer
Sep 30 11:11:32 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),mems_allowed=0,global_oom,task=rustfs,pid=157822,uid=10001
Sep 30 11:11:32 kernel: Out of memory: Killed process 157822 (rustfs) total-vm:12873888kB, anon-rss:7135136kB, file-rss:0kB, shmem-rss:0kB, UID:10001

RustFS therefore had approximately 6.8 GiB anonymous RSS when killed.
Docker restart
Health check for container ... error: timed out starting health check for container

Health check for container ... error: OCI runtime exec failed: exec failed: cannot exec in a stopped container

restarting container ... exitCode=137 ... manualRestart=false restartCount=1 restartPolicy="{unless-stopped 0}"

Memory progression
RustFS container memory measured with docker stats:
11:01:09  248 MiB
11:02:05  498 MiB
11:02:16  1.06 GiB
11:02:39  2.33 GiB
11:03:34  3.89 GiB
11:03:56  4.79 GiB
11:04:19  5.46 GiB
11:04:30  5.91 GiB
11:04:41  6.37 GiB
11:04:52  6.55 GiB
11:05:26  6.83 GiB
11:05:40  6.86 GiB

Host swap progressed from 0 to the full ~3.8 GiB before the OOM.
Warp result
Reqs: 635
Errs: 0
Objs: 635
Bytes: 6350.0MiB

Report: PUTPART. Concurrency: 5. Ran: 9m31s
Average: 11.02 MiB/s, 1.10 obj/s
Average request latency: 8789.7 ms
p50: 8878.7 ms
p90: 9141.9 ms
p99: 9141.9 ms
Fastest: 697.1 ms
Slowest: 355618.5 ms

The requested benchmark duration was 5 minutes, but the command took 9m31s to complete because outstanding requests became extremely slow under memory pressure.

Screenshots

No response

Additional context

The issue was also observed with a 16 GiB RustFS VM during a higher-concurrency multipart test.

In that separate run, the Linux kernel also killed the RustFS process due to global OOM:

Out of memory: Killed process ... (rustfs) anon-rss:15183416kB

This corresponds to approximately 14.5 GiB anonymous RSS. For the 8 GiB reproduction described in this issue, short 2-minute tests were also performed with:

• 10 parts
• 25 parts
• 50 parts
• 75 parts
• 100 parts using:
• 10 MiB part size
• 5 total PUTPART concurrency All short tests completed successfully at approximately 32-34 MiB/s with no request errors. The sustained 5-minute workload is what causes memory to continue increasing until the host reaches OOM.