MinerU API - RSS grows unbounded across documents until OOM
Search before asking | 提交之前请先搜索
- I have searched the MinerU Readme and found no similar bug report.
- I have searched the MinerU Issues and found no similar bug report.
- I have searched the MinerU Discussions and found no similar bug report.
Consult the online AI assistant for assistance | 在线 AI 助手咨询
- I have consulted the online AI assistant but was unable to obtain a solution to the issue.
Description of the bug | 错误描述
Hello, I'm using MinerU 3.4.4 FastAPI interface in my project in docker container and I've noticed that out of memory error occurs (RAM, not VRAM) - after each document memory usage is increasing, till OOM and process is being killed. I spent few hours to find the issue and I noticed that memory is not being properly cleared from the heap.
Data collected with memray . As you can see heap size is constant after each parsing (small spikes on blue plot), while RSS is increasing.
Temporary fix:
I found solution by manually using libc.malloc_trim(0) after parsing each document window - maybe it will be helpful for someone (in this case I'm using hybrid-http-client):
# /backend/hybrid/hybrid_analyze.py
import ctypes
libc = ctypes.CDLL("libc.so.6")
# in aio_doc_analyze, at the end of
for window_index, window_start in enumerate(range(0, page_count, effective_window_size or 1)):
...
libc.malloc_trim(0)
With this fix, memory usage decreases after parsing each document chunk and remains stable at a plateau throughout the entire execution.
How to reproduce the bug | 如何复现
To reproduce:
- Build official MinerU docker image
wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/global/Dockerfile
docker build -t mineru:latest -f Dockerfile .
- Run mineru-api
docker compose -f compose.yaml --profile api up -d
- Upload any PDFs and observe memory grow:
docker container stats mineru-api
- After each upload memory usage is increasing (expected: after successful parse memory usage decreases)
Operating System Mode | 操作系统类型
Linux
Operating System Version| 操作系统版本
Ubuntu 24.04 (WSL)
Python version | Python 版本
3.12
Software version | 软件版本 (mineru --version)
>=2.5
Backend name | 解析后端
vlm
Device mode | 设备模式
cuda
Source: opendatalab/MinerU