[PP-OCRv6][Performance] High CPU inference latency and low CPU utilization - need Small model accuracy with lower latency

Search before asking

• I have searched the PaddleOCR Docs and found no similar bug report.
• I have searched the PaddleOCR Issues and found no similar bug report.
• I have searched the PaddleOCR Discussions and found no similar bug report.

Bug (问题描述)

Problem

We are trying to achieve low OCR latency while maintaining good accuracy for invoices and table documents.

We initially ran PaddleOCR with:

2 vCPU / 4 GB RAM

and then increased the infrastructure to:

4 vCPU / 8 GB RAM

However, increasing the resources provided only around 0.2 seconds latency improvement.

More importantly, during OCR inference the container does not appear to fully utilize the available CPU.

Although approximately 3.5 vCPU is allocated to the pod, observed CPU consumption is typically only around 1-1.5 vCPU.

Memory consumption is also only approximately 1-1.5 GB.

Therefore, increasing the pod from 2 vCPU to approximately 4 vCPU has not resulted in proportional inference performance improvement.

Example profiling:

OCR_INFERENCE_MS=7549 OCR_PARSING_MS=25 TOTAL_REQUEST_MS=8115

OCR_INFERENCE_MS=6981 OCR_PARSING_MS=14 TOTAL_REQUEST_MS=7017

OCR_INFERENCE_MS=8871 OCR_PARSING_MS=42 TOTAL_REQUEST_MS=8937

OCR_INFERENCE_MS=8555 OCR_PARSING_MS=20 TOTAL_REQUEST_MS=9300

Almost all request latency is inside PaddleOCR inference. Application parsing takes only a few milliseconds.

‍♂️ Environment (运行环境)

Environment

We are running PaddleOCR as an OCR API on AWS EKS.

• PaddleOCR: 3.7.0
• PaddlePaddle: 3.2.0
• PaddleX: 3.7.x
• Python: 3.10
• Device: CPU
• AWS Region: eu-north-1
• EC2 instance: c6i.xlarge
• Instance resources: 4 vCPU / 8 GiB RAM
• Pod CPU request/limit: 3500m
• Pod memory: 5-6 GiB
• Architecture: linux/amd64

Runtime configuration:

PADDLEOCR_CPU_THREADS=3 PADDLEOCR_ENABLE_MKLDNN=true OMP_NUM_THREADS=3 MKL_NUM_THREADS=3 OPENBLAS_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 PADDLEOCR_DET_LIMIT=1024

Minimal Reproducible Example (最小可复现问题的Demo)

PP-OCRv6 Tiny vs Small

We tested Tiny models to improve latency:

text_detection_model_name="PP-OCRv6_tiny_det" text_recognition_model_name="PP-OCRv6_tiny_rec"

Latency improved significantly.

However, Tiny causes accuracy problems with small/narrow text in invoice tables.

For example, our invoice contains a narrow column:

DG

with values:

1 1 1 1 ...

Some of these values are not captured when using Tiny.

We therefore tested:

text_detection_model_name="PP-OCRv6_small_det" text_recognition_model_name="PP-OCRv6_tiny_rec"

and increased:

text_det_limit_side_len=1280

However, the small-text accuracy issue still occurs.

Using:

text_detection_model_name="PP-OCRv6_small_det" text_recognition_model_name="PP-OCRv6_small_rec"

provides better accuracy, but inference latency becomes significantly higher.

Therefore we currently have this trade-off:

Tiny: Good latency, insufficient accuracy for small invoice/table text.

Small: Good accuracy, but significantly higher CPU latency.