[Rust] Implement ONNX session pool for TableProcessor to improve parallelism
Author: lfgranjaCreated Feb 19, 2026Updated Feb 19, 2026
Problem
The current TableProcessor implementation in the Rust engine uses a std::sync::Mutex<Session> to serialize all ONNX inference operations:
pub struct TableProcessor {
session: Option<std::sync::Mutex<Session>>,
model_type: TableModelType,
}
This design means that all table extraction operations are serialized, even when processing multiple pages in parallel. This creates a bottleneck when:
- Processing multi-page PDFs with many tables
- Running on multi-core systems where parallelism could be exploited
- Using Rayon's
par_iter()for page processing
Proposed Solution
Implement a session pool pattern that allows multiple concurrent inference operations:
Option 1: Fixed-size Session Pool
pub struct TableProcessor {
sessions: Vec<Arc<Mutex<Session>>>,
pool_size: usize,
next_idx: AtomicUsize,
}
Option 2: Thread-local Sessions (simpler)
Use thread_local! to give each Rayon thread its own session, avoiding lock contention entirely.
Option 3: Async-aware Session Pool
If we move to fully async processing, use tokio::sync::Semaphore with multiple sessions.
Benefits
- Improved throughput for table-heavy documents
- Better utilization of multi-core systems
- Reduced latency for batch processing
Considerations
- ONNX Runtime sessions are not thread-safe, so each session still needs protection
- Memory usage increases with pool size (each session has its own allocator state)
- Need to benchmark to find optimal pool size
Current Workaround
The existing implementation works correctly but may be slower for documents with many tables due to serialization.
Related
- Issue #205 (Table Transformer ONNX output contract)
- PR #9 (FusionEngine implementation)
Source: HKUDS/RAG-Anything