production-ocr-course

Table of Contents

• Table of Contents
• Course Overview
• Who is this course for?
• Course Breakdown: Week by Week
• Getting Started
• ✨ Beyond Traditional OCR: The SLM Advantage
• Pipeline Architecture: Deep Dive into Throughput & Scaling
• The Exact Document Workflow
• Robustness: Why a Pre-layout Encoder Improves Fidelity
• Technical Report: Document Handoff
• ️ Formal Architecture Assessment: Production Robustness
• Scaling Philosophy: Real-Time Workloads (Metric-Driven)
• The tech stack
• Contributors
• License

Course Overview

Most OCR tutorials stop at "call an API and get some text back." This isn't that.

Instead, we're building a production-grade, self-scaling Visual Document Understanding pipeline, deployed for real on Kubernetes (AKS or GKE), that goes far beyond flat text extraction: it reasons about charts, tables, and layout the way a human reader would — powered by a Small Language Model (Qwen 3.5) instead of a bloated frontier model.

By the end of this course, you'll have your own event-driven OCR system capable of:

• Understanding documents, not just transcribing them — charts, tables, handwriting, and contextual reasoning via Qwen 3.5 (4B)
• ⚡ Serving generative OCR at 1.86 pages/second with vLLM's continuous batching, PagedAttention, and Multi-Token Prediction (MTP)
• Ingesting files through a high-concurrency Rust (Axum) gateway, decoupled from GPU inference via Redis
• Running a zero-copy, /dev/shm-based document handoff between the layout encoder and the inference engine
• ☸️ Auto-scaling T4 (layout) and A100 (inference) node pools independently with KEDA, from zero to bursting load
• Locking the whole pipeline behind an Internal Load Balancer + Enterprise API Gateway (Azure APIM / GCP API Gateway), with zero public exposure
• Wrapping the pipeline as an MCP server for native use by AI agents, including Claude Code

Excited? Let's get started!

---

---

Who is this course for?

This course is for ML/AI Engineers and Platform Engineers who already know how to call an OCR API and want to know what it takes to run one in production: GPU node pools, autoscaling economics, network security, and the systems-design tradeoffs behind serving a generative model at throughput.

Course Breakdown: Week by Week

We run this as a 6-week hands-on engineering cohort. The entire repository is open-source from day one — every week combines a deep-dive systems article with a step-by-step codebase walkthrough.

Weekly Cadence

• Weekly Production Article (Wednesday): Architectural deep dives, systems design math, and codebase explanations.
• ️ Live Office Hours (Friday): Live coding, cluster provisioning, load-testing, scaling demonstration, and Q&A.

Week	Focus	Hands-on
⛵ 1. Kubernetes for AI Systems	Pods, Services, Node Pools, resource scheduling	Cluster setup on AKS/GKE, T4 & A100 node pools, GPU drivers/operators with proper security profiles and taints
2. SOTA OCR Approaches & VDU	Single-stage end-to-end models vs. our two-stage layout-first pipeline	Evaluating GLM-OCR SDK's layout detection and measuring performance trade-offs
⚡ 3. Deploying the vLLM Inference Engine	Continuous batching, PagedAttention, scheduling optimizations	Deploying Qwen 3.5 4B on vLLM, tuning MAX_NUM_BATCHED_TOKENS, chunked prefills, Multi-Token Prediction (MTP) — standalone deployment walkthrough
4. Rust Ingest Gateway	High-concurrency gateways for heavy payloads; ownership, borrowing, async Rust	Building the Axum gateway (client_rt_producer), 10MB limits, atomic HSET writes to Redis — standalone deployment walkthrough
5. Async Architectures & Zero-Copy Ingestion	Queue buffers, dynamic batching collectors, RAM-disk transfer	Building the Python worker (client_rt_consumer), 100ms collection window, /dev/shm handoff, scale-to-zero with KEDA — standalone deployment walkthrough
️ 6. Enterprise Gateways & Claude Code MCP	Security boundaries, JWT verification, rate limiting, agentic workflows	Configuring Azure APIM / GCP API Gateway policies and wrapping the pipeline in an MCP server for Claude Code — standalone deployment walkthrough

Getting Started

Start with account setup, then GPU quota, then the full deployment guide for your cloud of choice:

1. Create your cloud account & claim free credits
   • Azure Account Setup — the course's primary cloud
   • GCP Account Setup — optional, if you'd rather run on Google Cloud
2. Request GPU quota (T4 + A100 — this is the step most people get stuck on; free/trial accounts cannot run GPUs)
   • Azure GPU Access & Quota Prerequisites
   • GCP GPU Access & Quota Prerequisites
3. Deploy the pipeline
   • Azure Kubernetes Service (AKS) Deployment Guide
   • Google Kubernetes Engine (GKE) Deployment Guide
   • ☁️ Cloud Provider Comparison & Discrepancies Matrix

---

✨ Beyond Traditional OCR: The SLM Advantage

Modern OCR has evolved from simple character recognition to Visual Document Understanding (VDU). By leveraging Small Language Models (SLMs) like Qwen 3.5 (4B), this pipeline moves beyond "flat" text extraction to provide:

• Fine-Grained Chart & Image Description: Unlike task-specific models that only see text, Qwen 3.5 can interpret trends in graphs, describe complex diagrams, and identify semantic relationships between visual elements.
• Contextual Reasoning: The model understands the intent of a document. It can distinguish between an invoice total and a line-item subtotal based on spatial reasoning, not just keyword matching.
• High-Density Recognition: Achieving state-of-the-art results on benchmarks like OmniDocBench, these models handle "noisy" real-world inputs—handwriting, annotations, low-resolution scans—with a record-breaking accuracy that outperforms much larger frontier models.

Key Technical Innovations

• Hybrid Orchestration: Combines a deterministic PP-DocLayoutV3 visual encoder (via GLM-OCR SDK) with the Qwen 3.5 language decoder. This ensures that every image crop is semantically labeled before it hits the generative engine.
• Multi-Token Prediction (MTP): The architecture is optimized for MTP-enabled VLMs, resulting in a ~50% throughput increase by predicting multiple tokens per decoding step without increasing VRAM overhead.
• Zero-Copy RAM Handoff: Documents are rasterized into /dev/shm (Shared Memory), allowing the layout engine and the inference engine to share high-resolution buffers without costly disk I/O.

---

Pipeline Architecture: Deep Dive into Throughput & Scaling

The architecture leverages the GLM-OCR SDK as a high-concurrency orchestration layer to bridge the gap between deterministic layout analysis and generative SLM recognition.

1. The Engine: vLLM + MTP-Accelerated Inference

By deploying models via vLLM, we leverage continuous batching and PagedAttention to maximize the A100's utility.

• MTP Speed Hack: Because vLLM supports Multi-Token Prediction (MTP), the generative stage is significantly faster than traditional VLMs, reaching throughputs of 1.86 pages/second for PDFs.
• Two-Stage Parallelism: While the vLLM sidecar handles the intensive recognition of Batch N, the Orchestrator/Layout pod is already pre-processing Batch N+1 using the PP-DocLayoutV3 detector.
• Dynamic Cropping: The system doesn't feed full-page images into the SLM (which is token-expensive). Instead, it crops semantic regions (paragraphs, tables, formulas, charts), allowing the model to focus its parameters on high-density information.

2. Throughput Optimization Matrix: Tuning for A100

To maximize the ROI of the A100 nodes, the system is tuned to balance the asymmetric load between CPU-bound preprocessing and GPU-bound inference.

Parameter	Optimized Value	Technical Rationale
max_workers	512	Matches the MAX_NUM_SEQS of the vLLM server. Ensures that the worker can saturate the continuous batching engine with concurrent region-recognition requests.
layout/batch_size	4	For multi-page PDFs, this allows the PP-DocLayoutV3 model to process multiple pages in a single GPU forward pass, reducing kernel launch overhead.
layout/workers	4	Utilizes multi-core CPUs for parallel image decoding, resizing, and normalization, preventing the GPU from idling while waiting for input tensors.
connection_pool	1024	Prevents HTTP connection exhaustion when handling high-concurrency bursts across the 512 workers.

3. vLLM Engine Optimizations: Pushing the A100

To fully utilize the massive 80GB VRAM of the A100 node for models like Qwen3.5-4B, we aggressively tune the internal parameters of the vLLM engine:

• MAX_NUM_SEQS=512: Increased from default to allow the vLLM continuous batching scheduler to process up to 512 concurrent image crops or text completions in parallel.
• MAX_NUM_BATCHED_TOKENS=262144: Massively increased to eliminate the "prefill bottleneck". Because each layout crop contains ~6K tokens, a standard batch size limits prefilling to only ~5 crops at a time. By pushing this to 262K, the engine can prefill ~42 images simultaneously in a single forward pass.
• MAX_MODEL_LEN=16384: Explicitly reduced from 32K. Setting this to 16K perfectly covers the worst-case scenario (6K image + 8K generation) while preventing overly pessimistic reservations. This frees up gigabytes of VRAM to physically accommodate the 512 sequences.
• Chunked Prefill Enabled (--enable-chunked-prefill): Prevents large prefill tasks from blocking decoding steps of currently running requests. By chunking prefills and co-scheduling them with decoding tokens, the engine guarantees lower latency spikes under heavy concurrent workloads.
• CUDA Graphs Enabled (No --enforce-eager): Critical for Multi-Token Prediction (MTP). We allow vLLM to compile CUDA graphs during cold-start. This eliminates the massive CPU bottleneck caused by dispatching thousands of micro-kernels per generation step.

4. Motivation: Hardware Asymmetry & Workload Profiling

Colocating the SDK worker and vLLM across T4 (Layout) and A100 (OCR) node pools allows for optimized resource utilization. The system uses a Sequential Dynamic Batching strategy to saturate the GPU without causing kernel contention.

Component	Hardware Target	Workload Profile	Batching Strategy
Worker (SDK)	T4 GPU	Layout Bound: Uses PP-DocLayoutV3 to identify regions.	Dynamic Collector: Batches up to 4 tasks in 100ms windows to reduce kernel overhead.
vLLM (SLM)	A100 GPU	Compute Bound: High-speed generative OCR/Reasoning using MTP.	Continuous Batching: Sa