Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM
Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM
Try Sparrow Online | Quick Start | ️ Installation | Examples | Agents
--- ## Sparrow [ Try Sparrow Online](https://sparrow.katanaml.io) Production-ready structured data extraction powered by ML, LLMs & Vision LLMs. Turn invoices, receipts, statements, forms and images into clean structured data. Sparrow is an **API-first platform** built for enterprise document intelligence. It provides RESTful APIs for structured data extraction, instruction processing, and multi-agent workflow orchestration — all running on your own infrastructure with no external API calls or cloud dependencies. **Platform capabilities:** - **Structured Extraction API**: Submit documents via REST and receive validated JSON — integrate directly into any backend or data pipeline - **Instruction Processing**: Beyond document extraction — text processing, validation, and decision making via the instruction inference API - **Agent Framework**: Orchestrate multi-step workflows with custom agents, visual monitoring via Prefect, and robust error handling - **Pluggable Pipelines**: Mix and match Vision LLM (Sparrow Parse), Text LLM (Sparrow Instructor), and Agent pipelines depending on the task - **Multiple Backends**: MLX on Apple Silicon, vLLM on NVIDIA, Ollama, Hugging Face, Mistral OCR — same API surface across all ### Sparrow UI Features The web UI provides a visual interface on top of the same API: - **Drag & Drop**: Upload documents directly - **Real-time Processing**: See results instantly - **Data Query**: JSON based schema for data query - **Structured Output**: JSON structured output ## Table of Contents - [✨ Key Features](#-key-features) - [️ Architecture](#️-architecture) - [ Quickstart](#-quickstart) - [️ Installation](#️-installation) - [ Examples](#-examples) - [ CLI Usage](#-cli-usage) - [ API Usage](#-api-usage) - [ Sparrow Agent](#-sparrow-agent) - [ Dashboard](#-dashboard) - [ Pipeline Comparison](#-pipeline-comparison) - [⚡ Performance Tips](#-performance-tips) - [ Troubleshooting](#-troubleshooting) - [⭐ Star History](#-star-history) - [ License](#-license) ## ✨ Key Features **Universal Document Processing**: Handle invoices, receipts, forms, bank statements, tables **Pluggable Architecture**: Mix and match different pipelines (Sparrow Parse, Instructor, Agents) ️ **Multiple Backends**: MLX, Ollama, vLLM, Docker, Hugging Face Cloud GPU, Mistral OCR **Multi-format Support**: Images (PNG, JPG) and multi-page PDFs **Schema Validation**: JSON schema-based extraction with automatic validation **API-First Design**: RESTful APIs for easy integration **Instruction Calling**: Text processing, validation, decision making with Gemma, Mistral, Qwen 3.6, etc. **Visual Monitoring**: Built-in dashboard and agent workflow tracking **Enterprise Ready**: Rate limiting, usage analytics, commercial licensing available **Local Vision LLMs**: Mistral, Qwen 3.6, DeepSeek OCR, dots.ocr, Gemma 4, etc. ☁️ **Cloud OCR Backend**: Mistral OCR for cloud document extraction ## ️ Architecture ### Core Components | Component | Purpose | Use Case | |-----------|---------|----------| | **[Sparrow ML LLM](https://github.com/katanaml/sparrow/tree/main/sparrow-ml/llm)** | Main API engine | Document processing pipelines | | **[Sparrow Parse](https://github.com/katanaml/sparrow/tree/main/sparrow-data/parse)** | Vision LLM library | Structured JSON extraction | | **[Sparrow Agents](https://github.com/katanaml/sparrow/tree/main/sparrow-ml/agents)** | Workflow orchestration | Complex multi-step processing | | **[Sparrow OCR](https://github.com/katanaml/sparrow/tree/main/sparrow-data/ocr)** | Text recognition | OCR preprocessing | | **[Sparrow UI](https://github.com/katanaml/sparrow/tree/main/sparrow-ui/)** | Web interface | Interactive document processing | ## Quickstart ### Prerequisites - **Python 3.12.10+** (use `pyenv` for version management) - **macOS** (for MLX backend) or **Linux/Windows** (for other backends) - **GPU** (make sure GPU have enough memory to run selected Vision LLM) ### 30-Second Setup ```bash # 1. Install pyenv and Python 3.12.10 pyenv install 3.12.10 pyenv global 3.12.10 # 2. Create virtual environment python -m venv .env_sparrow_parse source .env_sparrow_parse/bin/activate # Linux/Mac # or .env_sparrow_parse\Scripts\activate # Windows # 3. Install Sparrow Parse pipeline git clone https://github.com/katanaml/sparrow.git cd sparrow/sparrow-ml/llm pip install -r requirements_sparrow_parse.txt # 4. For macOS: Install poppler for PDF processing brew install poppler # 5. Start the API server python api.py ``` Before running `pip install -r requirements_sparrow_parse.txt`, check your platform. If you are on macOS and want to run MLX backend, go to `requirements_sparrow_parse.txt` and make sure `sparrow-parse[mlx]` libary reference is defined. If you are running Sparrow on Linux/Windows, make sure to use `sparrow-parse` library reference, this will skip MLX related libraries. ### First Document Extraction ```bash # Extract data from a bonds table ./sparrow.sh '[{"instrument_name":"str", "valuation":0}]' \ --pipeline "sparrow-parse" \ --options mlx \ --options mlx-community/Qwen2.5-VL-72B-Instruct-4bit \ --file-path "data/bonds_table.png" ``` **Result:** ```json { "data": [ {"instrument_name": "UNITS BLACKROCK...", "valuation": 19049}, {"instrument_name": "UNITS ISHARES...", "valuation": 83488} ], "valid": "true" } ``` Use `--options mlx` for MLX backend, `--options ollama` for Ollama backend, `--options vllm` for vLLM backend, `--options mistral` for Mistral OCR cloud backend. Make sure to provide correct Vision LLM model name, download model first separately with MLX, vLLM or Ollama. ## ️ Installation ### Quick Setup ```bash # 1. Clone repository git clone https://github.com/katanaml/sparrow.git cd sparrow ``` **For complete installation instructions**, see our [detailed environment setup guide](environment_setup.md). ### Essential Steps Summary 1. **Python Environment**: Install Python 3.12.10 using pyenv 2. **Virtual Environments**: Create separate environments for different pipelines: - `.env_sparrow_parse` - for Sparrow Parse (Vision LLM) - `.env_instructor` - for Instructor (Text LLM) - `.env_ocr` - for OCR service (optional) 3. **System Dependencies**: Install poppler for PDF processing 4. **Requirements**: Install pipeline-specific dependencies, for example: `pip install -r requirements_sparrow_parse.txt` ### Platform-Specific Notes **macOS:** ```bash brew install poppler # Required for PDF processing ``` **Ubuntu/Debian:** ```bash sudo apt-get install poppler-utils libpoppler-cpp-dev ``` **Apple Silicon**: MLX backend available for optimal performance **NVIDIA/AMD GPU**: Use vLLM or Ollama backend **Cloud**: Use Mistral OCR backend **CPU Only**: Use smaller models or Hugging Face cloud backend ### Verification ```bash # Test installation python api.py --port 8002 # Visit http://localhost:8002/api/v1/sparrow-llm/docs ``` ## Examples ### Bank Statement Processing ```bash # Extract all data from bank statement ./sparrow.sh "*" \ --pipeline "sparrow-parse" \ --options mlx \ --options mlx-community/Qwen2.5-VL-72B-Instruct-4bit \ --file-path "data/bank_statement.pdf" ``` View Complete JSON Output ``` … ``` ### Financial Tables ```bash # Extract structured data from financial table ./sparrow.sh '[{"instrument_name":"str", "valuation":0}]' \ --pipeline "sparrow-parse" \ --options mlx \ --options mlx-community/Qwen2.5-VL-72B-Instruct-4bit \ --file-path "data/bonds_table.png" ``` View JSON Output ``` … ``` ### Invoice Processing ```bash # Extract invoice with cropping for better accuracy ./sparrow.sh "*" \ --pipeline "sparrow-parse" \ --options mlx \ --options mlx-community/Qwen2.5-VL-72B-Instruct-4bit \ --crop-size 60 \ --file-path "data/invoice.pdf" ``` View Complete JSON Output ``` … ``` ### Multi-page PDF Processing ```bash # Process multi-page PDF with structured output per page ./sparrow.sh '{"table": [{"description": "str", "latest_amount": 0, "previous_amount": 0}]}' \ --pipeline "sparrow-parse" \ --options mlx \ --options mlx-community/Qwen2.5-VL-72B-Instruct-4bit \ --file-path "data/financial_report.pdf" \ --debug-dir "debug/" ``` View JSON Output ``` … ``` ### Text Instruction Processing ```bash # Instruction-based processing ./sparrow.sh "instruction: do arithmetic operation, payload: 2+2=" \ --pipeline "sparrow-instructor" \ --options mlx \ --options lmstudio-community/Mistral-Small-3.2-24B-Instruct-2506-8bit # Instruction processing with document input ./sparrow.sh "check if business entity Chapman, Kim and Green is invoice issuing party" --pipeline "sparrow-parse" --instruction --options mlx --options lmstudio-community/Mistral-Small-3.2-24B-Instruct-2506-8bit --file-path "invoice_1.jpg" ``` **JSON Output:** ``` The result of 2 + 2 is: 4 ``` ### Stock Data Function Calling ```bash # Function calling example ./sparrow.sh assistant --pipeline "stocks" --query "Oracle" ``` **JSON Output:** ```json { "company": "Oracle Corporation", "ticker": "ORCL" } ``` **Additional Output:** ``` The stock price of the Oracle Corporation is 186.3699951171875. USD ``` ### Table/Form Processing with custom Sparrow logic ```bash ./sparrow.sh "{\"items\": [{\"instrument_name\":\"str\", \"quantity\": \"float\", \"valuation\":\"int\"}]}" \ --pipeline "sparrow-parse" \ --debug --table --table-template "sparrow_generic_table" \ --options mlx --options mlx-community/Ministral-3-14B-Instruct-2512-8bit \ --options mlx --options mlx-community/dots.ocr-bf16 --file-path "data/bonds_table.png" ``` ### Query Hints ```bash ./sparrow.sh "[{\"instrument_name\":\"str\", \"valuation\":\"int\"}]" \ --pipeline "sparrow-parse" --debug --options mlx \ --options mlx-community/gemma-4-31b-it-8bit \ --file-path "data/bonds_table.png" --hints-file-path "data/llm_hints_eu.json" ``` ## CLI Usage ### Basic Syntax ```bash ./sparrow.sh "" --pipeline "" [OPTIONS] --file-path "" ``` ### Command Line Arguments | Argument | Type | Description | Example | |----------|------|-------------|---------| | `query` | JSON/String | Schema or instruction | `'[{"field":"str"}]'` | | `--pipeline` | String | Pipeline to use | `sparrow-parse` | | `--file-path` | Path | Input document | `data/invoice.pdf` | | `--hints-file-path` | Path | Query hints | `data/hints.json` | | `--options` | String | Backend configuration | `mlx,model-name` | | `--instruction` | Boolean | Sparrow query will be used as instruction | `--instruction` | | `--validation` | Boolean | Sparrow query will be used for field validation | `--validation` | | `--markdown` | Boolean | Markdown pre-processing | `--markdown` | | `--ocr` | Boolean | Experimental functionality | `--ocr` | | `--table` | Boolean | Experimental functionality | `--table` | | `--table-template` | String | Experimental functionality | `--name` | | `--crop-size` | Integer | Border cropping pixels | `60` | | `-No open issues yet, or sync has not completed.