MetaScreener

Open-source multi-LLM ensemble for systematic review screening

---

Overview

MetaScreener automates the screening phase of systematic reviews using an ensemble of open-source large language models. Instead of relying on a single model, it runs multiple LLMs in parallel and aggregates their outputs through a calibrated confidence pipeline — producing transparent, reproducible decisions with uncertainty quantification.

Key idea: Upload your search results from PubMed/Scopus/etc., define your review criteria (PICO/PEO/SPIDER), and MetaScreener reads each title & abstract — returning include/exclude decisions with confidence scores. Uncertain cases are routed to human review.

Why MetaScreener?

Feature	Description
Multi-LLM Ensemble	4+ open-source models vote together — no single point of failure
Calibrated Confidence	Post-hoc calibration (Platt/Isotonic) maps raw scores to true probabilities
Tiered Decisions	Auto-decide on high-confidence cases, flag uncertain ones for human review
Element Consensus	Per-element (P/I/C/O) agreement scoring across models
Active Learning	Human feedback loop recalibrates model weights in real time
Full Reproducibility	temperature=0.0, seed=42, audit trail on every decision
Cost-Effective	~$0.003–0.009 per paper using free-tier API providers

---

Architecture

MetaScreener uses a Hierarchical Consensus Network (HCN) — a 4-layer screening pipeline:

…

---

Supported Models

15 open-source LLMs via OpenRouter, organized by capability:

Tier	Models	Characteristics
Flagship	DeepSeek V3, Qwen 3, Kimi K2.5	Best accuracy, strong medical knowledge
Strong	Llama 4 Maverick, GLM 5, MiniMax M2.7, Nous Hermes 4, Nvidia Nemotron, Cogito 671B, AI21 Jamba	Good balance of cost and performance
Lightweight	Gemma 3 27B, Mistral Small 4, Phi 4	Fast and cheap, good for budget screening

Recommended presets:

Preset	Models	Cost/Paper	Best For
Balanced	4 models	~$0.005	Most reviews
Precision	2 thinking + 2 large	~$0.009	High-stakes reviews
Budget	1 anchor + 3 fast	~$0.003	Large-scale screening

---

Quick Start

Prerequisites

• Python 3.11+
• An API key from OpenRouter (free sign-up, pay-per-use)

Option A: Install with pip

pip install metascreener

Then start the Web UI:

python -m metascreener
# → Open http://localhost:8000

Option B: Docker (no Python needed)

docker pull chaokunhong/metascreener:latest

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="sk-or-v1-your-key-here" \
  chaokunhong/metascreener
# → Open http://localhost:8000

Option C: From source (for developers)

Requires uv and Node.js 18+.

git clone https://github.com/ChaokunHong/MetaScreener.git
cd MetaScreener

uv sync --extra dev        # Install Python dependencies
python run.py              # Start FastAPI + Vite dev servers
# → Backend: http://localhost:8000
# → Frontend: http://localhost:5173

Configuration

Set your API key via one of these methods:

1. Web UI — Go to Settings page and paste your key
2. Environment variable:

   export OPENROUTER_API_KEY="sk-or-v1-your-key-here"
   

---

Web UI Workflow

MetaScreener provides a modern web interface (Vue 3) with a guided step-by-step workflow:

Step	Page	What It Does
0	Criteria	AI-generated PICO/PEO/SPIDER criteria from your research question
1	Settings	Select models, adjust thresholds, configure API keys
2	Screening (TA)	Upload search results → run title/abstract screening → review decisions
3	Screening (FT)	Upload PDFs → run full-text screening with intelligent chunking
4	Extraction	Extract structured data (tables, fields) from included PDFs
5	Quality	Risk-of-bias assessment (RoB 2 / ROBINS-I / QUADAS-2)
6	Evaluation	Performance metrics, calibration diagnostics, visualizations
7	History	Session audit trail with full decision provenance

Screening Flow

1. Upload your search export (.ris, .bib, .csv, .xlsx)
2. Define criteria — type your research question and let AI generate PICO criteria, or upload existing criteria
3. Run screening — papers are processed one-by-one with real-time progress
4. Review — each paper shows: decision, confidence score, tier, per-element consensus, model agreement
5. Override — disagree with a decision? Override it and the system learns from your feedback
6. Export — download results as CSV, Excel, JSON, or RIS

---

How It Works

Layer 1 — Parallel LLM Inference

Each paper is sent to 4+ LLMs simultaneously. Each model returns:

• Decision (include / exclude / uncertain)
• Confidence score (0–1)
• Per-element assessment (population, intervention, comparison, outcome)
• Reasoning chain explaining the decision

Layer 2 — Semantic Rule Engine

Hard rules auto-exclude papers that violate non-negotiable criteria (e.g., wrong language, animal study when humans required). Soft rules apply score penalties for partial matches.

Layer 3 — Calibrated Confidence Aggregation (CCA)

Raw model scores are calibrated using Platt scaling or isotonic regression, then aggregated with tier-weighted blending. The Element Consensus Score (ECS) measures per-element agreement across models — if all models agree that Population matches but disagree on Outcome, this is captured precisely.

Layer 4 — Decision Router

Papers are routed to one of four tiers:

Tier	Condition	Action
0	Hard-rule violation	Auto-exclude
1	High ECS (≥0.60) + full model agreement	Auto-decide
2	Moderate ECS (≥0.10) + within dissent tolerance	Auto-decide with lower confidence
3	Low confidence or high disagreement	Route to human review

---

Supported File Formats

Format	Extension	Source
RIS	.ris	PubMed, Scopus, Web of Science, Ovid, Embase
BibTeX	.bib	Google Scholar, Zotero, Mendeley
CSV	.csv	Any spreadsheet (must have title and abstract columns)
Excel	.xlsx	Any spreadsheet (must have title and abstract columns)
PDF	.pdf	For full-text screening and data extraction

---

Project Structure

…

---

Evaluation Metrics

MetaScreener computes standard systematic review metrics:

• Sensitivity (recall) — proportion of relevant studies correctly identified
• Specificity — proportion of irrelevant studies correctly excluded
• WSS@95 — Work Saved over Sampling at 95% recall
• AUROC — Area Under Receiver Operating Characteristic curve
• Brier Score — calibration quality (how well confidence matches correctness)
• Calibration Curves — visual diagnostic for model probability calibration
• Bootstrap 95% CI — confidence intervals on all metrics

---

Development

# Install dependencies
uv sync --extra dev

# Run tests (all offline, no API keys needed)
uv run pytest

# Lint
uv run ruff check src/

# Type check
uv run mypy src/

# Dev server (FastAPI + Vite hot reload)
python run.py

Docker Build

# Slim build (daily use)
docker build -f docker/Dockerfile --target slim -t metascreener .

# Full build (reproducibility experiments + visualization extras)
docker build -f docker/Dockerfile --target full -t metascreener:full .

---

Reproducibility

MetaScreener is designed for TRIPOD-LLM compliant reproducibility:

• temperature = 0.0 on all LLM calls (deterministic output)
• seed = 42 on all stochastic operations
• Full audit trail: every decision logs model outputs, rule violations, confidence scores, and timestamps
• Session persistence: all screening sessions saved as JSON for re-analysis
• Calibration diagnostics: Brier scores, calibration curves, and confidence intervals

---

Tech Stack

Component	Technology
Backend	FastAPI, uvicorn, Pydantic 2
Frontend	Vue 3 (Composition API), TypeScript, Vite, Pinia
LLM Routing	LiteLLM → OpenRouter API
PDF Processing	PyMuPDF (fitz), Tesseract OCR
Data Processing	pandas, NumPy, scikit-learn, SciPy
Visualization	Plotly, Chart.js
Logging	structlog (structured JSON)
Testing	pytest, MockLLMAdapter (fully offline)

---

Citation

If you use MetaScreener in your research, please cite:

@software{hong2025metascreener,
  author    = {Hong, Chaokun},
  title     = {{MetaScreener}: Open-Source Multi-LLM Ensemble for Systematic Review Screening},
  year      = {2025},
  url       = {https://github.com/ChaokunHong/MetaScreener},
  license   = {Apache-2.0}
}

---

License

Apache License 2.0 — free for academic and commercial use.