OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards
OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards
OpenJudge is an open-source evaluation framework for AI applications (e.g., AI agents or chatbots) designed to evaluate quality and drive continuous application optimization.
In practice, application excellence depends on a trustworthy evaluation workflow: Collect test data → Define graders → Run evaluation at scale → Analyze weaknesses → Iterate quickly.
OpenJudge provides ready-to-use graders and supports generating scenario-specific rubrics (as graders), making this workflow simpler, more professional, and easy to integrate into your workflow. It can also convert grading results into reward signals to help you fine-tune and optimize your application.
** Try it now!** Visit openjudge.me/app to use graders online — no installation required. Test built-in graders, build custom rubrics, and explore evaluation results directly in your browser.
2026-06-17 - PawBench v1.0 - A Model × Harness co-evaluation benchmark for agentic AI: 150 tasks · 9 models · 3 harnesses, with public prompts, graders, task labels, submissions, and leaderboard slices. GitHub | Leaderboard
2026-04-07 - Skill Graders - 5 new LLM-based graders for evaluating AI Agent Skill packages: threat analysis (AITech taxonomy), declaration alignment, completeness, relevance, and design quality. Documentation | Cookbook
2026-03-10 - ️ New Skills - Claude authenticity verification, find skills combo, and more. Browse Skills
2026-02-12 - Reference Hallucination Arena - Benchmark for evaluating LLM academic reference hallucination. Documentation | Leaderboard
2026-01-27 - Paper Review - Automatically review academic papers using LLM-powered evaluation. Documentation
2026-01-27 - ️ OpenJudge UI - A Streamlit-based visual interface for grader testing and Auto Arena. Try Online | Run locally: streamlit run ui/app.py
Access 50+ production-ready graders featuring a comprehensive taxonomy, rigorously validated for reliable performance.
Focus: Semantic quality, functional correctness, structural compliance
Key Graders:
Relevance - Semantic relevance scoringSimilarity - Text similarity measurementSyntax Check - Code syntax validationJSON Match - Structure complianceFocus: Agent lifecycle, tool calling, memory, plan feasibility, trajectory quality
Key Graders:
Tool Selection - Tool choice accuracyMemory - Context preservationPlan - Strategy feasibilityTrajectory - Path optimizationFocus: Image-text coherence, visual generation quality, image helpfulness
Key Graders:
Image Coherence - Visual-text alignment
Text-to-Image - Generation quality
Image Helpfulness - Image contribution
Multi-Scenario Coverage: Extensive support for diverse domains including Agent, text, code, math, and multimodal tasks. Explore Supported Scenarios
Holistic Agent Evaluation: Beyond final outcomes, we assess the entire lifecycle—including trajectories, Memory, Reflection, and Tool Use. Agent Lifecycle Evaluation
✅ Quality Assurance: Every grader comes with benchmark datasets and pytest integration for validation. View Benchmark Datasets
Choose the build method that fits your requirements:
Using mainstream observability platforms like LangSmith or Langfuse? We offer seamless integration to enhance their evaluators and automated evaluation capabilities. We also provide integrations with training frameworks like VERL for RL training. See Integrations for details
Explore OpenJudge without writing a single line of code. Our online platform at openjudge.me/app lets you:
Don't want to install anything? Try OpenJudge online — use graders directly in your browser, no setup needed.
pip install py-openjudge
More installation methods can be found in the Quickstart Guide.
Complete Quickstart can be found in the Quickstart Guide.
A simple example to evaluate a single response:
…
Use multiple built-in graders to comprehensively evaluate your LLM application: Explore All built-in graders
Business Scenario: Evaluating an e-commerce customer service agent that handles order inquiries. We assess the agent's performance across three dimensions: relevance, hallucination, and tool selection.
…
Generate a custom grader from task description without labeled data: Zero-shot Rubrics Generation Guide
When to use: Quick prototyping when you have no labeled data but can clearly describe your task.
…
Learn evaluation criteria from labeled examples: Data-driven Rubrics Generation Guide
When to use: You have labeled data and need high-accuracy graders for production use, especially when evaluation criteria are implicit.
…
Seamlessly connect OpenJudge with mainstream observability and training platforms:
| Category | Platform | Status | Documentation |
|---|---|---|---|
| Observability | LangSmith | ✅ Available | LangSmith Integration Guide |
| Langfuse | ✅ Available | Langfuse Integration Guide | |
| Other frameworks | Planned | — | |
| Training | verl | ✅ Available | VERL Integration Guide |
| Trinity-RFT | Planned | — |
Have a framework you'd like us to prioritize? Open an Issue!
OpenJudge is the foundation of a growing evaluation ecosystem. These projects share OpenJudge's philosophy of evaluation-driven optimization while targeting specific verticals.
The same model can behave very differently depending on which agent runtime (harness) it runs inside. PawBench evaluates the model and the harness together, keeping enough metadata to analyze both dimensions independently:
$$\text{Agent Performance} = f(\text{Model}, \text{Harness})$$
| Dimension | Coverage |
|---|---|
| Tasks | 150 tasks from 6 sources (ClawEval, QwenClawBench, PinchBench, SkillsBench, WildClawBench, self-built) |
| Models | 9 models (Qwen, Claude, GLM, etc.) |
| Harnesses | 3 harnesses (QwenPaw, OpenClaw, Hermes) |
| Task labels | 5 dimensions: scenario, capability, complexity, modality, environment |
Key findings from v1.0: harness design alone can shift a model's score by 10+ points — a gap comparable to many model upgrades. PawBench provides slice diagnostics to pinpoint whether regressions come from the model, the harness, or the grader. GitHub | Leaderboard | Documentation
We love your input! We want to make contributing to OpenJudge as easy and transparent as possible.
** Adding New Graders** — Have domain-specific evaluation logic? Share it with the community! ** Reporting Bugs** — Found a glitch? Help us fix it by opening an issue ** Improving Docs** — Clearer explanations or better examples are always welcome ** Proposing Features** — Have ideas for new integrations? Let's discuss!
See full Contributing Guidelines for coding standards and PR process.
Join our DingTalk group to connect with the community:
OpenJudge was previously distributed as the legacy package
rm-gallery(v0.1.x). Starting from v0.2.0, it is published aspy-openjudgeand the Python import namespa
No open issues yet, or sync has not completed.