#1334·cleanlab

Optional EvalPort interop for Datalab label-quality results

Author: adhabnr-uxCreated Aug 22, 2026Updated Aug 22, 2026

Hi cleanlab team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first per CONTRIBUTING.md before writing any code.

I installed cleanlab (2.9.0) and ran the real Datalab API rather than guessing:

python
lab = Datalab(data=data, label_name="labels", task="classification")
lab.find_issues(pred_probs=pred_probs)
lab.get_issues()

get_issues() returns one row per example with columns like is_label_issue (bool), label_score (float, 0–1), plus the same is_*_issue / *_score pair for every other issue type that ran (outlier, non_iid, class_imbalance, etc.). get_issue_summary() aggregates those into issue_type / score / num_issues rows.

That's a very direct match for EvalPort's other half: each example in the audited dataset is a TestCase, and the is_label_issue + label_score (or any other issue-type's bool+score pair) that Datalab computes per example is exactly the shape of an EvalPort Grader result — a per-TestCase pass/fail plus a numeric score — so Datalab.get_issues() output could round-trip into an EvalPort ResultSet with very little translation.

Two ways I could see this landing, and I don't have a strong preference:

  1. A standalone cleanlab-openeval-adapter package in the EvalPort repo, depending on cleanlab as a normal dependency. Zero footprint on this repo.
  2. A small optional module inside this repo if you'd rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.