Optional EvalPort interop for Datalab label-quality results
Hi cleanlab team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first per CONTRIBUTING.md before writing any code.
I installed cleanlab (2.9.0) and ran the real Datalab API rather than guessing:
lab = Datalab(data=data, label_name="labels", task="classification")
lab.find_issues(pred_probs=pred_probs)
lab.get_issues()get_issues() returns one row per example with columns like is_label_issue (bool), label_score (float, 0–1), plus the same is_*_issue / *_score pair for every other issue type that ran (outlier, non_iid, class_imbalance, etc.). get_issue_summary() aggregates those into issue_type / score / num_issues rows.
That's a very direct match for EvalPort's other half: each example in the audited dataset is a TestCase, and the is_label_issue + label_score (or any other issue-type's bool+score pair) that Datalab computes per example is exactly the shape of an EvalPort Grader result — a per-TestCase pass/fail plus a numeric score — so Datalab.get_issues() output could round-trip into an EvalPort ResultSet with very little translation.
Two ways I could see this landing, and I don't have a strong preference:
- A standalone
cleanlab-openeval-adapterpackage in the EvalPort repo, depending oncleanlabas a normal dependency. Zero footprint on this repo. - A small optional module inside this repo if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
Source: cleanlab/cleanlab