Map `eval/lib/grader.py`'s judge output onto the EvalPort interchange format (portable ResultSet for WorldVQA/SimpleQA/exact-match)

eval/lib/grader.py already produces something very close to a portable eval-results format — it's just not written to disk in one. I maintain EvalPort (Apache-2.0), an open interchange spec + Python/TS SDK for LLM eval test cases, graders, and results, and I think there's a genuine, low-effort fit here. Posting as a proposal/question, not a PR — happy to build it if there's interest, or take it entirely to the EvalPort side if you'd rather this repo stay judge-only.

What grade_file already has

Reading the real code (not paraphrasing):

• grade_file(task, path, grader_model=DEFAULT_GRADER_MODEL, concurrency=16, llm_judge=False) reads rows = [json.loads(l) for l in open(path)], each row carrying problem, original_data, and final_response.
• For WorldVQA-family tasks (WORLDVQA_TASKS), build_ground_truth(task, original_data) produces the reference string (e.g. "Any of: " + " | ".join(refs) for encyclopedic_vqa), and JUDGE_WORLDQA_PROMPT_EN.format(question=..., model_answer=..., ground_truth_answer=...) builds the judge prompt.
• Each row's verdict comes from parse_label(judge_text) → one of "correct" | "incorrect" | "unattempted".
• grade_file returns exactly this shape:
  python复制

  {
      "task": task, "file": path, "n": n,
      "correct": c, "incorrect": inc, "unattempted": una,
      "errors": len(errs), "score": c / n if n else 0.0,
  }

• SimpleQA tasks go through the same per-row loop but with SIMPLEQA_GRADER_TEMPLATE and an A/B/C→correct/incorrect/unattempted mapping; nq/nq_tables/triviaqa use grade_exact_match, which returns the identical dict shape via string normalization instead of an LLM call.

That per-row verdict + aggregate dict is basically EvalPort's Result/ResultSet shape already, just not serialized as one.

The mapping

Sketch (real field names, not illustrative pseudo-fields)

from openeval.validate import validate_suite, validate_result_set

def row_to_test_case(task: str, i: int, row: dict) -> dict:
    od = row.get("original_data", {})
    return {
        "id": f"{task}_{i}",
        "input": row.get("problem", ""),
        "expected_output": build_ground_truth(task, od),  # from lib/grader.py, unmodified
        "graders": ["worldvqa_judge"],
    }

def row_to_result(task: str, i: int, row: dict, label: str) -> dict:
    return {
        "test_case_id": f"{task}_{i}",
        "grader_results": [{
            "grader_id": "worldvqa_judge",
            "score": {"correct": 1.0, "incorrect": 0.0, "unattempted": 0.0}[label],
            "passed": label == "correct",
            "metadata": {"pixelrag_label": label},  # keeps "unattempted" distinguishable from "failed"
        }],
        "passed": label == "correct",
    }

def result_set_from_grade_file(res: dict, rows: list[dict], labels: list[str], suite_id: str, run_id: str) -> dict:
    # res is grade_file()'s own return value — untouched.
    return {
        "version": "1.0.0",
        "suite_id": suite_id,
        "run_id": run_id,
        "results": [row_to_result(res["task"], i, r, l) for i, (r, l) in enumerate(zip(rows, labels))],
        "metadata": {"pixelrag": {"aggregate": res}},  # grade_file's dict, verbatim
    }

grader.py would need to hand back per-row labels alongside the aggregate (right now labels is a local in grade_file and only the aggregate is returned/printed) — a one-line change if this is worth doing, not a rewrite.

Precedent in EvalPort for this exact shape of problem

Two adapters that hit the same "framework returns real per-item scores + a real aggregate, don't fabricate either" constraint, verified by reading their code:

• lm-eval-harness-openeval-adapter — lm-eval's simple_evaluate(..., log_samples=True) already returns one real score per document; the adapter maps that 1:1 to Results and preserves the framework's own aggregate verbatim under metadata["lm_eval"]["aggregate"] — the same pattern proposed above.
• huggingface-evaluate-openeval-adapter — handles the harder case where the framework's API is aggregate-only by design, computing real per-example scores via repeated .compute() calls rather than interpolating from the aggregate.

grader.py's case is actually simpler than either: it already computes a real per-row verdict (labels[i]) before collapsing to the aggregate, so nothing needs to be recomputed — just retained and re-shaped.

What I'm asking

Not proposing a PR against this repo. Two honest options and I'd rather hear which (if either) is wanted before writing code:

1. A ~50-line pixelrag-openeval-adapter living entirely in the EvalPort repo (adapters/), depending on nothing from PixelRAG except the JSONL shape grade_file already reads/writes. Zero footprint here.
2. If useful upstream, a small opt-in --openeval-output <path> flag on grader.py's CLI that writes the ResultSet alongside the normal printed summary — happy to open that as an actual PR with tests if it's wanted.

If neither is useful, no worries — closing this is a fine outcome too.

— Sahi, independent contributor (not affiliated with this project)