Idea: EvalPort ResultSet as a portable export for compare_runs.py-style comparisons

Hi SWE-agent team,

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open spec (Apache 2.0) for portable LLM evaluation test suites and result sets — JSON Schemas plus Python/TS SDKs that validate against them.

I was reading through sweagent/run/ and noticed two things that line up closely with what EvalPort is for:

• extract_pred.py produces the SWE-bench prediction shape (instance_id, model_patch, model_name_or_path) from a .traj file when the .pred wasn't saved.
• compare_runs.py reads results.json files (submitted_ids / resolved_ids) and does per-instance pass/fail comparison across one or many runs — exactly the "which instances did this run get right vs. that run" problem EvalPort's ResultSet schema (spec/schemas/resultset.json) is built to standardize, just scoped to SWE-agent's own runs today.

A small to_evalport() converter over results.json would let someone line up SWE-agent's SWE-bench runs against results from a different agent/harness that also emits EvalPort ResultSets — same comparison compare_runs.py already does, just portable across tools instead of only across your own run directories. Rough sketch using the real required fields from the schema:

import json
from pathlib import Path

def to_evalport(results_path: str, run_id: str) -> dict:
    data = json.loads(Path(results_path).read_text())
    submitted = set(data["submitted_ids"])
    resolved = set(data.get("resolved_ids", data.get("resolved", [])))
    return {
        "version": "1.0.0",
        "suite_id": "swe-bench",
        "run_id": run_id,
        "started_at": "<from run metadata>",
        "results": [
            {
                "test_case_id": iid,
                "passed": iid in resolved,
                "grader_results": [{
                    "grader_id": "resolved", "type": "custom",
                    "score": 1.0 if iid in resolved else 0.0,
                    "passed": iid in resolved,
                }],
            }
            for iid in sorted(submitted)
        ],
        "summary": {
            "total": len(submitted),
            "passed": len(resolved),
            "pass_rate": len(resolved) / len(submitted) if submitted else 0,
        },
    }

Full spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

No pressure if this isn't a priority — compare_runs.py already does the hard part internally, I just wanted to flag the portability angle since it seemed like a short hop. Happy to sketch a real converter PR against a sample results.json if there's interest.

Thanks for SWE-agent — digging through the run/ harness to write this was a good excuse to read the code.