#2137·MetaGPT

Optional EvalPort interop for AFlow benchmark datasets and results

Author: adhabnr-uxCreated Aug 22, 2026Updated Aug 22, 2026

Hi MetaGPT team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first before writing any code.

I read the real code in metagpt/ext/aflow/benchmark/ rather than guessing:

  • BaseBenchmark.load_data() reads a JSONL file of problems — each row is functionally EvalPort's TestCase (an input plus an expected/reference answer), and the file as a whole is a Suite.
  • BaseBenchmark.evaluate_problem() / calculate_score() grade a prediction against the expected output and return a PASS/FAIL plus a score — this is EvalPort's Grader output shape.
  • BaseBenchmark.run_evaluation() aggregates those into an average score, average cost, and total cost via save_results_to_csv() — this is functionally a ResultSet.

So an AFlow benchmark (gsm8k/math/mbpp/humaneval/hotpotqa/drop are already implemented as BaseBenchmark subclasses) could export its problem set and graded results as EvalPort JSON, letting it move to another eval tool without a bespoke converter, or letting AFlow load a Suite built elsewhere as a benchmark.

Two ways I could see this landing, and I don't have a strong preference:

  1. A standalone metagpt-openeval-adapter package in the EvalPort repo, depending on metagpt as a normal dependency. Zero footprint on this repo.
  2. A small optional module inside this repo if you'd rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.

Source: FoundationAgents/MetaGPT