Optional EvalPort interop for AFlow benchmark datasets and results
Hi MetaGPT team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first before writing any code.
I read the real code in metagpt/ext/aflow/benchmark/ rather than guessing:
BaseBenchmark.load_data()reads a JSONL file of problems — each row is functionally EvalPort'sTestCase(an input plus an expected/reference answer), and the file as a whole is aSuite.BaseBenchmark.evaluate_problem()/calculate_score()grade a prediction against the expected output and return aPASS/FAILplus a score — this is EvalPort'sGraderoutput shape.BaseBenchmark.run_evaluation()aggregates those into an average score, average cost, and total cost viasave_results_to_csv()— this is functionally aResultSet.
So an AFlow benchmark (gsm8k/math/mbpp/humaneval/hotpotqa/drop are already implemented as BaseBenchmark subclasses) could export its problem set and graded results as EvalPort JSON, letting it move to another eval tool without a bespoke converter, or letting AFlow load a Suite built elsewhere as a benchmark.
Two ways I could see this landing, and I don't have a strong preference:
- A standalone
metagpt-openeval-adapterpackage in the EvalPort repo, depending onmetagptas a normal dependency. Zero footprint on this repo. - A small optional module inside this repo if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
Source: FoundationAgents/MetaGPT