EvalPort: portable interchange for ComputeAccuracy/ComputeSimilarity results

Hi LLaMA-Factory team — I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open JSON-Schema spec for portable LLM eval test cases, graders, and result sets.

src/llamafactory/train/sft/metric.py's ComputeAccuracy and ComputeSimilarity dataclasses consume each EvalPrediction and return a score_dict of accuracy/rouge-1/rouge-2/rouge-l/bleu-4 scores — close to EvalPort's GraderResult (one grader run per metric) aggregated into a ResultSet per eval run. Since LLaMA-Factory runs the same SFT eval loop across 100+ models, mapping these callables' output onto EvalPort's TestCase/GraderResult schema would let users compare SFT eval runs across models/datasets in a common format rather than parsing each run's raw Trainer log. Just flagging the fit, not asking you to prioritize it.

Schemas: https://github.com/adhabnr-ux/evalport/tree/main/spec/schemas

— Sahi, independent contributor (not affiliated with LLaMA-Factory)