Optional EvalPort interop for Example/Label export data
Hi doccano team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first before writing any code.
I read the real backend models rather than guessing (backend/examples/models.py, backend/labels/models.py on master):
Example(backend/examples/models.py) hastext,meta(JSON), and ascorefield (FloatField, default 100).- The abstract
Labelbase (backend/labels/models.py), whichCategory,Span,Relation,TextLabel,BoundingBox, andSegmentationall extend, carriesprob(FloatField, default 0.0 — the auto-labeling confidence) andmanual(BooleanField— whether the label was manually confirmed vs. machine-suggested), each tied back to itsexample.
That maps onto EvalPort's two halves fairly directly: an Example (text + meta) is a TestCase, a project's examples form a Suite, and each Example's attached Labels — with their real prob confidence and manual flag — are exactly the shape of an EvalPort Grader result (a per-TestCase score plus a pass/fail-style signal) that a ResultSet could be built from.
Two ways I could see this landing, and I don't have a strong preference:
- A standalone
doccano-openeval-adapterpackage in the EvalPort repo, talking to a doccano project over its REST API and depending on nothing from this repo directly. Zero footprint on this repo. - A small optional export module inside this repo if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
Source: doccano/doccano