#2499·doccano

Optional EvalPort interop for Example/Label export data

Author: adhabnr-uxCreated Aug 22, 2026Updated Aug 23, 2026

Hi doccano team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first before writing any code.

I read the real backend models rather than guessing (backend/examples/models.py, backend/labels/models.py on master):

  • Example (backend/examples/models.py) has text, meta (JSON), and a score field (FloatField, default 100).
  • The abstract Label base (backend/labels/models.py), which Category, Span, Relation, TextLabel, BoundingBox, and Segmentation all extend, carries prob (FloatField, default 0.0 — the auto-labeling confidence) and manual (BooleanField — whether the label was manually confirmed vs. machine-suggested), each tied back to its example.

That maps onto EvalPort's two halves fairly directly: an Example (text + meta) is a TestCase, a project's examples form a Suite, and each Example's attached Labels — with their real prob confidence and manual flag — are exactly the shape of an EvalPort Grader result (a per-TestCase score plus a pass/fail-style signal) that a ResultSet could be built from.

Two ways I could see this landing, and I don't have a strong preference:

  1. A standalone doccano-openeval-adapter package in the EvalPort repo, talking to a doccano project over its REST API and depending on nothing from this repo directly. Zero footprint on this repo.
  2. A small optional export module inside this repo if you'd rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.