Idea: version EvalPort eval-suites as ClearML Datasets (lineage-tracked eval regression sets)

Hi ClearML team — floating a lightweight idea, not a feature request against clearml core.

EvalPort (https://github.com/adhabnr-ux/evalport) is an open spec + JSON Schema (with Python/TS SDKs) for portable LLM eval artifacts — test suites, test cases, and result sets — meant to keep eval data from being locked into one platform's format.

Looking at clearml/datasets/dataset.py's actual API (Dataset.create(dataset_name, dataset_project, dataset_tags, parent_datasets=...), .add_files(path), .finalize(), Dataset.get(dataset_id=..., dataset_tags=...), .get_local_copy()), it already does exactly what an EvalPort test suite needs for storage: versioned, tagged, lineage-tracked JSON artifacts. A code-only pattern, no ClearML changes needed:

from clearml import Dataset

ds = Dataset.create(
    dataset_name="regression-suite-v3",
    dataset_project="llm-evals",
    dataset_tags=["evalport", "testsuite"],
)
ds.add_files("testsuite.json")  # validated against EvalPort's TestSuite schema
ds.finalize()

# later, in a Task that runs the eval:
suite_path = Dataset.get(dataset_tags=["evalport", "testsuite"], dataset_name="regression-suite-v3").get_local_copy()
# ...run the eval, then version the results as a child dataset:
results_ds = Dataset.create(
    dataset_name="regression-suite-v3-results",
    parent_datasets=[ds.id],
    dataset_tags=["evalport", "resultset"],
)

That gives full lineage from suite -> eval run -> result set using ClearML's own primitives — an EvalPort ResultSet is just a JSON file whose dataset has parent_datasets pointing back at the suite it was scored against.

Not asking for any code change here, just wanted to check whether this resonates before writing up a short example. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md. Feel free to close if it's not a fit — thanks for ClearML either way.