#10690·sktime

[ENH] Containerized benchmarking

Author: jgyasuCreated Jul 31, 2026Updated Sep 16, 2026
Labelsenhancementmodule:metrics&benchmarking

Currently, benchmark experiments in sktime run as a single process over the Cartesian product of estimators and task objects. This can sometimes be problematic, especially in the case of FMs, since they tend to have very narrow or specific dependency sets. Running a benchmark with estimators that have conflicting dependency requirements can therefore lead to failures.

To solve this, each task-estimator pair should ideally be run in a separate virtual environment or container. This requires breaking up the benchmark experiment into smaller sub-experiments. However, we do not want users to have to set up 50 separate benchmark experiments if they want to benchmark 25 estimators on 2 tasks, for example.

We therefore need to think about how to internally generate and run these sub-experiments from a single user-defined benchmark, how to execute each sub-experiment in a separate container or virtual environment, and how the results from these independent runs will be collected and merged into the final benchmark result.

(This issue is for sktime internal staff)