[ENH] ensure codecov is correct given partition/matrix design testing
Since we've introduced partition/matrix design in testing (https://github.com/alan-turing-institute/sktime/pull/3137 - estimators are distributed across os/version so that each os and each version sees each estimator at least once), the good news is that test time is down to 30-50min per CI item, from 1-2 horus.
The bad news is that codecov seems to be affected by this, generally reporting a lower percentage than actual, since not all lines are touched in a given CI item, and they seem to overwrite each other's codecov report.
There is also a minor effect that some code is run only conditionally, e.g., estimators with a python version bound are run only on compatible python versions.
What one would instead expect to happen is that the final codecov report takes the union of lines touched across all CI items (at least once), rather than the more or less randomly/arbitrarily picked report of the CI item that runs longest.
How to do this, I'm not sure, but perhaps someone with good codecov experience sees an easy way to do this. Possibly only a few lines of code change in the CI, for someone who has the right background...
Source: sktime/sktime