[ENH] Regression tests validating foundation model outputs against original/source implementations
Description
Add regression tests that validate sktime estimator outputs against outputs from the original/source implementation (3rd-party package, paper code, etc.), not only API-level checks such as shape, dtype, or index.
sktime estimators can silently change behavior compared to their source implementation without any test detecting it. For example, #10626 showed that TinyTimeMixerForecaster had diverged from Granite-TSFM due to padding-mask differences. This was only discovered through a manual comparison.
These tests should:
- Ensure that sktime matches the source implementation.
- Catch future behavior changes early, instead of allowing subtle discrepancies to become hard-to-debug issues later.
This is especially important for foundation model wrappers (TTM, Chronos, Moirai, etc.). These models use pretrained weights in a zero-shot setting, so small implementation differences are not corrected through training. Differences in tokenization, scaling, masking, preprocessing, etc. directly affect the final forecast.
What to do for each estimator
- Use a small fixed dataset (
load_airlinepreferred for consistency; random data is acceptable if more suitable). - Generate reference outputs directly from the source implementation, without using sktime.
- Store only a small slice of the output, not the complete prediction.
- Add a sktime regression test using the same data and parameters, and compare with
np.testing.assert_allclose. - If outputs do not match, investigate and fix the underlying issue instead of only increasing tolerances.
- Cover checkpoints/variants that exercise different code paths.
- Test the complete estimator pipeline end-to-end; do not mock internal components.
- If a new test file is created, update the estimator tag
tests:specificto include the new test file. - Use
run_test_for_classfor skipping the pytest
Reference implementation
See #10627 / TinyTimeMixerForecaster golden-output tests.
Estimators to cover
-
Chronos2Forecaster#10677 -
ChronosForecaster#10692 -
FalconTSTForecaster -
FlowStateForecaster -
HFTransformersForecaster#10751 -
KronosForecaster -
LagLlamaForecaster#10682 -
MantisForecaster#10736 -
MOIRAIForecaster#10686 -
Moirai2Forecaster#10687 -
MomentFMForecaster#10685 -
PatchTSMixerForecaster -
PatchTSTForecaster -
TiRexForecaster -
TimeLLMForecaster -
TimeMoEForecaster -
TimerForecaster -
TimerS1Forecaster -
TinyTimeMixerForecaster#10627 -
TimesFM2Forecaster#10674 -
TimesFMForecaster -
TotoForecaster#10684 -
Toto2Forecaster#10683 -
WindFMForecaster#11043
More estimators can be added once the pattern is established and the initial set of tests is merged.
To contribute, pick one estimator from the list above and open a PR adding the corresponding regression tests.
In the PR description, include the source implementation/code that was used to generate the reference outputs. This should make it clear how the expected values in the tests were produced and allow others to reproduce or verify the reference outputs in the future.
Source: sktime/sktime