#10672·sktime

[ENH] Regression tests validating foundation model outputs against original/source implementations

Author: geetu040Created Jul 28, 2026Updated Sep 17, 2026
Labelsgood first issuemodule:testsenhancement

Description

Add regression tests that validate sktime estimator outputs against outputs from the original/source implementation (3rd-party package, paper code, etc.), not only API-level checks such as shape, dtype, or index.

sktime estimators can silently change behavior compared to their source implementation without any test detecting it. For example, #10626 showed that TinyTimeMixerForecaster had diverged from Granite-TSFM due to padding-mask differences. This was only discovered through a manual comparison.

These tests should:

  • Ensure that sktime matches the source implementation.
  • Catch future behavior changes early, instead of allowing subtle discrepancies to become hard-to-debug issues later.

This is especially important for foundation model wrappers (TTM, Chronos, Moirai, etc.). These models use pretrained weights in a zero-shot setting, so small implementation differences are not corrected through training. Differences in tokenization, scaling, masking, preprocessing, etc. directly affect the final forecast.

What to do for each estimator

  • Use a small fixed dataset (load_airline preferred for consistency; random data is acceptable if more suitable).
  • Generate reference outputs directly from the source implementation, without using sktime.
  • Store only a small slice of the output, not the complete prediction.
  • Add a sktime regression test using the same data and parameters, and compare with np.testing.assert_allclose.
  • If outputs do not match, investigate and fix the underlying issue instead of only increasing tolerances.
  • Cover checkpoints/variants that exercise different code paths.
  • Test the complete estimator pipeline end-to-end; do not mock internal components.
  • If a new test file is created, update the estimator tag tests:specific to include the new test file.
  • Use run_test_for_class for skipping the pytest

Reference implementation

See #10627 / TinyTimeMixerForecaster golden-output tests.

Estimators to cover

  • Chronos2Forecaster #10677
  • ChronosForecaster #10692
  • FalconTSTForecaster
  • FlowStateForecaster
  • HFTransformersForecaster #10751
  • KronosForecaster
  • LagLlamaForecaster #10682
  • MantisForecaster #10736
  • MOIRAIForecaster #10686
  • Moirai2Forecaster #10687
  • MomentFMForecaster #10685
  • PatchTSMixerForecaster
  • PatchTSTForecaster
  • TiRexForecaster
  • TimeLLMForecaster
  • TimeMoEForecaster
  • TimerForecaster
  • TimerS1Forecaster
  • TinyTimeMixerForecaster #10627
  • TimesFM2Forecaster #10674
  • TimesFMForecaster
  • TotoForecaster #10684
  • Toto2Forecaster #10683
  • WindFMForecaster #11043

More estimators can be added once the pattern is established and the initial set of tests is merged.

To contribute, pick one estimator from the list above and open a PR adding the corresponding regression tests.

In the PR description, include the source implementation/code that was used to generate the reference outputs. This should make it clear how the expected values in the tests were produced and allow others to reproduce or verify the reference outputs in the future.