#11196·sktime

[BUG] Detrender on unbalanced panels create nulls

Author: RobKueblerCreated Sep 18, 2026Updated Sep 18, 2026
Labelsbug

Describe the bug

Detrender.fit_transform produces NaN rows when applied to an unbalanced panel (e.g. different lengths).

The root cause is in Detrender._get_fh_from_X (sktime/transformations/detrend/_detrend.py):

python
def _get_fh_from_X(self, X):
    """Obtain fh from X, which can be simple or hierarchical."""
    if not isinstance(X.index, pd.MultiIndex):
        time_index = X.index
    else:
        time_index = X.index.get_level_values(-1).unique()
    ...

For panel data, this takes the union of all time-index values across every instance and builds a single shared ForecastingHorizon from it. That fh is then used for all instances via forecaster.predict(fh=fh, ...). If instance A does not have observations at every timestamp instance B has, forecaster.predict still produces a row for instance A at those timestamps. X - X_pred (in _transform) then aligns on the union of both indices, so the timestamps that exist in X_pred but not in the original X show up as new rows filled with NaN.

In other words, the transform does not treat each series in the panel independently: it silently pools their time indices into one shared forecasting horizon, and instances that don't cover the full union of timestamps get spurious NaN rows appended (row count of the output can even be larger than the input, as shown below: 9 input rows -> 10 output rows).

I verified this is purely an indexing/fh bug, not a mis-fit of the trend itself: even with two instances that have very different slopes, the actual detrended (non-NaN) values are correct per-instance — only the index handling is broken.

To Reproduce

python
import pandas as pd
import numpy as np

from sktime.transformations.detrend import Detrender
from sktime.forecasting.trend import TrendForecaster

# Panel of 2 time series with DIFFERENT lengths (4 and 5),
# as a pandas DataFrame with a 2-level MultiIndex (time index last),
# which is the standard "pd-multiindex" sktime Panel mtype.
idx1 = pd.MultiIndex.from_product([[0], range(4)], names=["instance", "time"])
idx2 = pd.MultiIndex.from_product([[1], range(5)], names=["instance", "time"])

s1 = pd.Series([10.0, 20.0, 30.0, 40.0], index=idx1)   # linear trend, slope 10
s2 = pd.Series([1.0, 2.0, 3.0, 4.0, 5.0], index=idx2)  # linear trend, slope 1

panel = pd.concat([s1, s2]).to_frame(name="value")
print(panel)

t = Detrender(forecaster=TrendForecaster())
out = t.fit_transform(panel)
print(out)

print("input shape:", panel.shape)
print("output shape:", out.shape)
print("any NaN:", out.isna().any().any())

Output:

               value
instance time       
0        0      10.0
         1      20.0
         2      30.0
         3      40.0
1        0       1.0
         1       2.0
         2       3.0
         3       4.0
         4       5.0
               value
instance time       
0        0       0.0
         1       0.0
         2       0.0
         3       0.0
         4       NaN
1        0       0.0
         1       0.0
         2       0.0
         3       0.0
         4       0.0

input shape: (9, 1)
output shape: (10, 1)
any NaN: True

Instance 0 only has 4 observations (time 0-3), but the output contains a 5th row for it at time=4 that never existed in the input, filled with NaN. The output also has more rows than the input (10 vs 9).

Expected behavior

Each time series (instance) in the panel should be detrended independently, using only its own time index. The transformed panel should:

  • have exactly the same index as the input (same instances, same per-instance timestamps, same shape: (9, 1) in the example above),
  • contain no NaN values that weren't already in the input,
  • not "invent" timestamps for an instance that never had observations there just because another instance in the panel did.

Additional context

  • This only manifests on Panel-typed X (pd-multiindex / pd_multiindex_hier mtypes), i.e. when instances do not all share an identical time index. On plain Series input, or a panel where every instance happens to share the exact same time index, the bug does not trigger, since the "union of time indices" then coincides with each instance's own index.
  • Detrender declares "scitype:instancewise": True in its tags, which suggests each instance is expected to be handled independently, but X_inner_mtype includes "pd-multiindex" / "pd_multiindex_hier" directly (not just "pd.DataFrame"), so _fit/_transform receive the whole panel at once rather than being vectorized/looped over instances by the base class. The bug is that _get_fh_from_X then computes a single, shared fh for the whole panel instead of a per-instance fh.
  • Confirmed that the underlying forecaster fit itself is correct per-instance (tested with two instances with very different slopes — the non-NaN residuals were correct for both), so the fix should be scoped to how fh/prediction indices are derived and aligned per instance, rather than to the forecasting logic itself.

Versions

System: python: 3.14.3 (main, Mar 26 2026, 00:00:00) [GCC 16.0.1 20260321 (Red Hat 16.0.1-0)] executable: /home/robert/Projects/sktime/.venv/bin/python machine: Linux-6.19.10-300.fc44.x86_64-x86_64-with-glibc2.43

Python dependencies: pip: 26.2.1 sktime: 1.1.0 scikit-learn: 1.7.2 scikit-base: 1.0.2 numpy: 2.4.6 scipy: 1.18.1 pandas: 2.3.3 matplotlib: 3.11.2 joblib: 1.5.3 huggingface-hub: None numba: 0.67.0 pmdarima: None pytorch-forecasting: None skforecast: None skpro: 2.14.0 statsforecast: None statsmodels: 0.15.0 transformers: None tsfresh: None tslearn: None torch: None tensorflow: None