permutation_importance computes non-zero importance for features the model never uses when max_samples < 1

[!WARNING] This issue is not yet ready for a PR. If you are interested in contributing to scikit-learn, please have a look at our contributing guidelines, and in particular the sections for new contributors and the "Needs triage" label.

Introduce yourself

I'm an undergraduate economics student at UC Davis. I use scikit-learn for data analysis and use permutation importance to understand which features matter. This is my first issue filed against scikit-learn.

Describe the bug and give evidence about its user-facing impact

What: When max_samples is set below 1, features that the model never uses still get non-zero permutation importances. By definition, shuffling a column the model does not look at cannot change the predictions, so its importance should be exactly 0. With max_samples=1.0 the importance is indeed 0; the problem only appears when a subsample is used.

My example: The model only uses feature 0; features 1 and 2 are pure noise that the model never reads. With max_samples=0.5, both features 1 and 2 get an importance of −0.0104. The 5 repetitions (n_repeats=5) return exactly the same value, so the standard deviation is 0. The bias changes in magnitude and sign with the subsample size: −0.026 at 0.2 and +0.006 at 0.1. This shows it is sampling noise, not a real feature contribution.

Cause (I read sklearn/inspection/_permutation_importance.py): The baseline score is computed on all rows, while the shuffled score is computed on the subsampled rows only, and the two are subtracted. The difference therefore mixes in the gap between the full-data score and the subsample score, which has nothing to do with the feature. The docstring says a fresh sample is drawn for each repetition; in practice the subsample is drawn once per feature before the repetitions, and all features share the same random seed — so repeating does not average the bias away (std = 0), and every irrelevant feature gets the same bias.

Impact on users: max_samples exists to save time on large datasets. With this bug, irrelevant features get spuriously positive or negative importances, which can make them look useful or harmful, and can also corrupt feature ranking.

AI usage disclosure: I used an AI assistant to help identify this problem and write the reproduction script. The description above is my own, and the reproduction was run on my own computer.

Steps/Code to Reproduce

import numpy as np, sklearn
from sklearn.linear_model import LinearRegression
from sklearn.inspection import permutation_importance
from sklearn.metrics import r2_score
print(sklearn.__version__)
rng = np.random.RandomState(0)
n = 200
X = rng.randn(n, 3)
y = 3 * X[:, 0] + rng.randn(n)          # only feature 0 matters
X[:, 1:] = rng.randn(n, 2)                # features 1, 2: pure noise
model = LinearRegression().fit(X[:, :1], y)  # model uses ONLY feature 0 ...
class OnlyFirst:                              # ... wrapped so it accepts 3 columns
    def fit(self, X, y): return self
    def predict(self, X): return model.predict(X[:, :1])
    def score(self, X, y): return r2_score(y, self.predict(X))
est = OnlyFirst()

r = permutation_importance(est, X, y, n_repeats=5, max_samples=0.5, random_state=0)
print("importances_mean (max_samples=0.5):", r.importances_mean)
print("importances (features 1,2 are ignored by the model, so must be exactly 0):")
print(r.importances[1:])

# Independent expectation: permuting a column the model never reads cannot change
# predictions, so the permuted score equals the score on the SAME rows -> importance 0.
r_full = permutation_importance(est, X, y, n_repeats=5, max_samples=1.0, random_state=0)
print("importances_mean (max_samples=1.0):", r_full.importances_mean)
print("baseline (all rows) r2:", r2_score(y, est.predict(X)))
for ms in (0.2, 0.1):
    r = permutation_importance(est, X, y, n_repeats=5, max_samples=ms, random_state=0)
    print(f"max_samples={ms}: importances_mean =", r.importances_mean, " importances_std =", r.importances_std)

Expected Results

For every repetition, the importances of features 1 and 2 should be exactly 0 — identical to the result with max_samples=1.0.

Actual Results

1.9.1
importances_mean (max_samples=0.5): [ 1.7655429  -0.01042625 -0.01042625]
importances (features 1,2 are ignored by the model, so must be exactly 0):
[[-0.01042625 -0.01042625 -0.01042625 -0.01042625 -0.01042625]
 [-0.01042625 -0.01042625 -0.01042625 -0.01042625 -0.01042625]]
importances_mean (max_samples=1.0): [1.79471992 0.         0.        ]
baseline (all rows) r2: 0.9095261981105188
max_samples=0.2: importances_mean = [ 1.76648851 -0.02631336 -0.02631336]  importances_std = [0.30605997 0.         0.        ]
max_samples=0.1: importances_mean = [1.66914758 0.00620688 0.00620688]  importances_std = [0.73546823 0.         0.        ]

Versions

System:
    python: 3.14.7 (main, Aug  5 2026, 10:29:49) [Clang 21.0.0 (clang-2100.1.1.101)]
executable: /Users/luca/Projects/oss-contrib/.venv-skl/bin/python
   machine: macOS-26.6.2-arm64-arm-64bit-Mach-O

Python dependencies:
      sklearn: 1.9.1
          pip: 26.2.1
   setuptools: None
        numpy: 2.5.3
        scipy: 1.18.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.6.0
threadpoolctl: 3.7.0
     narwhals: 2.26.0

Built with OpenMP: True

Interest in fixing the bug

Yes, I'd be willing to fix this. Idea: draw a fresh subsample for every repetition, and compute both the baseline and the shuffled scores on the same subsampled rows before subtracting. Add a regression test asserting that features the model does not use have exactly 0 importance when max_samples < 1.