Add Population Stability Index (PSI) as a model/data drift metric
[!WARNING] This issue is not yet ready for a PR. If you are interested in contributing to scikit-learn, please have a look at our contributing guidelines, and in particular the sections for new contributors and the "Needs triage" label.
Describe the workflow you want to enable
I want to detect when a feature's or a model score's distribution has shifted between two datasets (e.g. training vs. a more recent production sample). Using Population Stability Index - sum((new_pct - base_pct) * log(new_pct / base_pct)) over aligned bins/categories is a standard for doing so.
This is one of the most common "has my model gone stale" checks in production ML monitoring, sitting next to sklearn.metrics conceptually (it's a distance/divergence metric between two empirical distributions, similar in spirit to things like scipy.stats.entropy/KL divergence that sklearn already depends on via scipy).
Describe your proposed solution
sklearn.metrics.population_stability_index(base, new, bins=10) for continuous input (auto-binning via quantiles of base) and direct category alignment for categorical/discrete input, returning the scalar PSI (and optionally a per-bin breakdown table).
possibly also some rolling_psi method for evaluating the PSI over different time periods.
Describe alternatives you've considered, if relevant
Checked scikit-learn's and statsmodels' issue trackers directly for prior discussion of PSI - found none in either (no open request, nothing closed as wontfix). Purpose-built alternatives exist as separate packages, but nothing really maintained, while PSI is old and widely used operationally.
Additional context
Population Stability Index has been standard practice in credit scoring model monitoring since well before modern ML tooling (commonly attributed to the SAS/banking scorecard literature from the 1990s-2000s); today it's also used more broadly for feature/data drift detection outside credit scoring specifically.
Source: scikit-learn/scikit-learn