IdxMax and IdxMin return the position of NaN instead of the extreme value
Bug Description
IdxMax and IdxMin return the position of the extreme value inside the
rolling window. Both use ndarray.argmax / ndarray.argmin, which do not skip
NaN: as soon as the window holds a NaN, they return its position instead.
Max and Min go through pandas.rolling.max/min and do skip NaN, so the two
contradict each other on the same series. Mad and Rank in the same file
filter NaN explicitly (x[~np.isnan(x)]); IdxMax and IdxMin do not.
This reaches Alpha158. qlib/contrib/data/loader.py builds three field
families on these two operators:
| field | expression | meaning per the comment in the loader |
|---|---|---|
IMAX{d} |
IdxMax($high, d)/d |
days since the last high (Aroon) |
IMIN{d} |
IdxMin($low, d)/d |
days since the last low (Aroon) |
IMXD{d} |
(IdxMax($high,d)-IdxMin($low,d))/d |
distance between the two |
$high and $low are NaN while an instrument is suspended.
To Reproduce
- Run the script below on a clean checkout. It only replaces the data access of a leaf feature; the operators under test run their real code.
import numpy as np
import pandas as pd
from qlib.data.base import Feature
from qlib.data.ops import IdxMax, Max
class SeriesFeature(Feature):
def __init__(self, name, values):
super().__init__(name)
self._values = values
def _load_internal(self, instrument, start_index, end_index, freq):
return pd.Series(self._values, dtype="float64")
values = [1.0, 5.0, np.nan, 2.0, 3.0]
window = 3
idx = IdxMax(SeriesFeature("a", values), window).load("x", 0, len(values) - 1, "day")
mx = Max(SeriesFeature("b", values), window).load("x", 0, len(values) - 1, "day")
for i in range(len(values)):
w = values[max(0, i - window + 1) : i + 1]
print(f"{w} Max={mx.iloc[i]} IdxMax={idx.iloc[i]} points at {w[int(idx.iloc[i]) - 1]}")- Output on
be725493:
[1.0] Max=1.0 IdxMax=1.0 points at 1.0
[1.0, 5.0] Max=5.0 IdxMax=2.0 points at 5.0
[1.0, 5.0, nan] Max=5.0 IdxMax=3.0 points at nan
[5.0, nan, 2.0] Max=5.0 IdxMax=2.0 points at nan
[nan, 2.0, 3.0] Max=3.0 IdxMax=1.0 points at nanEvery window holding a NaN is wrong.
Expected Behavior
IdxMax points at the value Max reports for the same window, and IdxMin at
the value Min reports.
Environment
- Qlib version: 0.9.8.dev34 (editable install)
- Python version: 3.12.13
- OS: MacOS 26.6.2, arm64
- Commit number: be725493eb1a6bbb42bf11b37aa7669f59610ff1
- numpy 2.5.3, pandas 2.3.3
Additional Notes
Scale on real data. csi300, 30 instruments, 2019, qlib_data_simple:
| measurement | value |
|---|---|
| rows | 7320 |
NaN in $high |
76 (1.04 %) |
| instruments holding at least one NaN | 30 of 30 |
IMAX5 differing between argmax and nanargmax |
224 of 7314 (3.06 %) |
One NaN spoils the window for N days, so the effect is wider than the share of NaN itself.
Source: microsoft/qlib