Rolling computes the wrong window when offset is not the negation of period (in-memory engine)
Checks
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of Polars.
Reproducible example
from datetime import datetime
import polars as pl
def rolling(ts, period, offset, time_zone=None):
lf = pl.LazyFrame({"t": ts})
if time_zone is not None:
lf = lf.with_columns(pl.col("t").dt.replace_time_zone(time_zone))
lf = lf.rolling(index_column="t", period=period, offset=offset).agg(pl.len())
return lf.collect()["len"].to_list(), lf.collect(engine="streaming")["len"].to_list()
# Case 1 -- calendar month. offset="-1mo", period="28d".
# The window of the row at t is (t + offset, t + offset + period]:
# 2024-03-01 -> (2024-02-01, 2024-02-29] -> 0
# 2024-03-31 -> (2024-02-29, 2024-03-28] -> 1 (2024-03-01)
# 2024-04-01 -> (2024-03-01, 2024-03-29] -> 0
mem, stream = rolling(
[datetime(2024, 3, 1), datetime(2024, 3, 31), datetime(2024, 4, 1)], "28d", "-1mo"
)
print(f"month in-memory={mem} streaming={stream} expected=[0, 1, 0]")
# Case 2 -- daylight saving. offset="-1d", period="24h", Europe/Amsterdam.
# Clocks go forward 2024-03-31 02:00 -> 03:00, so that calendar day is 23h long and
# 2024-03-31 00:00 + 24h lands at 2024-04-01 01:00. For t = 2024-04-01 00:00 the window is
# (2024-03-31 00:00, 2024-04-01 01:00]
# which holds both 2024-04-01 00:00 and 2024-04-01 01:00 -> 2.
mem, stream = rolling(
[datetime(2024, 3, 31), datetime(2024, 4, 1), datetime(2024, 4, 1, 1)],
"24h",
"-1d",
time_zone="Europe/Amsterdam",
)
print(f"dst in-memory={mem} streaming={stream} expected=[1, 2, 2]")Log output
month in-memory=[1, 2, 2] streaming=[0, 1, 0] expected=[0, 1, 0]
dst in-memory=[1, 1, 2] streaming=[1, 2, 2] expected=[1, 2, 2]Issue description
Report by an agent while implementing distributed rolling.
The in-memory engine gives the wrong window contents whenever offset and period are different durations that happen to share the same nanosecond estimate. The streaming engine is correct in both cases above.
The cause is a chain of three steps:
- Duration::duration_ns() is an estimate — it scores a month as exactly 28 days and a day as exactly 86 400 s, so it is blind to both variable month length and DST.
- group_by_values picks the fast lookbehind path on offset.duration_ns() == period.duration_ns(), using that approximate test to establish an exact property.
- That path assumes t is the window's right edge and takes the shortcut let upper = t, whose own comment states the precondition it needs: "We have period == -offset". When the precondition does not actually hold, every window gets the wrong upper bound.
-1mo vs 28d and -1d vs 24h both satisfy step 2 without being inverses, which is why two unrelated-looking inputs fail the same way. The error direction depends on whether the calendar unit is longer or shorter than its estimate: the month case makes the window too wide (over-counts), the DST case too narrow (under-counts).
The guard inside the iterator is a debug_assert! restating the same approximate condition, so it cannot catch the misclassification even in debug builds. Two further branch conditions further down group_by_values compare duration_ns() the same way, so the shortcut branch is unlikely to be the only place this can misclassify.
Streaming is unaffected because RollingWindower tests the property structurally, if self.offset == -self.period, and otherwise derives the bound with real date arithmetic. Neg only flips the negative flag and keeps the fields, so that test correctly distinguishes -1mo from -28d.
Scope. Any in-memory rolling where offset is passed explicitly and is not the structural negation of period. The default offset=None becomes exactly -period, so ordinary usage is safe, and rolling_*_by is safe too since it builds offset by negating period. Note that #22980 would widen this, by making the shortcut reachable from rolling expressions as well.
Suggested fix. Use the structural test offset == -period for the branch condition, matching RollingWindower, and change the debug_assert! to that same condition so it becomes a real check.
Expected behavior
In-memory engine behavior matches the correct streaming engine behavior.
Installed versions
Replace this line with the output of pl.show_versions(). Leave the backticks in place.Source: pola-rs/polars