GenericDataSource returns the last value of the series (not the previous one) for missing dates when the index holds datetime.date objects

Summary

GenericDataSource.get_data() with MissingDataStrategy.fill_forward (or interpolate) returns wildly wrong values for dates missing from the series, whenever the series index holds plain datetime.date objects instead of a pandas.DatetimeIndex.

The missing-date row is appended to the end of the series, the result of sort_index() is discarded, and ffill() then fills the new row with the final value of the entire series rather than the value of the preceding date.

In a backtest with PredefinedAssetEngine, whose timer walks pd.bdate_range (so every exchange holiday is a "missing date"), this silently marks-to-market — and fills orders — at prices from the far end of the data set. In our runs, Christmas 2018 was valued at August 2026 closes, producing multi-million swings in backtest.performance on every US holiday.

Reproduction

import datetime as dt
import pandas as pd
from gs_quant.backtests.data_sources import GenericDataSource, MissingDataStrategy

dates = [dt.date(2024, 1, d) for d in (2, 3, 4, 5, 8, 9, 10, 11, 12, 15)]
prices = pd.Series([100.0, 101, 102, 103, 104, 105, 106, 107, 108, 999.0], index=dates)

src = GenericDataSource(prices.copy(), MissingDataStrategy.fill_forward)
print(src.get_data(dt.date(2024, 1, 6)))   # missing date (Saturday)

Output (gs-quant 2.1.4, Python 3.12, pandas 2.x):

999.0        # <- the LAST value of the series; expected 103.0 (ffill of Jan 5)

The same lookup returns the correct 103.0 when the series is given a DatetimeIndex:

p2 = prices.copy()
p2.index = pd.DatetimeIndex([pd.Timestamp(d) for d in prices.index])
print(GenericDataSource(p2, MissingDataStrategy.fill_forward).get_data(dt.date(2024, 1, 6)))
# 103.0  (correct)

Root cause

gs_quant/backtests/data_sources.py, GenericDataSource.get_data():

if isinstance(self.data_set.index, pd.DatetimeIndex):
    self.data_set.at[pd.to_datetime(state)] = np.nan
    self.data_set = self.data_set.sort_index()      # assigned -> OK
else:
    self.data_set.at[state] = np.nan                # appended at the END
self.data_set.sort_index()                          # result DISCARDED (no-op)
if self.missing_data_strategy == MissingDataStrategy.interpolate:
    self.data_set = self.data_set.interpolate()
elif self.missing_data_strategy == MissingDataStrategy.fill_forward:
    self.data_set = self.data_set.ffill()           # fills tail NaN with the last value

In the non-DatetimeIndex branch the bare self.data_set.sort_index() call does nothing (sort_index is not in-place), so the injected NaN row stays at the tail of the series and ffill() propagates the last observation into it. interpolate() is similarly corrupted for the same reason.

Fix is one line — assign the sort result in the shared path:

self.data_set = self.data_set.sort_index()

(or move the assignment into the else branch to mirror the DatetimeIndex branch).

Related: caller's Series is mutated in place

A secondary effect of the same block: self.data_set.at[state] = np.nan writes into the Series object the caller passed to the constructor (no defensive copy is taken). After a single missing-date lookup, the caller's own Series contains injected NaN rows:

shared = pd.Series([100.0, 101, 102],
                   index=[dt.date(2024, 1, 2), dt.date(2024, 1, 3), dt.date(2024, 1, 4)])
GenericDataSource(shared, MissingDataStrategy.fill_forward).get_data(dt.date(2024, 1, 6))
print(shared.tail(2))
# 2024-01-04    102.0
# 2024-01-06      NaN   <- injected into the caller's object

If the caller also uses that Series for signal generation (a natural pattern with PredefinedAssetEngine), the NaN silently poisons downstream computations. Taking a copy in __post_init__, or avoiding the in-place .at write, would fix this.

Environment

• gs-quant 2.1.4 (also present on current master)
• Python 3.12, pandas 2.x, Windows 11

Impact

• Any PredefinedAssetEngine backtest using a date-indexed GenericDataSource with fill_forward/interpolate and a date range containing non-trading weekdays (i.e. virtually any realistic daily backtest) values holdings and fills orders at end-of-series prices on those dates.
• Both defects are silent: mark_to_market skips NaN holdings (abs(units) > epsilon is False for NaN), so corruption shows up only as inexplicable performance numbers, never as an exception.