#2315 Benchmark weighting: default backtest compares an equal-weighted portfolio against a cap-weighted index

1. Summary

In the default backtest configuration, TopkDropoutStrategy allocates cash equally across holdings while the reported excess return subtracts a cap-weighted index (SH000300) as a plain difference. Over 2017-01-03..2020-07-31 that weighting mismatch alone is worth -4.13% per year (information ratio -0.723) with no alpha involved, measured by simply holding the point-in-time CSI300 constituents equally weighted. Because equal weighting underperformed the cap-weighted index in this window, the offset makes reported model excess returns lower than a weighting-consistent comparison would show, so this is a measurement-convention question rather than a result that flatters Qlib.

2. Evidence in code

Repository commit 79633dd9506ea689e5400dea0197717b5b3d74b7 (2026-07-23).

Portfolio is equal-weighted by cash. qlib/contrib/strategy/signal_strategy.py:266
```
value = cash * self.risk_degree / len(buy) if len(buy) > 0 else 0
```

Excess return is a plain difference against the benchmark series. qlib/workflow/record_temp.py:507-512

analysis["excess_return_without_cost"] = risk_analysis(
    report_normal["return"] - report_normal["bench"], freq=_analysis_freq
)
analysis["excess_return_with_cost"] = risk_analysis(
    report_normal["return"] - report_normal["bench"] - report_normal["cost"], freq=_analysis_freq
)

The shipped example configs set a cap-weighted index as that benchmark. examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml:5 (benchmark: &benchmark SH000300) and line 23 inside backtest:. The same default appears in qlib/workflow/record_temp.py:410.

An equal-weight benchmark is anticipated but not implemented. qlib/backtest/profit_attribution.py:238

:param bench: The benchmark for comparing. TODO: if no benchmark is set, the equal-weighted is used.

I searched the GitHub issue tracker via the search API for existing discussion of benchmark weighting and did not find a matching issue; apologies if I missed one.

3. Measurement

3a. Model-independent control

To isolate the weighting convention from any alpha, I held the point-in-time CSI300 constituents equally weighted and compared them with SH000300 over the same window the example configs backtest.

Window: 2017-01-03 .. 2020-07-31, 871 trading days
Universe: D.instruments(market="csi300"), so Qlib's own membership spans apply
Constituents with a usable return per day: median 291, max 300
Statistics from Qlib's own qlib.contrib.evaluate.risk_analysis

Series	Daily mean	Daily std	Annualized return	Information ratio	Max drawdown
Equal-weight CSI300	0.000355	0.012662	+8.44%	0.432	-39.96%
`SH000300` (cap-weighted)	0.000467	0.012298	+11.10%	0.585	-37.05%
Equal-weight minus cap-weighted	-0.000174	0.003708	-4.13%	-0.723	-19.50%

No model is involved in this table.

3b. Same effect on an actual model run

I then ran examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml (single seed, seed=0) and re-measured the identical daily return series against both benchmarks. As a sanity check, my recomputed SH000300 daily return matched the bench column Qlib produced to max absolute difference 0.00e+00.

Metric	vs `SH000300` (cap-weighted)	vs equal-weight CSI300	Difference
Annualized excess, without cost	+17.33%	+22.51%	+5.18 pt
Annualized excess, with cost	+12.80%	+17.96%	+5.16 pt
Information ratio, without cost	2.005	3.347	+1.342
Information ratio, with cost	1.481	2.670	+1.189
Daily std of excess	0.005605	0.004360	-22.2%
Max drawdown, with cost	-9.94%	-4.25%	—

Single-seed IC figures for that run: IC 0.0475, ICIR 0.3892, Rank IC 0.0506, Rank ICIR 0.4213. examples/benchmarks/README.md:45 reports IC 0.0448±0.00, ICIR 0.3660±0.00, Rank IC 0.0469±0.00, Rank ICIR 0.3877±0.00, annualized return 0.0901±0.00, IR 1.0164±0.00, max drawdown -0.1038±0.00 over 20 seeds. My IC-family numbers are close but not inside the quoted ±0.00 bands, and my backtest annualized return (0.1280 with cost) differs from the README's 0.0901. I did not chase that gap down; it may come from the seed count, the dataset snapshot, or environment differences, and it does not affect the comparison in the table above, which uses one return series measured two ways.

4. Implication

For any window in which equal weighting and cap weighting diverge, excess_return_with_cost and excess_return_without_cost carry a systematic component that reflects the weighting convention rather than the model. In this window that component is about -4.13% per year, i.e. reported excess returns for equal-weighted strategies are understated by roughly that amount. The sign is window-dependent: in a period where equal weighting outperforms, the same mismatch would overstate them instead.

5. Reproduction

Dataset: v2/qlib_data_cn_1d_latest.zip, 196.5 MB

  python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

  import qlib
  from qlib.constant import REG_CN
  from qlib.data import D
  from qlib.contrib.evaluate import risk_analysis

  qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)

  START, END = "2017-01-01", "2020-08-01"

  px = D.features(D.instruments("csi300"), ["$close"],
                  start_time=START, end_time=END, freq="day")["$close"].unstack("instrument")
  # NOTE: pandas defaults pct_change to fill_method="pad", which turns a suspended name into a
  # 0% day and drags the cross-sectional mean toward zero. With the default the median number
  # of constituents contributing a return per day is 376; with fill_method=None it is 291.
  ew = px.pct_change(fill_method=None).mean(axis=1, skipna=True)

  bm = D.features(["SH000300"], ["$close"], start_time=START, end_time=END,
                  freq="day")["$close"].unstack("instrument")["SH000300"].pct_change(fill_method=None)

  idx = ew.index.intersection(bm.index)
  print(risk_analysis(ew.reindex(idx).dropna()))
  print(risk_analysis(bm.reindex(idx).dropna()))
  print(risk_analysis((ew.reindex(idx) - bm.reindex(idx)).dropna()))

Environment: Windows 11, Python 3.12.8 in a clean venv, pyqlib 0.9.7 from PyPI, numpy 1.26.4, pandas 2.1.4, lightgbm 4.7.0.

Two environment notes that cost me a couple of runs and may help others, neither of which is a Qlib defect:

On Windows, a script that builds an Alpha158 dataset must sit behind if __name__ == "__main__":, otherwise the joblib workers re-import the module and re-enter dataset construction.
Recent MLflow releases refuse the filesystem tracking backend by default; setting MLFLOW_ALLOW_FILE_STORE=true restores the current recorder behaviour.

6. Suggested fix

Either of these would resolve the ambiguity; the second is much cheaper:

Implement the equal-weight benchmark already sketched at qlib/backtest/profit_attribution.py:238, and let benchmark in the backtest config accept something like "equal_weight" in addition to an instrument code, so the benchmark weighting can be matched to the strategy's weighting.
Document the convention where excess_return_* is defined (record_temp.py, and the report/benchmark pages in docs/), stating that the default benchmark is cap-weighted while TopkDropoutStrategy is equal-weighted, and that the resulting excess return therefore contains a weighting component. A one-line note next to the examples/benchmarks/README.md results table would reach the most readers.

I am happy to open a PR for either if the direction is agreeable.

7. What I checked and found sound

I looked at these while investigating and want to record that they held up, so this issue is not read as a broader criticism:

Point-in-time index membership is real in the shipped data. instruments/csi300.txt has 820 rows for 690 unique symbols, 109 symbols carry more than one membership span, and the maximum for a single symbol is 5 (SH601991: 2005-01-01..2010-12-31, 2012-01-04..2012-12-31, 2013-07-01..2014-12-12, 2015-06-15..2016-12-09, 2017-12-11..2019-06-14). 520 of 820 rows (63.4%) end before the dataset's last calendar day (2020-09-25). Of the 441 symbols whose CSI300 membership ended early, 434 still have a directory under features/ (3,875 symbol directories in total). The spans are enforced at qlib/data/data.py:624-628, and scripts/data_collector/index.py:204-238 reconstructs them from index change history. Separately, scripts/data_collector/crowd_source/README.md:4 already documents that the Yahoo-sourced data itself can miss delisted names.
No look-ahead in the signal-to-execution path. The signal is read from the previous step at qlib/contrib/strategy/signal_strategy.py:140-143 (get_step_time(trade_step, shift=1)), and the Alpha158/Alpha360 label Ref($close, -2)/Ref($close, -1) - 1 (qlib/contrib/data/handler.py:90,152) spans T+1 close to T+2 close, consistent with execution at the T+1 close. Normalizers fit only on the training window (qlib/data/dataset/processor.py:197-205), with an explicit in-code warning that fit_end_time must not include test information.
Transaction costs are on by default and included in the published table. Defaults are open_cost=0.0015, close_cost=0.0025, min_cost=5.0 (qlib/backtest/exchange.py:48-50), and examples/run_all_model.py:155-160,178-180 populates the README table from 1day.excess_return_with_cost.*.

8. Limitations of this report

Measured on CSI300 over 2017-01-03..2020-07-31 only. I did not test the US region, other index universes, or other windows, and the sign and size of the offset are window-dependent.
The model run is a single seed, and as noted in 3b it does not match the README's 20-seed backtest figures; I did not investigate that difference.
The equal-weight benchmark I built rebalances daily and ignores transaction costs on the benchmark side, which is the same convention the cap-weighted index comparison uses but is worth stating.
I did not test whether an equal-weight benchmark would change the ranking of the models in examples/benchmarks/README.md, only the level of a single model's reported excess return.