Temporal Knowledge Leakage in LLM Backtesting (Model-Level Future Information Contamination)

First of all, excellent work on TradingAgents.
The architecture, agent coordination design, and historical data pipeline are very impressive.

However, I would like to raise a concern regarding a potential evaluation issue in LLM-based backtesting:

Temporal Knowledge Leakage from the Base LLM

From the documentation and implementation, TradingAgents already appears to handle traditional future leakage correctly:

• date-aware data fetching
• historical news restriction
• timestamp-based retrieval
• historical OHLCV snapshots

This prevents explicit future data exposure.

However, there is another layer of leakage that may still exist:

Model-Level Future Knowledge Contamination

For example:

• using a 2026-trained LLM
• while backtesting decisions in 2023

Even if the prompt context is historically constrained, the model weights themselves may already encode:

• macro outcomes
• company trajectories
• major crises
• sector bubbles
• future market narratives

Examples:

• NVDA AI boom
• FTX collapse
• SVB failure
• post-rate-hike market behavior
• inflation outcomes

This means the model may unintentionally perform:

• hindsight reasoning
• retroactive narrative fitting
• future-aware macro interpretation

rather than genuine ex-ante reasoning.

---

Question

Has the team considered this issue formally?

Specifically:

1. Do you view TradingAgents backtests as:

   • historically grounded simulations
   • or strictly causal backtests?

2. Have you explored:

   • time-frozen checkpoints
   • older foundation models
   • cutoff-constrained continual pretraining
   • temporal contamination evaluation

3. Are there plans to introduce:

   • model cutoff declarations
   • leakage benchmarks
   • temporal purity evaluation protocols

---

Possible Research Direction

A potentially rigorous protocol could be:

Backtest Period	Model Knowledge Cutoff
2023	<= 2022
2024	<= 2023

with:

• retrieval cutoff
• embedding cutoff
• reranker cutoff
• memory cutoff

all aligned temporally.

Beside above questions, what solutions has your team come up with for this problem?

---

I believe this is becoming an important issue for the entire LLM-for-finance research community, especially as reasoning models become stronger.

Would love to hear the team's perspective on this.