[Feature Proposal]: Ingestion Gatekeeper Middleware for Token Bloat Reduction

[Feature Proposal]: Ingestion Gatekeeper Middleware for Token Bloat Reduction in Deep Research Loops

Problem Statement

When open_deep_research sub-agents conduct iterative web searches (e.g. via Tavily or scrapers), raw web page text (3,000–5,000 words per page) is appended directly to researcher_messages. By iteration 3, researchers re-read 30,000+ input tokens per turn, up to 85% of which is identical background fluff ("What is an EV battery", "Sony in 1991").

This results in:

• High API token costs ($0.50 to $2.00 per research run).
• Latency and rate-limit bottlenecks.
• Attention focus degradation ("lost-in-the-middle").

---

Proposed Solution: ConceptDiffEngine Ingestion Middleware

We have engineered and benchmarked an ingestion gatekeeper layer (open-research-lite) that sits between web search output and the researcher agent state:

1. Session Knowledge Graph: Maintains an in-memory graph state of all entities, dates, and metrics learned during a research session.
2. Dual-Layer Extraction: Uses Gemini Flash structured output (or a 0-ms local NLP parser) to extract Subject-Predicate-Object triplets (Subject ──► Predicate ──► Object).
3. Concept-Diff Classification:
   • DISCARD: Repetitive background fluff deleted immediately ($0 tokens spent).
   • DIFF_ADD: Novel assertions added to graph & passed in compact Diff Payload (~150 words).
   • DIFF_CONFLICT: Surfaced price or numerical variances explicitly highlighted ($110 vs $140/kWh).

---

Benchmark & Empirical Evidence

Tested across live web searches on Deep Research Bench queries (2026 solid state battery Wh/kg breakthroughs):

Metric	Upstream open_deep_research	With ConceptDiffEngine	Performance Gain
Live Web Words Fetched	9,159 words	1,083 words	88.2% Token Reduction ⚡
Input Tokens Passed	11,906 tokens	1,407 tokens	88.2% Token Savings
Ingestion API Cost	$0.0298	$0.0035	8.5x Cost Reduction
Final Report Precision	100% technical depth	100% technical depth	Zero Loss of Information

---

‍♂️ Request to Maintainers

We have built a fully tested, backwards-compatible implementation configured via an optional flag (enable_concept_diff: bool = True in configuration.py).

Would the maintainers be open to a Pull Request bringing this feature to open_deep_research? We have the PR branch ready to submit.