MCP tool-search / deferred-loading design

Submission checklist

• This is a feature request, not a bug report.
• I searched existing issues and didn't find this feature.
• I checked the docs and README for existing functionality.
• This request applies to this repo (deepagents) and not an external package.

Area (Required)

• deepagents (SDK)
• dcode
• talon
• acp
• evals
• harbor
• daytona
• modal
• quickjs
• runloop
• vercel
• langsmith-sandbox
• Other / not sure / general

Feature description

dcode connects to every trusted/approved MCP server at session startup and eagerly loads the full JSON tool-call schema for every advertised tool into the agent's toolset. With a moderately sized real-world MCP setup (9 servers, 178 tools total), this consumes the majority of the context window before the user sends a single message.

Measured evidence (LangSmith traces, 9 MCP servers — mem0-mcp, langchain-docs, langchain-reference, cursor, slack, github, prompt_manager, postgres, jira_prod, google_workspace — single trivial "hello" message, i.e. pure startup cost):

Model	Prompt tokens	Context window	% consumed at "hello"
claude-sonnet-5	92,690	150,000	61.8%
gemini-3.6-flash	43,945	150,000	29.3%

Same 178-tool payload (~193-198KB raw JSON) in both traces — Claude's tokenizer consumed ~2.1x more tokens for identical content, meaning severity compounds with model choice, not just MCP config.

Concentration: 2 of 9 servers (google_workspace: 110 tools, jira_prod: 40 tools) account for 84% of all bound tools. A fix does not need to solve the general case — even a narrow threshold-based fix recovers most of the loss.

This is a known, previously-discussed gap (#616, #2127, #2121, #4836), but this issue proposes a specifically different design from the most concrete prior proposal, #4836 ("Dynamic Tool Allocation"), which is still open. #4836's 3-stage pipeline (LLM router → BM25/dense indexer → LLM reranker) drew maintainer pushback specifically over adding two extra LLM calls to every turn without benchmarks justifying the latency (https://github.com/langchain-ai/deepagents/issues/4836#issuecomment-5178876969). That objection is valid and this issue does not ask to revisit that design — it proposes a zero-extra-model-call alternative instead.

Proposed solution (optional)

Adopt a deferred-loading + search-meta-tool design, modeled directly on Anthropic's shipped Tool Search Tool (https://www.anthropic.com/engineering/advanced-tool-use), supplemented with a one-time offline description-enrichment + embedding pass at MCP-server-connect time (informed by the Tool-DE and Re-Invoke retrieval literature — see Additional context). Every component here is validated in the literature as not requiring a reasoning-LLM call per turn, which is the direct answer to the #4836 objection.

Why not #4836's original design: the retriever/indexer half of #4836 is sound (it's essentially the same technique as Gorilla's and ToolLLM's tool retrievers, both peer-reviewed and validated at far larger scale than our 178-tool case). The part that drew objection — the second LLM-reranker call in the per-turn hot path — is exactly what this proposal drops. RestGPT (arXiv:2306.06624) is the literature's own cautionary example of a multi-LLM-call tool pipeline, describing it as inefficient in practice; Anthropic's production system gets an 85% token reduction and an accuracy improvement (Opus 4: 49%→74%; Opus 4.5: 79.5%→88.1% on large tool libraries) using pure search (BM25/regex/embedding) with zero separate reranker model.

Runtime flow:

1. At MCP-server-connect time (once per session, cached until a server's tool list/schema hash changes): extract name/description/params for every tool; run a one-time LLM-assisted enrichment pass adding a short when_to_use disambiguation hint per tool (Tool-DE-style — matters most for near-duplicate CRUD tools within one server, e.g. Google Workspace's 110); compute and cache embeddings for the enriched descriptions. This is a batch job, never in the per-turn latency path.
2. At session startup: bind only (a) a small always-on core toolset (dcode's built-ins + user-pinned high-frequency tools) and (b) one search_tools(query) meta-tool + a generic call_tool(server, name, args) dispatcher. Do not bind the other ~170+ tools' full schemas into context.
3. On a turn requiring a capability: the agent calls search_tools(query) — embedding similarity + optional BM25/regex fallback against the pre-computed cache, no LLM call, a vector lookup only — returning the top-K candidates with full schemas expanded into context for that turn.
4. The agent calls the real tool via call_tool (or has the matched schema expanded directly into a native tool_use block, mirroring Anthropic's approach).
5. Concentration-aware default: servers above a configurable tool-count threshold (suggested default: 20, matching Anthropic's own documented threshold) are deferred/searchable by default; small servers stay always-loaded since eagerly loading a handful of tools costs negligible tokens and gains nothing from search latency.

Config surface:

• Per-MCP-server flag (e.g. in .mcp.json): toolLoading: "eager" | "deferred" | "auto", where "auto" defers any server whose tool count exceeds a configurable deferThreshold (default 20).
• Global deferThreshold setting, plus per-tool pinning (server+name) to force specific high-frequency tools to stay eager regardless of server-level policy.
• Cache location/TTL for the offline embedding/enrichment index (e.g. ~/.deepagents/tool-index-cache/), auto-invalidated on tool-list/schema changes.
• search_tools topK setting, default 5-10 (consistent with the retrieval literature's evaluated shortlist depths).

Expected impact (extrapolated from Anthropic's own measured 85% reduction on a smaller 58-tool/5-server case): bringing this setup's Claude startup cost from 61.8% of a 150k window to roughly single digits, with the largest gains from deferring the two dominant servers. This is an extrapolation, not a re-measured result — validating token count and tool-selection accuracy before/after on dcode's actual traffic should be the acceptance criterion for any implementation PR.

Additional context (optional)

Related issues*: #4836 (prior proposal, different design — see above), #4658 (tool-selection middleware), #616, #2127, #2121 (earlier discussion of the same underlying MCP context-bloat problem), #4202 (semantic memory search — same retrieval-architecture family, different subsystem).

Vendor precedent: Anthropic's Tool Search Tool (https://platform.claude.com/docs/en/agents-and-tools/tool-use/manage-tool-context) — shipped, documented with production numbers, recommends tool search specifically past ~20 tools (this setup is at 178). Anthropic's own tool-use guidance states function-selection accuracy "degrades once you exceed 30-50 available tools"; OpenAI's function-calling guidance recommends under 20 functions per turn.

Retrieval literature supporting the no-extra-LLM-call design:

• Gorilla (arXiv:2305.15334) — BM25/embedding retriever, no LLM reranker, ~1,600 APIs.
• ToolLLM/ToolBench (arXiv:2307.16789) — embedding-based API retriever over 16,464 APIs; retrieved set outperformed ground-truth set on pass/win rate.
• ToolRet (arXiv:2503.01763) — benchmark showing naive embedding search underperforms on tool descriptions specifically (lower lexical overlap than typical IR) — motivates the enrichment step below.
• Tool-DE — one-time offline LLM enrichment of tool docs before embedding (not per-turn); BM25 nDCG@10 36.4→39.3, dense +2-4 pts.
• Re-Invoke (arXiv:2408.01875, EMNLP Findings 2024) — synthetic query expansion at indexing time, unsupervised, scales to large/dynamic toolsets without fine-tuning (fits a config-driven CLI like dcode where toolsets differ per user).
• Tool-to-Agent Retrieval (arXiv:2511.01854) — evaluated on LiveMCPBench (real MCP tools, not synthetic); +19.4% Recall@5 from embedding tool+server jointly, directly relevant to disambiguating near-duplicate tools within one server.
• RestGPT (arXiv:2306.06624) — the field's own cautionary tale against multi-LLM-call tool pipelines ("every plan-execution loop requires at least four interactions with LLMs, making execution efficiency low").

Scale note: this repo's own #4836 discussion cited lack of benchmarks as the blocker. The above is real trace data from a 9-server/178-tool dcode configuration, not a synthetic estimate.

If this direction looks right, I'd be glad to take this on if assigned.