Allow built-in tools to be deferred/lazy-loaded to reduce token burn
Description
I'd like a config option to make selected built-in tools discoverable/lazy instead of always including their full schemas in the top-level tool list.
Something like:
tools:
forceDiscoverable:
- hub
- task
- eval
- todo
- web_searchThe tools would still be available through the existing xd:// discovery mechanism. The point is just to avoid sending their schemas on every model call.
Right now the alternatives are basically:
- keep the tools enabled and pay their prompt cost every turn
- remove them entirely with
--tools
I'd like a middle ground.
There is some good evidence this matters:
A recent community experiment modified OMP to do essentially this and benchmarked several reduced tool/prompt configurations. The
p7ksetup keptread,bash,edit,glob,grep, andwritetop-level, while movinghub,eval,task,todo, andweb_searchbehind lazy discovery. It substantially reduced prompt size without hurting the tested workloads: https://www.reddit.com/r/PiCodingAgent/comments/1w0rm4e/cut_omps_oh_my_pi_system_prompt_from_226k_to_59k/Frontier Harness Eval also shows a pretty large efficiency gap between Pi and OMP on the same model/tasks. OMP used about twice the normalized cost while getting similar task performance: https://frontierharness.org/
I'm not claiming the tool schemas explain the entire Pi/OMP gap. OMP has a much more complicated harness and there are plenty of other possible contributors.
But this seems like a relatively easy thing to make configurable, and unlike --tools, it doesn't require giving up OMP features.
This is especially useful for tools like hub and task: I want them available when I actually need orchestration/subagents, but I don't need their schemas in front of the model during every normal read/edit/test loop.
Possible config:
tools:
forceDiscoverable:
- hub
- task
- eval
- todo
- web_search
xdevDocs: catalogDefault behavior could stay exactly as it is today.
This also seems related to the idea in #4934, except applied to built-in tools rather than MCP connections. OMP already has the lazy/discoverable-tool machinery; what's missing is a supported way to choose which built-ins use it.
It would also make it much easier to benchmark this properly on frontier models:
A: stock OMP
B: stock OMP + low-frequency tools deferredSame model, same tasks, same prompt, same everything else.
That would tell us how much of OMP's extra token burn is simply the always-visible tool surface.
Use Case
OMP burns way more tokens than Pi and in many cases this is not for any good reason, and the gap is bridgeable.
Area
Tools
Proposed Solution
No response
Alternatives Considered
No response
Source: can1357/oh-my-pi