[Enhancement] Speculative tools execution

Motivation

Some tools need no user-supplied parameters and may be run faster and cheaper —typical examples are FAQ/Q&A look-ups. For these we want speculative execution:

• Execute immediately. With the full loadedContext (interaction history, customer, agent, etc.) the engine runs the tool without inference of whether it should evaluate the outcome. After execution, inspects the returned ToolResult and decides whether to keep it or discard it (#416 )
• Example: a Q&A tool produces a very detailed answer. but we have a guideline that want to ask a follow-up question, but the extra info from the Q&A tool might make the guideline not work as good as without this info. Speculative execution is orthogonal to the lifespan flag (#416) - a parameter-less tool can still mark its result as response or session, and the engine applies both decisions - first “accept or reject”, then “store for how long” (two bits).

Solution Proposal

If a tool has no parameters and no overlap - it will run without being evaluated.

• A new tool batch class will be created for this kind of tool calls: InstantToolBatch
• InstantToolBatch will instantly create a tool call with no arguments, empty insights and empty GenerationInfo.
• Because the tool candidate does not get a LLM evaluation it must be compared programmatically to the already-staged tool calls to make sure it doesn't run twice with the exact same arguments. For this reason a new filter function will be added before execute_tool_calls:

async def remove_duplicate_executions(
    tool_calls: list[ToolCall],
    staged_events: Sequence[EngineEvent],
) -> list[ToolCall]

• This function returns the input tool calls after filtering out all identical tool calls (with identical arguments). This function will also be used as a fail-safe for LLM-evaluated tool calls (for the cases the LLM failed to identify such duplicate calls)

Discussion

Changing motivation to only latency-reduction

New Solution proposal

As explained in #319, the basic speculative execution is performed concurrently with the Guideline Matching. But there are some changes to it.

---
title: Speculative Tool Execution
config:
  mirrorActors: false
  theme: base
---
sequenceDiagram
participant  Engine
participant  GM  as  GuidelineMatcher
participant  TC  as  ToolCaller
participant  TB  as  SpeculativeRunner

Engine  -)  TB  : Populate speculative-enabled tools
activate  TB
Engine  -->>  TB  : Begin inference of speculative-enabled tools
TB  ->>  TC  : Speculatively infer tool calls
activate TC
Engine  -->>  GM  : Propose relevant guidelines
activate  GM


TC-->>TB : 
deactivate TC
TB->>TC  : Execute speculative tools
activate TC
GM  -->>  Engine: Guidelines
deactivate  GM
TC-->>TB : ToolResult

deactivate TC

Engine  ->>  TC  : Infer and execute tool calls (from activated guidelines)
TC  ->>  TB  : Get results
TB  -->>  TC  : ToolsResults
deactivate  TB
activate  TC
TC  ->>  TC  : Infer remaining tool calls (if any)
TC  ->>  TC  : Execute tools
TC  -->>  Engine: Tool results
deactivate  TC

This is only for starter, there is also an extended version for it, containing a cache for tool results (and tool calls) which can also be used for further optimization.