Recovery: persist completed tool results at execution-batch boundaries
Problem / Motivation
The current runtime checkpoint has a crash-consistency window during multi-tool responses. If tool A completes, tool B starts, and the gateway crashes before tool C runs, the durable checkpoint may still record all calls as pending. After restart, recovery cannot distinguish completed work (and its external side effects) from unfinished or uncertain work.
This makes recovery unnecessarily lose completed tool results and evidence that a side-effecting tool already ran. The existing recovery policy should remain conservative: a restart must not implicitly resume model/tool work, and pending calls must still require explicit user confirmation.
Proposed Solution
Advance the existing runtime checkpoint at completed tool-execution batch boundaries. A partial checkpoint could record:
assistant tool calls = [A, B, C]
completed_tool_results = [A_result]
pending_tool_calls = [B, C]
Maintain the invariant: completed_ids union pending_ids == assistant_tool_call_ids; completed_ids intersect pending_ids == empty. Reuse the existing runtime_checkpoint, completed_tool_results, pending_tool_calls, tool batching logic, and recovery materialization path. Recovery should preserve A's real result, materialize interrupted results for B and C, discard provider-native conversation state for partial/uncertain checkpoints as it does today, and continue waiting for explicit confirmation without replaying pending tools.
Alternatives Considered
Persisting every internal tool event would add unnecessary checkpoint churn. Introducing a new task ledger, scheduler, or recovery subsystem would be broader than needed. Batch boundaries are a focused durability boundary because read-only/concurrency-safe tools already run in batches, while side-effecting tools run as singleton batches.
Related Component
Other
Additional Context
Suggested regression tests:
- Partial sequential completion: A completes while B remains pending; after simulated interruption, assert A is in completed_tool_results and B is in pending_tool_calls.
- Partial recovery: preserve A's real result and give B an interrupted-tool result.
- No implicit replay: recovery waits for explicit confirmation and does not execute B.
- Partition validation: reject overlapping IDs, incomplete unions, and duplicate IDs.
- Legacy compatibility: keep all-pending awaiting_tools and all-completed tools_completed checkpoints valid.
Non-goals: no automatic replay, new task ledger, new scheduler, exactly-once guarantee, change to ordinary agent behavior, or new public configuration. This follows the general durability principle used by workflow runtimes such as LangGraph and DeerFlow, while keeping the implementation within nanobot's existing checkpoint primitives.
Source: HKUDS/nanobot