Usage a capability adds to `ctx.usage` inside a durable operation is credited on replay but not on a first run
Description
A capability operation marked @durable_operation reports the usage it produced in one of two ways, and only one of them reaches the run's span.
- Executed in process, it adds to
ctx.usageitself. - Replayed, it returns a
usage_deltafor the caller to fold in, because the activity's context can't reach the spans open in the caller.
#8456 records the replayed delta, so a replayed operation reaches the run's span. A first run's direct mutation does not. The same capability therefore reports one number the first time and another on a replay.
The gap is narrow: ctx.usage is documented read-only by convention, and usage_delta is the supported way for an operation to report usage. It is worth fixing anyway, or worth deciding explicitly that direct mutation is unsupported and the asymmetry is the answer.
Why this is an issue and not a PR
It was one (#8462, closed). The mechanism went through four defects, each found after the previous fix was called done, and two by reviewers:
- Crediting only the replayed delta loses the first run's direct mutation. (The starting point, above.)
- Crediting the whole
ctx.usagedelta counts a nested run twice. An operation that startsAgent.run(usage=ctx.usage)has that run record its own usage the ordinary way; adding the delta on top reported 153 input tokens where the replay reported 102. - Subtracting what the operation's own task recorded leaves a concurrent sibling's usage in the remainder.
@durable_operationis callable from any hook, so an operation can overlap a sibling tool: with one tool invoking an operation and another delegating withusage=ctx.usage, the run reported 2020 against a true 2010. - Holding the watchers by value let two of them unregister each other.
RunUsagecompares by value, so two runs that hadn't recorded anything yet were indistinguishable tolist.remove, which dropped whichever it found first.
The shape that survived all four: mirror what is recorded into that RunUsage object (by identity, so a nested or concurrent run's records are included whoever makes them), and credit the difference from the object's own delta — the part nothing recorded.
That branch is now stale in a way that matters: it was cut before #8456 switched the span attribute from a run's subtree to a run's own requests, so it still carries the old fan-out and would reinstate it if merged. Whatever lands here should be built fresh on top of #8456, and all four cases re-verified against the current semantics, since they were worked out against the old ones.
Worth deciding first
- Should a direct
ctx.usagemutation count as the invoking run's own usage at all? Under the semantics #8456 settles on, a nested run's requests belong to the nested run — so the answer isn't automatic. - Is the mechanism's cost justified? It needs process-global mutable state and a hook on every usage record, to serve a pattern the docs discourage. "Direct mutation is unsupported, document the asymmetry" is a legitimate resolution that closes this issue with no code.
Related
- #8456 — records the replayed
usage_delta; this is the remaining half - #8462 — the closed PR, for the working history
Filed by Claude Code on @DouweM's behalf.
Source: pydantic/pydantic-ai