Run an agent on a model that only answers questions: what `TypeSafeModel` (Jev) still needs from the framework

TypeSafeModel runs an agent whose output_type is made of typed questions on TypeSafe's Jev, which answers questions with a confidence and does nothing else: no text, no tools, no files, no streaming. Everything else an agent expects from a model is either refused with a UserError today or done by the framework on Jev's behalf. This tracks the second list: the direction from @adtyavrdhn's handoff on #8450, plus what an audit against TypeSafe's API docs and a look at the ecosystem since launch have added. None of it blocks #8450.

Shipped or in review

• #8450 — the model: bool/bounded float → yes-no, Literal/Enum → pick-one, IntEnum with member docstrings → rubric, a list of options → one yes-no per option, X | None → an explicit "None of these." option, a nested model → its fields; confidence, probabilities and scores in provider_details; message history as the material judged; enum member docstrings in every tool schema; a streamed run gets the whole answer as one event.
• #8450 also has the off-ramp (items 2 and 3 below as first designed): with tools attached, Jev is asked one Choice question, "which of these does this call for?", over the output type and the tools. A tool with no arguments (or an output function that takes nothing) Jev calls itself, once per run; a tool with arguments is raised as ToolCallProposed, a ModelAPIError, so FallbackModel(jev, llm) hands that whole step to the LLM with no configuration. typesafe_tool_call_threshold (default 0.8) is how sure Jev has to be before a tool is taken over the output type. Falling back below a confidence is documented with the existing fallback_on handler. Measurements are in the comments below.
• #8453 — FileUnderstanding: a model that reads files describes them, Jev judges the description. The first "the framework does what Jev can't" capability.
• #8454 — Classifier evaluator: one typed question per case.
• #8464 — an adapter can't tell replayed history from the current run's messages, which is why system_prompt= is judged rather than asked.

Designed, not built

1. A hand-off rate signal. FallbackModel and ToolCallProposed make delegation work, but nothing tells you 94% of calls fell back. The metric for everything here is the pair (accuracy, hand-off rate); accuracy alone can't see a chain that always delegates. provider_details['tool'] carries every pick and its probabilities, so the rate is computable today from the responses; what is missing is a place it is reported.

2. The default threshold: decided, 0.6. 0.8 was first chosen for precision (93% of tools taken were the rule's tool). On the four-way judgment (ticket / escalate / refund(amount) / draft a reply) 0.6 agrees with the rule on 62/120 against 43 at 0.8, and with GPT-5.6 Sol on 86/120, which is where Opus 5 and Sol agree with each other (87). #8450 ships 0.6; the cost is more hand-offs, so a slower median when an LLM sits behind. Worth re-checking on a second task.

3. A union of two output types with fields as a choice first. Ticket | Escalation is refused; arg-less members are already hand-offs. Ask a Choice over the member names first, then fill the chosen member: two Jev calls at ~180 ms still beat one LLM call, and it is what TypeSafe's docs say to do when one judgement depends on another. A member Jev cannot express is then a declared off-ramp rather than an accident. The cost guard, stated as a rule: a type the cheap model cannot express is a coding error and stays a UserError; it must never silently route every call to the expensive model.

4. ModelProfile.supports_text_output. False for Jev. LLMJudge and GEval switch to a typed output shape when it's off, so the existing evaluators run on Jev without Classifier having to exist, and text output is refused for any such model in one place instead of inside TypeSafeModel.

5. The rest of the handoff list: a str field written by a fallback model; a reason on demand below a confidence threshold; Jev behind the harness Advisor; Classifier(output_type=float) into ROC-AUC; the eager profile-flag check that makes BinaryImage output raise even when ImageGeneration could produce it. Native tools stay refused.

6. A str field answered by picking, not writing. Jev cannot write text, but a string field whose answer is already in the material does not need writing: pull candidates out of the state deterministically (the schema's own pattern, or format: email / format: tel), offer them as a Choice with an explicit "none of these fits" option, and let Jev pick. Measured on 12 cases where every candidate had the same shape and only the description distinguished them (which of three addresses is the customer's, which of two case numbers is still open, which of three amounts is the overcharge): 12/12, confidence 0.91-1.0, 12 calls in 3.9 s. This is the half of item 5 that needs no fallback model at all, and it is bounded: it covers extraction, never summarising or writing a reply. Third-party kedi-typesafe ships exactly this for its own model class, with a 255-option cap and an error when no candidate is found rather than an invented value. Contrast with the rejected version of the same insight: spelling text character by character as a Choice over a 64-symbol alphabet is dead - Jev returned rfufetan for refund and produced 0/8 exact on both plain copying and base64, over 117 and 160 calls. Selection among whole, meaningful options is the strength; sequential composition is not.

Framework findings on the way, not Jev's: on the OpenAI Responses API, gpt-5.6-* with tool-mode structured output and function tools stops honouring tool_choice: required once a tool result is in the history and answers in prose (verified on the wire); NativeOutput avoids it, and the profile may want to default those models to it. A harness InputGuard block on a structured-output agent fails on TestModel too.

Jev × Monty

Code mode lives in pydantic-ai-harness, and its MontyExecutor is already a generic dispatcher (tool calls for code mode, sub-agent runs for dynamic workflows). In order of value:

• judge() inside the sandbox. A FunctionToolset tool wrapping a Jev agent, exposed into code mode; the LLM writes gather(judge(r.body, 'Is this a billing dispute?') for r in rows) and returns 5 rows of 500. Code mode's point is that intermediate results never enter the context window, and Jev is the first thing cheap enough to call in a loop inside it. Works today with zero new surface. To measure: how questions the LLM writes at runtime compare with hand-written ones.
• Jev at the host-function boundary, in before_tool_execute: one ~150 ms judgement per tool call a sandboxed program makes. The ecosystem has this without a sandbox already (Bicameral, pi-jev, jev-axi, and @adtyavrdhn's own pydantic-jev-examples); Monty is the half none of them have. pi-jev's shadow mode, logging what would have been blocked, is how to introduce any Jev guard honestly.
• Monty computes what Jev can't. Arithmetic, counting and dates are on TypeSafe's jaggedness page. A recipe where the program does those in Python and asks Jev only the semantic residue turns "what Jev answers badly" into a pattern.
• Rejected: Jev choosing among N program outputs. Numbers are its worst axis, and an == vote is free and better.

Prior art to link rather than race

jevantic (InputGuardrail/OutputGuardrail as capabilities, select() returning the original object), pydantic-jev-examples, awesome-typesafe, kedi-typesafe (its own TypeSafeModel plus a LangChain one; tools, streaming, nested models and optionals all refused, but the candidate-extraction idea in item 6 is worth taking).