feat(typesafe): harden TypeSafeClassifier against API limits and rate limiting
Submission checklist
- This is a feature request, not a bug report or usage question.
- I added a clear and descriptive title that summarizes the feature request.
- I used the GitHub search to find a similar feature request and didn't find it.
- I checked the LangChain documentation and API reference to see if this feature already exists.
- This is not related to the langchain-community package.
Package (Required)
- langchain
- langchain-openai
- langchain-anthropic
- langchain-classic
- langchain-core
- langchain-model-profiles
- langchain-tests
- langchain-text-splitters
- langchain-chroma
- langchain-deepseek
- langchain-exa
- langchain-fireworks
- langchain-groq
- langchain-huggingface
- langchain-mistralai
- langchain-nomic
- langchain-ollama
- langchain-openrouter
- langchain-perplexity
- langchain-qdrant
- langchain-typesafe
- langchain-xai
- Other / not sure / general
Feature Description
A few small improvements to make TypeSafeClassifier behave better in production, based on what the TypeSafe API and Jev 1.13 docs describe today:
- Validate the documented question limits before sending. The API accepts at most 255 options for a Choice and 2 to 10 levels for a Score. Right now we only check the minimums, so an oversized question goes over the network and comes back as a 422.
- Retry transient failures. TypeSafe asks clients to back off and retry on 429 and 529 Overloaded. The classifier currently makes a single attempt, even though we already parse retry-after into TypeSafeRateLimitError.retry_after_ms.
- Expose the models endpoint. GET /v1/models lists the available models and their release dates. A small list_models() / alist_models() helper would make it easy to discover versions and pin one (for example jev-1.13.0 instead of jev-latest). The docstring could also mention the jev-preview alias.
- Make traces more accurate.
- ls_model_name is set to the requested alias (jev-latest), while the response tells us which version actually answered. That's the useful value when comparing runs across model updates.
- ls_model_type is set to "chat", which doesn't really fit a classifier.
- request_id is on the response but never reaches the trace, which is exactly where you'd look for it when talking to TypeSafe support.
Use Case
TypeSafe's own docs push a fan-out pattern (many small classifications in parallel), and in an agent the classifier can run several times per step once the router and auto-mode middleware are enabled. With a limit of 1,200 requests/minute, 429s are easy to hit, and today every one of them bubbles up as an error that each caller has to handle on its own. Most other partner integrations retry these out of the box, so it's a bit surprising here.
The limit checks are mostly about failing fast with a clear message: when criteria are built dynamically (from a list of tools or a taxonomy), going over 255 options is easy, and a local validation error is much easier to debug than a 422 from the API.
On the tracing side, when jev-latest moves to a new release, being able to filter LangSmith runs by the actual model version makes it possible to spot behavior changes.
Proposed Solution
All changes are additive, with keyword-only parameters and defaults that keep today's behavior where it matters:
- Choice.criteria: add max_length=255. Score.criteria: add max_length=10.
- TypeSafeClassifier: add max_retries: int = 2. Retry on 408, 429 and 5xx (including 529) plus connection/timeout errors, with exponential backoff and jitter, and honor retry-after / retry-after-ms when present. This mirrors the defaults of the official SDK's RetryPolicy. max_retries=0 restores the current behavior.
- TypeSafeClassifier.list_models() / alist_models(): thin wrappers over GET /v1/models that reuse the configured clients and error handling.
- Tracing: record response.model and request_id in the run metadata, and revisit the ls_model_type value (happy to follow whatever maintainers prefer here).
I'm happy to split this into smaller PRs if that's easier to review (limits alone would be a very small one).
Alternatives Considered
For retries (and HTTP/2, which the SDK now supports as an extra), #40550 takes a different route: build the integration on top of the official typesafe-sdk, which would bring RetryPolicy along. That PR hasn't moved in a while and conflicts with master now, and part of it already landed through #40659, so I'm not sure it's still the plan.
If building on the SDK is still the intended direction, I'd drop the retry part from this proposal and just do the rest. If not, a built-in max_retries keeps the package free of an extra dependency. Could a maintainer weigh in on which way you'd like to go?
Additional Context
- Related: #40706 / #40707 (rejecting empty questions locally), which is the same "validate before sending" idea.
- TypeSafe API reference (limits, error codes): https://docs.typesafe.ai/api.md
- Models and aliases (jev-latest, jev-preview, GET /v1/models): https://docs.typesafe.ai/models.md
- SDK retry defaults: https://docs.typesafe.ai/sdk/python/api/retries.md
Social handles (optional)
Source: langchain-ai/langchain