`prepare_response_model` creates a new Pydantic class on every request, leaking one model class + validator per call

Author: philnoCreated Sep 7, 2026Updated Sep 9, 2026

Summary

prepare_response_model wraps the user's response model with create_model(...) and is called on every create() request. Each call therefore builds a brand-new Pydantic class, core schema and validator, and these are retained (in part by instructor's own lru_cache). For a single short-lived call this is invisible; for a long-running process making tens of thousands of structured-extraction calls it grows without bound.

Because the wrapping is idempotent already, the result can simply be memoized per response model.

Versions

instructor 1.16.0 (also reproduced on 1.14.5)
pydantic 2.12.x, Python 3.13, macOS

Reproducer

No API key or network needed — prepare_response_model is pure, and it is exactly what the patched client invokes per request (v2/core/patch.py:223 and :346).

python

import gc
import tracemalloc
from functools import cache

from pydantic import BaseModel, Field

import instructor
from instructor.utils.core import prepare_response_model

try:  # instructor >= 1.16
    from instructor.v2.core.function_calls import ResponseSchema as _Base
    from instructor.v2.providers.openai.schema import generate_openai_schema
except ImportError:  # instructor <= 1.15
    from instructor.processing.function_calls import OpenAISchema as _Base
    from instructor.processing.schema import generate_openai_schema


class UserDetail(BaseModel):
    """A user."""

    name: str
    age: int


class Wide(BaseModel):
    """A wider model, to show the cost scales with schema size."""

    f0: str = Field(..., description="Field zero, with a description.")
    f1: str = Field(..., description="Field one, with a description.")
    f2: int = Field(..., description="Field two, with a description.")
    f3: float = Field(..., description="Field three, with a description.")
    f4: bool = Field(..., description="Field four, with a description.")
    f5: str | None = Field(None, description="Field five, with a description.")
    f6: list[str] = Field(default_factory=list, description="Field six.")
    f7: dict[str, str] = Field(default_factory=dict, description="Field seven.")


N = 2000


def count_wrapper_classes() -> int:
    gc.collect()
    return sum(
        1 for o in gc.get_objects() if isinstance(o, type) and issubclass(o, _Base)
    )


def measure(label, fn, model):
    gc.collect()
    before = count_wrapper_classes()
    tracemalloc.start()
    snap_before = tracemalloc.take_snapshot()

    # What one request does: prepare the response model, then build its schema.
    held = []
    for _ in range(N):
        prepared = fn(model)
        generate_openai_schema(prepared)
        held.append(prepared)

    snap_after = tracemalloc.take_snapshot()
    grew = sum(s.size_diff for s in snap_after.compare_to(snap_before, "lineno"))
    tracemalloc.stop()

    print(
        f"  {label:12} {N} calls -> {count_wrapper_classes() - before:5d} classes "
        f"retained, {grew / 2**20:7.1f} MiB ({grew / N / 1024:5.1f} KiB/call), "
        f"distinct: {len({id(h) for h in held})}"
    )
    del held
    gc.collect()


memoized = cache(prepare_response_model)  # the one-line fix

print(f"instructor {instructor.__version__}\n")
for model in (UserDetail, Wide):
    print(f"{model.__name__} ({len(model.model_fields)} fields):")
    generate_openai_schema.cache_clear()
    measure("current", prepare_response_model, model)
    current_info = generate_openai_schema.cache_info()

    generate_openai_schema.cache_clear()
    measure("with @cache", memoized, model)
    fixed_info = generate_openai_schema.cache_info()

    print(f"    generate_openai_schema, current:     {current_info}")
    print(f"    generate_openai_schema, with @cache: {fixed_info}\n")

Output

instructor 1.16.0

UserDetail (2 fields):
  current      2000 calls ->  2000 classes retained,    18.1 MiB (  9.2 KiB/call), distinct: 2000
  with @cache  2000 calls ->     1 classes retained,     0.0 MiB (  0.0 KiB/call), distinct: 1
    generate_openai_schema, current:     CacheInfo(hits=0, misses=2000, maxsize=256, currsize=256)
    generate_openai_schema, with @cache: CacheInfo(hits=1999, misses=1, maxsize=256, currsize=1)

Wide (8 fields):
  current      2000 calls ->  2000 classes retained,    39.9 MiB ( 20.4 KiB/call), distinct: 2000
  with @cache  2000 calls ->     1 classes retained,     0.2 MiB (  0.1 KiB/call), distinct: 1
    generate_openai_schema, current:     CacheInfo(hits=0, misses=2000, maxsize=256, currsize=256)
    generate_openai_schema, with @cache: CacheInfo(hits=1999, misses=1, maxsize=256, currsize=1)

Cost per call scales with schema size (9.2 KiB/call for 2 fields, 20.4 KiB/call for 8). On a real model with ~50 nested fields and field descriptions I measured ~83 KiB per call.

Root cause

response_schema() builds a new class unconditionally, with no caching — v2/core/function_calls.py:592:

python

def response_schema(cls: type[Model]) -> type[Model]:
    ...
    schema = cast(
        type[BaseModel],
        wraps(cls, updated=())(
            cast(Any, create_model(cls.__name__, __base__=(cls, ResponseSchema)))
        ),
    )
    return cast(type[Model], schema)

It is reached from prepare_response_model (v2/core/response_model.py:108-109), which the patched client calls per request (v2/core/patch.py:223, :346).

Pre-1.16 the same code lived at processing/function_calls.py:793 (openai_schema) and utils/core.py:578 — the rename to v2 preserved the behaviour.

Aggravating factor: generate_openai_schema is @lru_cache(maxsize=256) (v2/providers/openai/schema.py:12), but it is keyed on the class that response_schema() just created. So the cache has a 100% miss rate (hits=0, misses=2000 above) and pins the last 256 dead classes — each with its schema and validator — alive. The cache actively makes the leak worse instead of preventing it.

Impact

A long-running process that issues many structured-extraction calls in a single interpreter grows monotonically until it is OOM-killed. The growth is proportional to number of calls, not to concurrency or payload size, so it doesn't reproduce in short scripts or tests and only appears in production batch workloads.

Suggested fix

Memoize the wrapping, keyed on the input model. prepare_response_model already short-circuits when handed a prepared model:

python

if inspect.isclass(working_model) and not issubclass(working_model, ResponseSchema):
    working_model = response_schema(working_model)

so returning a cached instance is behaviourally identical. Either response_schema or prepare_response_model is a suitable place. Two notes:

A plain lru_cache keyed on the class holds a strong reference to user model classes; a WeakKeyDictionary would avoid pinning dynamically-generated models.
Keys need to cover list[Model] / Iterable[Model] too. Those typing objects hash equal across expressions, so functools.cache handles them correctly as-is.

Workaround

For anyone hitting this before a fix lands, memoizing at the call site works and requires no changes to instructor:

python

from functools import cache
from instructor.utils.core import prepare_response_model

@cache
def prepared(response_model):
    return prepare_response_model(response_model)

# then pass prepared(MyModel) as response_model=...

This is safe because instructor re-prepares on every request and that re-preparation is a no-op for an already-prepared model. Verified that list[Model] still yields an IterableBase subclass and that iterable/streaming dispatch (which uses issubclass(..., IterableBase), not class identity) is unaffected.

Happy to open a PR if you'd like a particular one of the two locations.

Source: 567-labs/instructor

View original on GitHub View discussion on GitHub