#2725·smolagents

ActionStep.dict() returns raw non-JSON-serializable bytes for observations_images

Author: VANDRANKICreated Sep 1, 2026Updated Sep 15, 2026

ActionStep.dict() in src/smolagents/memory.py is meant to produce a JSON-serializable representation of a step: every other field goes through make_json_serializable() or an explicit .isoformat()/.dict() conversion. observations_images is the one exception:

python
"observations_images": [image.tobytes() for image in self.observations_images]
if self.observations_images
else None,

PIL.Image.tobytes() returns a raw Python bytes object, which json.dumps() cannot serialize:

python
from PIL import Image
import json
img = Image.new("RGB", (2, 2))
json.dumps(img.tobytes())
# TypeError: Object of type bytes is not JSON serializable

ActionStep.dict() is what AgentMemory.get_full_steps() and get_succinct_steps() return (used by agent.run(..., return_full_result=True) via RunResult.steps), so any step containing observations_images (set whenever a step includes screenshots, e.g. vision_web_browser.py) breaks the moment the caller does json.dumps(steps) or writes it to a JSON log/replay file, even though the method's whole purpose is to be JSON-safe.

Separately, tobytes() also drops the image's size and mode, so even a caller that base64-encodes the raw bytes themselves has no way to reconstruct the image (there is no width/height/mode stored anywhere in the dict).

Repro

python
from PIL import Image
from smolagents.memory import ActionStep
from smolagents.monitoring import Timing
import json

step = ActionStep(step_number=1, timing=Timing(start_time=0.0, end_time=1.0), observations_images=[Image.new("RGB", (10, 10))])
json.dumps(step.dict())  # TypeError: Object of type bytes is not JSON serializable

Expected: ActionStep.dict() produces a JSON-serializable dict for every field, consistent with the rest of the method (e.g. base64-encode the PNG bytes the way SafeSerializer/_to_json_safe already do elsewhere in the codebase for PIL images).

Actual: raises TypeError: Object of type bytes is not JSON serializable on json.dumps, and the existing test (tests/test_memory.py::test_action_step_dict) only checks "observations_images" in action_step_dict, not that the value round-trips through JSON, so this isn't caught by CI.

Verified against current main (commit 30bb116).