[BUG] AgentImage inverts pixel values when built from a tensor (255 - array * 255)
Describe the bug
AgentImage inverts the pixel values of any image built from a tensor. Black comes out white and white comes out black.
The conversion appears twice in src/smolagents/agent_types.py, in to_raw() at line 134 and in to_string() at line 156, both using the same expression:
array = self._tensor.cpu().detach().numpy()
img = PIL.Image.fromarray((255 - array * 255).astype(np.uint8))For a normalized image tensor with values in [0, 1], the scaling to byte range should be array * 255. The leading 255 - flips the intensity.
Steps to reproduce
import numpy as np
array = np.linspace(0.0, 1.0, 5, dtype=np.float32)
current = (255 - array * 255).astype(np.uint8)
expected = (array * 255).astype(np.uint8)
print(current) # [255 191 127 63 0]
print(expected) # [ 0 63 127 191 255]Expected behavior
0.0 is black and stays 0. 1.0 is white and stays 255.
Actual behavior
0.0 becomes 255 (white) and 1.0 becomes 0 (black). Every tensor-backed AgentImage is returned as a photographic negative, both when read through to_raw() and when written to disk through to_string().
Why this is likely unintentional
- The two call sites are identical, so a fix in one place without the other would make
to_raw()and the saved PNG disagree. - The
PIL.Image.Imageandbytesinput paths do no inversion, so the same picture changes polarity depending only on which type it was constructed from. - The surrounding code carries the comment
# There is likely simpler than load into image into save, which suggests this block has not been revisited.
If the inversion is deliberate for some tensor convention I have missed, it would help to state that in the docstring, because it is surprising from the call site.
Suggested fix
Drop the 255 - in both places:
img = PIL.Image.fromarray((array * 255).astype(np.uint8))Worth handling two related cases while touching this: an integer tensor already in [0, 255] should not be multiplied again, and a float tensor outside [0, 1] should probably be clipped rather than wrapped by the uint8 cast.
Happy to open a PR with the fix and a round-trip test asserting that a known gradient survives to_raw() and to_string() unchanged.
Source: huggingface/smolagents