Agent treats partially visible viewport shapes as peripheral
Problem
The agent's three default spatial prompt parts disagree with what "visible in the viewport" means.
On current main:
BlurryShapesPartUtilincludes a shape only whencontextBoundsBox.includes(bounds)ScreenshotPartUtiluses the same full-containment check before deciding which shapes to renderPeripheralShapesPartUtilclassifies every shape for which that containment check is false as outside the viewport
So a shape that intersects the viewport but crosses one edge is treated as peripheral rather than visible.
This is especially noticeable with large frames/images/diagrams: the user or agent can visibly see part of the shape on the real canvas, while the model-facing screenshot omits that shape entirely and the structured visible-shape list omits it too.
Expected
A shape whose masked page bounds collide with the agent context bounds should count as visible.
- visible/intersecting shapes should be eligible for the screenshot and blurry-shape representation
- peripheral clusters should contain shapes with no viewport intersection
The viewport can still crop the screenshot to request.bounds; the selected shape set just needs to include objects that contribute pixels inside those bounds.
Example
Viewport: { x: 0, y: 0, w: 500, h: 500 }
Shape bounds: { x: 450, y: 100, w: 200, h: 100 }
The shape contributes its leftmost 50px to the user's view, but viewport.includes(shapeBounds) is false, so today it is:
- absent from
blurryShapes - absent from the screenshot render input
- present in
peripheralShapes
Suggested direction
Use bounds collision / intersection for visible classification (e.g. the existing Box.collides API) and reserve full containment only for cases that truly require it.
It would also help to centralize this classification so screenshot, blurry, and peripheral parts cannot drift apart.
Acceptance criteria
- partially visible shapes appear in the screenshot input
- partially visible shapes appear in
blurryShapes - partially visible shapes do not appear in
peripheralShapes - fully outside shapes remain peripheral
- fully inside shapes preserve existing behavior
- tests cover each boundary case, including shapes larger than the viewport
Source: tldraw/tldraw