#10974·tldraw

Agent treats partially visible viewport shapes as peripheral

Author: kvnlooCreated Sep 28, 2026Updated Sep 28, 2026

Problem

The agent's three default spatial prompt parts disagree with what "visible in the viewport" means.

On current main:

  • BlurryShapesPartUtil includes a shape only when contextBoundsBox.includes(bounds)
  • ScreenshotPartUtil uses the same full-containment check before deciding which shapes to render
  • PeripheralShapesPartUtil classifies every shape for which that containment check is false as outside the viewport

So a shape that intersects the viewport but crosses one edge is treated as peripheral rather than visible.

This is especially noticeable with large frames/images/diagrams: the user or agent can visibly see part of the shape on the real canvas, while the model-facing screenshot omits that shape entirely and the structured visible-shape list omits it too.

Expected

A shape whose masked page bounds collide with the agent context bounds should count as visible.

  • visible/intersecting shapes should be eligible for the screenshot and blurry-shape representation
  • peripheral clusters should contain shapes with no viewport intersection

The viewport can still crop the screenshot to request.bounds; the selected shape set just needs to include objects that contribute pixels inside those bounds.

Example

Viewport: { x: 0, y: 0, w: 500, h: 500 }

Shape bounds: { x: 450, y: 100, w: 200, h: 100 }

The shape contributes its leftmost 50px to the user's view, but viewport.includes(shapeBounds) is false, so today it is:

  • absent from blurryShapes
  • absent from the screenshot render input
  • present in peripheralShapes

Suggested direction

Use bounds collision / intersection for visible classification (e.g. the existing Box.collides API) and reserve full containment only for cases that truly require it.

It would also help to centralize this classification so screenshot, blurry, and peripheral parts cannot drift apart.

Acceptance criteria

  • partially visible shapes appear in the screenshot input
  • partially visible shapes appear in blurryShapes
  • partially visible shapes do not appear in peripheralShapes
  • fully outside shapes remain peripheral
  • fully inside shapes preserve existing behavior
  • tests cover each boundary case, including shapes larger than the viewport