[Bug] Limitation of set of marks

I tried the prompt of Visit the official YC website and compile all enterprise information under the W25 B2B tag into a clear, well-structured table. Be sure to find all of it. from one of the Manus's use cases: https://manus.im/share/rVGPb6ocLpsjFkchJaP7iN?replay=1.

The agent opened the YC website sucessfully. However, it failed to select the W25 tag due there are too many marks from the marked image. See below:

This raises a clear issue of the limitation of set of marks. To address this, there may be three ways:

1. Do element selection in text. Do we really need set of marks on the screenshot? Can we just describe the elements and marks instead?
2. Support hybrid approach with mark and (x,y) axis method since there are some limitations, not all interactive elements could be identified; use (x,y) axis to interact with unlisted elements
3. Minimazing the marks overlaps. Maybe a better way to label the marks to avoid overlaps, which maybe not feasible if there are too many elements.
4. Agentic SoM. The SoM can turn off the SoM or only turn it on if it wants to interact with elements.or now seems 1,3,6,8 are promising solutions
5. Allow zooming in and out. May consume more tokens and require models to know when to zoom in and out.
6. Remove set of marks completely. Do we need marks at all? Since the computer use models such as claude-3-7-sonnet and openai [computer-use-preview] (https://platform.openai.com/docs/guides/tools-computer-use) can operate on the (x,y) axis, it maybe not necessary to the set of marks. Limitation: some models do not have ability to output (x,y) axis
7. Sent as multiple images. Distribute the marks into multiple images. But this may consume lots of tokens.
8. Add an element grouding step. Like Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI>Agents: https://arxiv.org/abs/2410.05243. But more computation and rely on the performance of the grounding model.

Above items are sorted based on priority