arXiv cs.AI paper: GUI-Lens uses repeated crops to help VLMs locate interface controls

arXiv cs.AI paper: GUI-Lens uses repeated crops to help VLMs locate interface controls

The authors of GUI-Lens argue that a VLM can recognize a requested control yet still fail to locate it precisely enough to click, especially on high-resolution, densely populated interfaces. Their method replaces a single direct click prediction with a sequence of focused visual observations: it uses OCR text and detected UI components as coordinate references, then lets the VLM choose a region and scale for the next enlarged crop. The process continues until the target is determined, and proposed crops and clicks are checked against the instruction. The authors report improved grounding accuracy across five GUI grounding benchmarks and six general-purpose base models, with up to 9.3 times the accuracy of direct grounding on ScreenSpot-Pro. This matters because the method makes localization a repeated, checked process rather than relying on one initial estimate; the evidence here, however, is benchmark performance, not a demonstration of reliability in deployed GUI agents. How does GUI-Lens perform on real-world interfaces beyond the five reported benchmarks?

Publication: 110e211a-0e66-451c-875d-57b771dadd30

attribution — pass: The draft attributes the argument, the method and the results to the GUI-Lens authors on arXiv. The paper's primary category is cs.CV, cross-listed to cs.AI, so the 'arXiv cs.AI paper' label matches the feed and is not misleading. Reported results are framed as the authors' reports, not as established facts.

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

summary — pass: The summary leads with the authors' argument: a model can recognize a control without locating it precisely enough to click. It then explains the crop-and-enlarge loop and the reported results in plain enough language. VLM, OCR, GUI grounding and ScreenSpot-Pro are not expanded inline. Small repair: write 'vision-language model (VLM)' and add tooltips for the other terms. The surrounding text still lets a general reader follow the central idea. The question is a genuine open issue, though it is repeated at the end of the summary.

GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents.

date — pass: originalPublished 2026-08-04 matches the v1 submission date. It is neither the 30 Oct 2026 v2 revision nor the November feed date.

[Submitted on 4 Aug 2026 (v1), last revised 30 Oct 2026 (this version, v2)]

support — pass: Each claim is directly supported by the abstract. The significance statement, that localization becomes a repeated, checked process rather than one initial estimate, follows from the source. The draft explicitly limits the evidence to benchmark performance rather than deployed reliability. No unsupported benefits or costs are implied.

Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates.

media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the source itself.