Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

A new arXiv preprint (2608.21794v1) finds that GUI grounding evaluations using sentence embeddings often conflate visible-label recovery with semantic grounding, as lexical baselines remain competitive at top-1 across three mobile and web benchmarks. The authors recommend reporting lexical baselines, label-type stratification, and deployable-fusion diagnostics, and release analysis scripts at https://github.com/qijia123/lexical-coupling-release.

arXiv:2608.21794v1 Announce Type: new Abstract: GUI grounding evaluations that expose UI elements as text metadata often treat high instruction-element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks, we show that this interpretation is frequently confounded by visible-label recovery. Lexical baselines remain competitive at top-1, label-poor targets remain weak for text-only methods, and encoder top-1 hits are predictable from lexical rank, candidate-pool size, and label type. We evaluate each action as a same-screen ranking task, comparing five off-the-shelf single-vector encoders with lexical baselines. Encoders recover some lexical misses, but deployable fusion gains are much smaller than target-aware oracle gains. These findings show that embedding-based evaluations can conflate visible-label recovery with semantic GUI grounding. Embedding-based evaluations should therefore report lexical baselines, label-type stratification, and deployable-fusion diagnostics. Our released repository provides analysis scripts and detexted per-step panels: https://github.com/qijia123/lexical-coupling-release.