{"slug": "asking-a-vision-model-what-and-where-in-the-same-call-makes-it-worse-at-both", "title": "Asking a vision model what and where in the same call makes it worse at both", "summary": "A developer built Handrail, an open-source (Apache-2.0) screen assistant that splits visual question-answering and UI element localization into two separate vision model calls against the same screenshot, finding that combining them degraded coordinate accuracy. The localization pass returns positions on a 0-1000 normalized grid rather than pixels, making it resolution- and DPI-independent, and the answering pass uses a 1600px JPEG while localization uses a native-resolution PNG. The developer reports that a cheaper \"lite\" model tier fabricated menu paths for UI not present in the screenshot, a failure mode the two-pass design aims to make legible and avoid.", "body_md": "Handrail takes a screenshot of whatever you are stuck in, answers your question about it, and then draws an arrow on the real control you need to touch. The obvious way to build that is one call: here is the screen, here is the question, give me the answer and the coordinates.\n\nThat is how I built it first, and it is worse at both halves.\n\nHandrail now makes two vision calls against the same screenshot.\n\nSeparating them helped for a reason that is obvious in hindsight. The answering pass is a reading and reasoning task over the whole screen. The pointing pass is a localisation task over one named target. Asking for both in one response makes the model hold a spatial answer in working memory while it composes prose, and the coordinates are the part that degrades.\n\nIt also makes failure legible. If the answer is right and the arrow is wrong, you know exactly which pass to fix.\n\nThe second pass does not return pixels. It returns a position on a **0-1000 normalised grid**, which Handrail then maps onto actual screen pixels.\n\nThis sounds like a detail and it removes an entire category of bug. Screens differ in resolution, in DPI scaling, and in how many of them are plugged in. If the model returns pixels, every one of those becomes arithmetic you have to get right on someone else's hardware. Normalised coordinates are resolution-independent by definition, so DPI and multi-monitor never enter the maths at all.\n\nThe two passes also do not get the same image. Answering uses a 1600px JPEG, which is enough to read a screen and cheap to send. Locating uses a native-resolution PNG, because the thing you are pinpointing may be a 12px chevron.\n\nSplitting the work also changed which model I could use. The default is now the cheapest one that reliably reads a screen. I tried going cheaper still, and the lite tier failed in the most dangerous way available to a screen assistant: it invented menu paths for UI that was not in the screenshot.\n\nThat is the failure mode worth designing against. A model that says it cannot see the control is recoverable. A model that confidently names a menu item that does not exist sends someone hunting through Settings for something that was never there, and the whole product exists to stop exactly that.\n\nSince the thing is looking at your screen, the trust boundary matters more than the features.\n\nA screen assistant that phones home is a different product, and a worse one.\n\nHandrail is open source, Apache-2.0, with tagged Windows and macOS releases: [https://github.com/M19K/handrail](https://github.com/M19K/handrail)", "url": "https://wpnews.pro/news/asking-a-vision-model-what-and-where-in-the-same-call-makes-it-worse-at-both", "canonical_source": "https://dev.to/maazkazi/asking-a-vision-model-what-and-where-in-the-same-call-makes-it-worse-at-both-26e9", "published_at": "2026-09-25 12:23:29+00:00", "updated_at": "2026-09-25 12:31:02.061291+00:00", "lang": "en", "topics": ["computer-vision", "ai-tools", "ai-products", "ai-agents"], "entities": ["Handrail", "GitHub", "M19K"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/asking-a-vision-model-what-and-where-in-the-same-call-makes-it-worse-at-both", "markdown": "https://wpnews.pro/news/asking-a-vision-model-what-and-where-in-the-same-call-makes-it-worse-at-both.md", "text": "https://wpnews.pro/news/asking-a-vision-model-what-and-where-in-the-same-call-makes-it-worse-at-both.txt", "jsonld": "https://wpnews.pro/news/asking-a-vision-model-what-and-where-in-the-same-call-makes-it-worse-at-both.jsonld"}}