Handrail takes a screenshot of whatever you are stuck in, answers your question about it, and then draws an arrow on the real control you need to touch. The obvious way to build that is one call: here is the screen, here is the question, give me the answer and the coordinates.
That is how I built it first, and it is worse at both halves.
Handrail now makes two vision calls against the same screenshot.
Separating them helped for a reason that is obvious in hindsight. The answering pass is a reading and reasoning task over the whole screen. The pointing pass is a localisation task over one named target. Asking for both in one response makes the model hold a spatial answer in working memory while it composes prose, and the coordinates are the part that degrades.
It also makes failure legible. If the answer is right and the arrow is wrong, you know exactly which pass to fix.
The second pass does not return pixels. It returns a position on a 0-1000 normalised grid, which Handrail then maps onto actual screen pixels.
This sounds like a detail and it removes an entire category of bug. Screens differ in resolution, in DPI scaling, and in how many of them are plugged in. If the model returns pixels, every one of those becomes arithmetic you have to get right on someone else's hardware. Normalised coordinates are resolution-independent by definition, so DPI and multi-monitor never enter the maths at all.
The two passes also do not get the same image. Answering uses a 1600px JPEG, which is enough to read a screen and cheap to send. Locating uses a native-resolution PNG, because the thing you are pinpointing may be a 12px chevron.
Splitting the work also changed which model I could use. The default is now the cheapest one that reliably reads a screen. I tried going cheaper still, and the lite tier failed in the most dangerous way available to a screen assistant: it invented menu paths for UI that was not in the screenshot.
That is the failure mode worth designing against. A model that says it cannot see the control is recoverable. A model that confidently names a menu item that does not exist sends someone hunting through Settings for something that was never there, and the whole product exists to stop exactly that.
Since the thing is looking at your screen, the trust boundary matters more than the features.
A screen assistant that phones home is a different product, and a worse one.
Handrail is open source, Apache-2.0, with tagged Windows and macOS releases: https://github.com/M19K/handrail