If you pair-program with an AI agent — Claude Code, Codex, Cursor, whatever — you've lived this moment. You take a screenshot of your UI, paste it into the chat, and then you start typing:
"See the button in the top-right? No, not that one, the
otherone, next to the misaligned icon — make it full width on mobile and fix the icon while you're at it."
You're describing pixels in prose. And the agent, bless it, does its best to guess which of the six buttons in the screenshot you actually meant.
It gets worse. A lot of chat UIs only keep the image from your clipboard and silently throw away any text you copied alongside it. So even the careful caption you wrote never makes it through. Half your context evaporates between Cmd+C and Cmd+V.
I hit this wall every single day. So I built a tool to fix it, and I open-sourced it. This post is about the underlying idea more than the tool — but I'll use the tool to make it concrete.
Humans don't describe pixels to each other. We point. "This one." "Right here." We use our finger, a cursor, a circle drawn in the air.
So the fix is obvious once you say it out loud: annotate the image with markers, and give each marker a stable identity the agent can reference. Instead of "the button in the top-right," you say "marker 1," and the image literally has a numbered pin on that button.
But a visual marker alone isn't enough, because the agent reads the image as a flat raster. It doesn't know that the ring labeled "1" sits at 62% across and 48% down. So the second half of the idea is: alongside the annotated image, emit a structured text block that maps every marker to a position and a description, then appends your actual instructions.
Put together, a single paste carries everything the agent needs:
An image is attached. Numbered (ringed) badges point to specific elements.
Markers (position in % of the image, top-left origin):
1. Primary CTA button · ~62 % × 48 %
2. Misaligned icon · ~12 % × 22 %
## Instructions
Make the CTA full-width on mobile and fix the icon alignment.
Now the agent doesn't guess. "Marker 1 is the primary CTA at roughly 62% × 48%. Marker 2 is the misaligned icon near the top-left." The ambiguity is gone, and your instructions are tied to concrete anchors instead of fuzzy spatial language.
A small but important detail: positions are expressed as percentages of the image, with a top-left origin — not absolute pixel coordinates. Agents reason about layout relatively ("near the top-left," "centered horizontally") far more reliably than they handle raw pixel math, and percentages survive any downscaling the chat UI applies to your image before the model sees it. If the platform resizes your 1280px capture to 768px, "62%" still points at the same button. "794px" doesn't.
Remember the chat UIs that keep only the image? There's a defense for that too: bake the legend into the picture. Render the marker descriptions and instructions directly onto the exported image as a caption block. It's redundant with the clipboard text on purpose — if the platform keeps both, great; if it keeps only the image, the legend is still there, pixels and all. Belt and suspenders.
An idea like this only pays off if invoking it costs less than just typing the description would. If annotating takes 30 seconds of fiddling, you'll skip it. So the workflow has to be near-instant:
That's the bar: capture-to-clipboard in under ten seconds, from anywhere, without leaving what you were doing. On macOS that means a menu-bar app, ScreenCaptureKit
for native-resolution multi-display capture, and rebindable global shortcuts. It stays out of your way until you hit the hotkey.
The implementation of all this is Pinpoint — a native macOS menu-bar app, built in SwiftUI + ScreenCaptureKit. Capture a region with a shortcut, drop numbered markers, add notes and arrows, and copy an annotated image plus the structured prompt above. Everything stays local; captures never leave your Mac. It's free, MIT-licensed, notarized and signed.
I built it for myself and use it every day. The interesting part for me was less the Swift and more the format question: what does a screenshot prompt actually need to contain for an agent to act on it without follow-up questions? The percentage-based marker map plus a baked-in legend is my current answer — but I'd genuinely like to hear how others are solving this.
The pattern generalizes beyond macOS and beyond this app: when you hand an agent an image, don't make it infer what you're pointing at. Give it stable, named anchors and a machine-readable map of where they are. Whether you build that into your own workflow, a browser extension, or a CLI, the principle holds — pointing beats describing, and a structured map beats a paragraph of spatial prose.
If you're shipping UI work with an AI agent, try replacing your next "the thing in the corner, no the other one" with a numbered marker. You'll feel the difference on the first paste.
Pinpoint is open source (MIT). Issues and PRs welcome — there are a couple of good-first-issues open. If you have a better prompt format for annotated captures, I want to see it.