{"slug": "stop-describing-pixels-to-your-ai-agent-annotate-them-instead", "title": "Stop describing pixels to your AI agent — annotate them instead", "summary": "A developer open-sourced Pinpoint, a macOS menu-bar app that lets users annotate screenshots with numbered markers and structured text blocks, solving the ambiguity of describing UI elements in prose to AI coding agents like Claude Code or Cursor. The tool captures a region, adds numbered pins and notes, then copies both an annotated image and a text legend mapping each marker to a percentage-based position, ensuring agents can reliably reference specific elements even when chat UIs discard clipboard text.", "body_md": "If you pair-program with an AI agent — Claude Code, Codex, Cursor, whatever — you've lived this moment. You take a screenshot of your UI, paste it into the chat, and then you start typing:\n\n\"See the button in the top-right? No, not that one, the\n\notherone, next to the misaligned icon — make it full width on mobile and fix the icon while you're at it.\"\n\nYou're describing pixels in prose. And the agent, bless it, does its best to guess which of the six buttons in the screenshot you actually meant.\n\nIt gets worse. A lot of chat UIs only keep the **image** from your clipboard and silently throw away any text you copied alongside it. So even the careful caption you wrote never makes it through. Half your context evaporates between Cmd+C and Cmd+V.\n\nI hit this wall every single day. So I built a tool to fix it, and I open-sourced it. This post is about the underlying idea more than the tool — but I'll use the tool to make it concrete.\n\nHumans don't describe pixels to each other. We point. \"This one.\" \"Right here.\" We use our finger, a cursor, a circle drawn in the air.\n\nSo the fix is obvious once you say it out loud: **annotate the image with markers, and give each marker a stable identity the agent can reference.** Instead of \"the button in the top-right,\" you say \"marker 1,\" and the image literally has a numbered pin on that button.\n\nBut a visual marker alone isn't enough, because the agent reads the image as a flat raster. It doesn't *know* that the ring labeled \"1\" sits at 62% across and 48% down. So the second half of the idea is: **alongside the annotated image, emit a structured text block that maps every marker to a position and a description**, then appends your actual instructions.\n\nPut together, a single paste carries everything the agent needs:\n\n```\n# Annotated capture — 1280×800 px\n\nAn image is attached. Numbered (ringed) badges point to specific elements.\nMarkers (position in % of the image, top-left origin):\n\n1. Primary CTA button · ~62 % × 48 %\n2. Misaligned icon · ~12 % × 22 %\n\n## Instructions\nMake the CTA full-width on mobile and fix the icon alignment.\n```\n\nNow the agent doesn't guess. \"Marker 1 is the primary CTA at roughly 62% × 48%. Marker 2 is the misaligned icon near the top-left.\" The ambiguity is gone, and your instructions are tied to concrete anchors instead of fuzzy spatial language.\n\nA small but important detail: positions are expressed as **percentages of the image**, with a top-left origin — not absolute pixel coordinates. Agents reason about layout relatively (\"near the top-left,\" \"centered horizontally\") far more reliably than they handle raw pixel math, and percentages survive any downscaling the chat UI applies to your image before the model sees it. If the platform resizes your 1280px capture to 768px, \"62%\" still points at the same button. \"794px\" doesn't.\n\nRemember the chat UIs that keep only the image? There's a defense for that too: **bake the legend into the picture.** Render the marker descriptions and instructions directly onto the exported image as a caption block. It's redundant with the clipboard text on purpose — if the platform keeps both, great; if it keeps only the image, the legend is still there, pixels and all. Belt and suspenders.\n\nAn idea like this only pays off if invoking it costs less than just typing the description would. If annotating takes 30 seconds of fiddling, you'll skip it. So the workflow has to be near-instant:\n\nThat's the bar: capture-to-clipboard in under ten seconds, from anywhere, without leaving what you were doing. On macOS that means a menu-bar app, `ScreenCaptureKit`\n\nfor native-resolution multi-display capture, and rebindable global shortcuts. It stays out of your way until you hit the hotkey.\n\nThe implementation of all this is **Pinpoint** — a native macOS menu-bar app, built in SwiftUI + ScreenCaptureKit. Capture a region with a shortcut, drop numbered markers, add notes and arrows, and copy an annotated image plus the structured prompt above. Everything stays local; captures never leave your Mac. It's free, MIT-licensed, notarized and signed.\n\nI built it for myself and use it every day. The interesting part for me was less the Swift and more the format question: *what does a screenshot prompt actually need to contain for an agent to act on it without follow-up questions?* The percentage-based marker map plus a baked-in legend is my current answer — but I'd genuinely like to hear how others are solving this.\n\nThe pattern generalizes beyond macOS and beyond this app: when you hand an agent an image, **don't make it infer what you're pointing at.** Give it stable, named anchors and a machine-readable map of where they are. Whether you build that into your own workflow, a browser extension, or a CLI, the principle holds — pointing beats describing, and a structured map beats a paragraph of spatial prose.\n\nIf you're shipping UI work with an AI agent, try replacing your next \"the thing in the corner, no the other one\" with a numbered marker. You'll feel the difference on the first paste.\n\n*Pinpoint is open source (MIT). Issues and PRs welcome — there are a couple of good-first-issues open. If you have a better prompt format for annotated captures, I want to see it.*", "url": "https://wpnews.pro/news/stop-describing-pixels-to-your-ai-agent-annotate-them-instead", "canonical_source": "https://dev.to/croustibat44/stop-describing-pixels-to-your-ai-agent-annotate-them-instead-1ofp", "published_at": "2026-07-15 10:03:00+00:00", "updated_at": "2026-07-15 10:30:19.449455+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "artificial-intelligence"], "entities": ["Pinpoint", "Claude Code", "Codex", "Cursor", "macOS", "SwiftUI", "ScreenCaptureKit"], "alternates": {"html": "https://wpnews.pro/news/stop-describing-pixels-to-your-ai-agent-annotate-them-instead", "markdown": "https://wpnews.pro/news/stop-describing-pixels-to-your-ai-agent-annotate-them-instead.md", "text": "https://wpnews.pro/news/stop-describing-pixels-to-your-ai-agent-annotate-them-instead.txt", "jsonld": "https://wpnews.pro/news/stop-describing-pixels-to-your-ai-agent-annotate-them-instead.jsonld"}}