{"slug": "your-ai-agent-can-plan-anything-it-still-can-t-click-post", "title": "Your AI agent can plan anything. It still can't click \"Post\".", "summary": "A developer has open-sourced Ghost Browser, a Chromium-based tool that lets AI agents operate real, logged-in browser sessions the way a person does. The project uses a Set-of-Mark technique that paints numbered boxes over interactive elements in the live DOM, screenshots the page, and lets the model act by referencing numbers rather than fragile CSS selectors. Risky actions like posting or messaging are gated behind a human approval step, with the developer noting the console UI still needs design help.", "body_md": "Half the posts on my feed this week are some version of \"what even *is* an AI agent.\" If-statements in a trench coat. Memory vs. RAG. Verification bottlenecks. All good debates — and all about the agent's *brain*.\n\nNobody's talking about its **hands**.\n\nHere's the wall I kept hitting building an autonomous agent: the model can decide, perfectly, that it should reply to a lead, check the company mailbox, or post an update. Then it just... can't. It has no way to actually do it. An LLM emits text; it doesn't click \"Post\" on a logged-in account.\n\nThe usual answer is a scraper or a Playwright script with CSS selectors. Two problems:\n\n`.btn-primary-v2` becomes `.button__cta`, and every script breaks silently.\nSo I built the missing piece and open-sourced it: [**Ghost Browser**](https://github.com/Wvdstoep/ghost-browser) — a real Chromium an agent operates the way a person does.\n\nThe core trick is **Set-of-Mark**. Before any decision, every interactive element on screen gets a numbered box painted over it *in the live DOM*, the page is screenshotted, the boxes are stripped off again, and the agent gets the annotated picture plus a numbered list. Then it just says `click 12`.\n\nThat's the whole reason it survives redesigns where selectors don't: a numbered screenshot is redrawn from whatever is on screen *right now*, and a language model is far better at looking at a picture than at parsing a DOM tree. No selector to get subtly wrong.\n\nYou sign into a site **by hand, once**, in a console. That session lives in a profile — isolated cookies and storage — and from then on the agent operates that real, logged-in session. No credentials in scripts, no re-auth loop. One profile per account, each labelled with the site it's signed into.\n\nReading is free. Posting, messaging, following, joining are not — they happen under a real name and the notification has already reached a person. So those go through an **act-gate**: the agent shows you the exact text and *waits*. Approve, edit, or reject.\n\nIt's enforced twice — the prompt says so, and a guard inspects the label of anything it's about to click and turns a \"Post\" or \"Join\" into a proposal anyway. \"Let it act without asking\" exists, it's off by default, and it should stay off until you've read a few of its drafts.\n\nThis is the tool a fully autonomous agent on my own platform uses for *anything* that needs a real browser — a master agent plans the work, specialised organs handle research/outreach/publishing, and every time one needs to touch the real web as a real account, it calls Ghost Browser. Without it, the whole thing is a planner with no way to act.\n\nA few things fall out of that design for free:\n\nThe engine is solid and production-tested. The console UI... was built by an engineer, not a designer, and it shows. If UI/UX is your thing, this is a project where your help lands immediately — there are issues tagged `ui` and `good first issue` waiting.\n\nRepo, setup guide (any VPS in ten minutes), and runnable examples: [https://github.com/Wvdstoep/ghost-browser](https://github.com/Wvdstoep/ghost-browser)\n\nWhat's your take — is \"hands\" the missing layer in the agent stack, or am I solving a problem you'd solve a different way? Genuinely curious.", "url": "https://wpnews.pro/news/your-ai-agent-can-plan-anything-it-still-can-t-click-post", "canonical_source": "https://dev.to/wvdstoep/your-ai-agent-can-plan-anything-it-still-cant-click-post-2bpb", "published_at": "2026-09-12 11:31:03+00:00", "updated_at": "2026-09-12 11:39:24.595329+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Ghost Browser", "Chromium", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/your-ai-agent-can-plan-anything-it-still-can-t-click-post", "markdown": "https://wpnews.pro/news/your-ai-agent-can-plan-anything-it-still-can-t-click-post.md", "text": "https://wpnews.pro/news/your-ai-agent-can-plan-anything-it-still-can-t-click-post.txt", "jsonld": "https://wpnews.pro/news/your-ai-agent-can-plan-anything-it-still-can-t-click-post.jsonld"}}