Half the posts on my feed this week are some version of "what even is an AI agent." If-statements in a trench coat. Memory vs. RAG. Verification bottlenecks. All good debates — and all about the agent's brain.
Nobody's talking about its hands.
Here's the wall I kept hitting building an autonomous agent: the model can decide, perfectly, that it should reply to a lead, check the company mailbox, or post an update. Then it just... can't. It has no way to actually do it. An LLM emits text; it doesn't click "Post" on a logged-in account.
The usual answer is a scraper or a Playwright script with CSS selectors. Two problems:
.btn-primary-v2 becomes .button__cta, and every script breaks silently.
So I built the missing piece and open-sourced it: Ghost Browser — a real Chromium an agent operates the way a person does.
The core trick is Set-of-Mark. Before any decision, every interactive element on screen gets a numbered box painted over it in the live DOM, the page is screenshotted, the boxes are stripped off again, and the agent gets the annotated picture plus a numbered list. Then it just says click 12.
That's the whole reason it survives redesigns where selectors don't: a numbered screenshot is redrawn from whatever is on screen right now, and a language model is far better at looking at a picture than at parsing a DOM tree. No selector to get subtly wrong.
You sign into a site by hand, once, in a console. That session lives in a profile — isolated cookies and storage — and from then on the agent operates that real, logged-in session. No credentials in scripts, no re-auth loop. One profile per account, each labelled with the site it's signed into.
Reading is free. Posting, messaging, following, joining are not — they happen under a real name and the notification has already reached a person. So those go through an act-gate: the agent shows you the exact text and waits. Approve, edit, or reject.
It's enforced twice — the prompt says so, and a guard inspects the label of anything it's about to click and turns a "Post" or "Join" into a proposal anyway. "Let it act without asking" exists, it's off by default, and it should stay off until you've read a few of its drafts.
This is the tool a fully autonomous agent on my own platform uses for anything that needs a real browser — a master agent plans the work, specialised organs handle research/outreach/publishing, and every time one needs to touch the real web as a real account, it calls Ghost Browser. Without it, the whole thing is a planner with no way to act.
A few things fall out of that design for free:
The engine is solid and production-tested. The console UI... was built by an engineer, not a designer, and it shows. If UI/UX is your thing, this is a project where your help lands immediately — there are issues tagged ui and good first issue waiting.
Repo, setup guide (any VPS in ten minutes), and runnable examples: https://github.com/Wvdstoep/ghost-browser What's your take — is "hands" the missing layer in the agent stack, or am I solving a problem you'd solve a different way? Genuinely curious.