I run a local LLM agent (Hermes) on my own machine. The problem was never the model — it was the interface. I had a Telegram tab open all day just to talk to it: type a command, wait, read a wall of text back, scroll. It felt like texting a very capable stranger.
So I built ** Ghost Vessel** — a monitor-resident, video-call-style avatar that fronts the agent. The name is the whole idea: the
Here's what actually turned out to be interesting to build.
The core idea is an output contract. Instead of treating the agent's reply as text to print, I split every reply into three planes:
Emotion beats are inline tags the model emits in-band with its answer:
[working]
— the avatar puts on glasses and takes notes while a task runs[confirm] deploy to prod?
— pops a human-in-the-loop approve/cancel, and the agent blocks on your keypress`[happy]`
/ `[concerned]`
/ … — fine-grained facial expressionsSo "run the build, and if it passes, deploy" becomes a little performance: it looks busy while working, shows you the log as a card, then leans in and asks before the irreversible step. The text you'd have skim-read becomes something you glance at.
The obvious way to animate a face is live inference. I didn't want that — the GPU is busy running the actual model.
Instead the avatar is ~30 pre-rendered clips, and the emotion beats just select and blend between them (blink-aligned seamless idle loops, a head-pose "settle gate" so an expression only reveals when the head is frontal). The avatar's runtime cost is basically video playback. Your GPU stays 100% on your LLM.
The tradeoff: no real-time lip-sync. I decided a believable talking mouth loop + expressive face reads as "a person on a call" far more than perfect phoneme-matching does, and it costs nothing at runtime. That single decision collapsed most of the hard engineering.
The agent integration is the part I was most unsure would work. It turned out clean: the app registers as a connector. The agent's gateway dials out to a local WebSocket the app hosts, and exchanges frames — so from the agent's side, the avatar is just "another channel," indistinguishable from Telegram.
Adapters for Hermes and OpenClaw are included, plus a demo responder so it runs with zero setup. And the chat pane serves the agent's live slash-command menu — type /
and you get the same 52 commands you'd see in the messenger, passed straight through.
The engine is MIT and usable on its own: clone it, drop a folder of clips named by emotion (happy.mp4
, working.mp4
, idle.mp4
, …), point it at your agent. Avatars are pure-data bundles — no code runs when you install one — so you can build your own; the full reproducible method is in the repo.
The part I'd most like feedback on is the emotion-beat output format — the contract that turns LLM text into a UI performance. Has anyone else built output contracts to drive an interface from model output? What broke, and what did models reliably get right vs. wrong? I'd genuinely like to compare notes.