Show HN: Self-Hosted AI That Sees, Plays Minecraft, and Reacts

An open-source AI streamer called Wallie can now play Minecraft autonomously, narrating its actions in character while gathering, crafting, and fighting. The system runs locally, supports customizable personas and voices, and aims to create a believable, non-repetitive streaming experience.

Wallie main demo.mp4 The open-source AI streamer that sees, hears & reacts — and actually feels alive. Quick Start quick-start · Features features · Dashboard the-dashboard · Architecture architecture · Roadmap roadmap ⭐ If you think an AI that watchesTwo seconds, and it genuinely helps a solo dev keep shipping.and hearsyour screen should exist — give it a star. Every AI streamer you've seen has the same problem. It says two sentences. Pauses. Says two more. Pauses. Describes your YouTube homepage like a screen reader. Forgets what it said thirty seconds ago. Ends every thought with "what do you guys think?" into a chat that doesn't exist. It's not a show — it's a tech demo on loop. Wallie is different. It's an AI that actually streams — develops thoughts across minutes, reacts to your screen like a person who's been using that computer all day, remembers what it covered an hour ago, takes opinions and sticks with them, drifts between topics the way real conversations drift, and shuts up when there's nothing worth saying. You design the streamer. Wallie runs the show. persona ──┐ topics ──┤ chat ──┤ vision ──┼──► Wallie ──► Voice TTS ──► OBS / virtual cable ──► your stream hearing ──┤ ──► Live2D avatar VTube Studio schedule ──┘ Pick the personality. Pick the voice. Pick the LLM. Pick the platform. Everything is swappable, everything is configurable from the browser, and the whole thing runs on your machine with your keys. Wallie doesn't just watch your screen anymore — it picks up the game and plays it. Right now that's Minecraft: it gathers, crafts, builds, fights, and explores entirely on its own, narrating every moment in character. Smooth, human-looking camera. No human at the keyboard, no script — a real playthrough with a personality. https://www.youtube.com/shorts/NEfsmppOXJk https://www.youtube.com/shorts/NEfsmppOXJk And here's the part that makes it believable: the commentary is grounded in what Wallie is actually doing — its real inventory, health, and current goal — not guessed from a blurry frame. So it says "heading down to mine diamonds" because it is, and brings up "that one time I dug straight into a lava pit" because that's its run. It actually plays — a planning brain sets the goal; Baritone https://github.com/cabaletta/baritone handles pathfinding & mining, and a custom Fabric mod handles crafting, combat, item pickup, and a buttery-smooth camera that looks human, not robotic. Grounded commentary — the streamer core stays in sync with the playing brain, so what it says matches what it does . One-click setup — the dashboard installs Fabric + every mod for you and backs up your current ones first . Any persona, any model — Play is just another mode. Vision, Hearing, and Play each toggle on/off independently. Flip on Playin the dashboard, pick a goal, and go — or jump straight in: python scripts/run wallie live.py Because the existing ones are toys. They work for a 2-minute demo and fall apart on a real stream. Here's what actually goes wrong after 10 minutes: | The problem | What usually happens | What Wallie does | |---|---|---| Repetition | Says "that's interesting" every 30 seconds | Dedupe engine + phrase cooldown + rolling summary that tracks everything already said | Short memory | Forgets the topic from 5 minutes ago | Rolling summarizer compresses old turns into bullet notes, injected into every prompt | Robotic vision | "I can see a YouTube homepage with several videos" | First-person ownership + SKIP escape hatch — narrates nothing, reacts to what matters | Question loops | Every segment ends with "what do you think, chat?" | Question detector throttles after one; next prompt forces a statement ending | No personality | Generic helpful assistant voice | Full persona system: energy, humor style, catchphrases, backstory, opinions, taboo topics | Choppy pacing | 2 sentences → long pause → 2 sentences | Pipeline overlap + longer thought development eliminates dead air | Broken threads | Sets up a story, never delivers it | Open-thread tracker forces the next segment to pay off teases and answer its own questions | Topic whiplash | Jumps from AI to cooking to space with no bridge | Association-based topic drift with configurable style rigid / natural / freeform | These aren't edge cases. They're the default behavior of most AI streamer projects. Wallie was built by hitting every one of these problems and refusing to ship until they were fixed. Six LLM providers. Three TTS engines. Three chat platforms. Mix and match per profile. | LLM | TTS | Chat | Avatar | |---|---|---|---| | OpenAI | Fish Audio | Twitch | VTube Studio Live2D | | Anthropic Claude | ElevenLabs | YouTube | — | | Google Gemini | Piper local, free | Kick | — | | Groq | — | — | — | | OpenRouter | — | — | — | | Ollama local, free | — | — | — | Swap providers without changing code. Run a fully offline stream with Ollama + Piper, or go premium with Claude + ElevenLabs. Not just a name and a system prompt. You design a character : Identity — name, handle, pronouns, age range, origin, archetype, backstory Voice — energy level chill → unhinged , humor style pick multiple: ironic, deadpan, absurd, observational, roast... , profanity level, formality Flavor — catchphrases, running gags, banned words, favorite topics, taboo topics Opinions — strong opinions toggle, admit uncertainty, break the fourth wall Extra notes — free text for anything the structured fields miss Save multiple personas. Switch between them with a dropdown. The single biggest technical challenge. Most AI chatbots break down after 20 minutes because the context window fills up. Wallie's approach: Rolling summarizer — every ~14 segments, a background LLM call compresses older history into tight bullet notes Session notes — that compressed memory is injected into every system prompt, so the streamer knows what it already said Cross-session memory — key facts and viewer interactions persist across streams Dedupe engine — paraphrase-aware similarity check bigram + trigram Jaccard catches the model repeating itself in different words Real streamers don't talk at a constant rate. They get excited, they pause to think, they go on tangents, they settle into a flow. Mood engine — arousal, valence, focus, and talkativity drift slowly over the stream. High arousal = faster, warmer output. Low talkativity = silence beats. Mood state feeds into the avatar for reactive animation. Attention engine — vision events are filtered through a probabilistic decision layer. Not every screen change gets a deep reaction. Some get a glance. Some get ignored. Some trigger a personal tangent. Silence beats — the streamer holds natural pauses when the mood says to. Dead air handled by design, not by accident. Pipeline overlap — next segment starts generating while current audio is still playing. No awkward 3-second gaps between thoughts. wallie minecraft.mp4 Screen reactions are the hardest part to get right. Wallie's approach: First-person ownership — Wallie owns whatever is on screen. Gaming: "I just got bodied by that boss", not "the character is fighting a boss". Browsing: "let me pull this up", not "the user is browsing". Never third-person, never narration. SKIP escape hatch — if there's nothing specific to name, the model outputs SKIP and stays quiet. No more narrating generic UIs. Activity adaptation — detects scrolling, typing, app-switching, video playback and adjusts reactions accordingly. Typing → ignore. App switch → react. Rapid browsing → wait until user settles. Each activity type gets context-aware prompting. Attention engine — not every screen change gets the same treatment. DEEP reactions 22% , quick GLANCEs 28% , personal TANGENTs 5% , deliberate IGNOREs 27% , and SILENCE beats 18% . Streak fatigue prevents reacting the same way twice in a row. Scene memory — remembers what it last said about the current screen. Dedupe threshold at 0.65 with per-sentence comparison catches paraphrased repetition. Wallie captures your system audio WASAPI loopback and reacts to it live, fused with vision into a single reaction — sight and sound, one voice. Just like you organize your senses, not two narrators talking over each other. Speech & lyrics — local speech-to-text faster-whisper transcribes videos, voice chat, and song lyrics. Auto-uses your GPU if available, clean CPU fallback if not. Real music understanding — pure-numpy DSP, no extra ML weight. It reads major/minor key → mood , tempo + beat strength, instrumentation texture bass-heavy / acoustic / airy / lo-fi , and production quality crisp / muddy / harsh . A sad song reads as melancholy; a beat drop reads as energy. It'll call a track a banger or roast a muddy mix. Multimodal fusion — what it hears is fused with what it sees into one coherent reaction, never two competing ones. No self-echo — content-based guard so Wallie never reacts to its own TTS bleeding back through the loopback. Pure-hearing mode — run it with vision off and it reacts to audio the way it reacts to a screen: stays quiet, listens, then reacts to what's playing. Enable it in the dashboard's Hearing section. Needs two extra deps: pip install soundcard faster-whisper . Play mode hands Wallie the controls. First game: Minecraft, survival, live and unscripted. Reliable autonomy — a planning brain picks high-level goals; a deterministic skill library crafting, smelting, full tool/armour sets, mining to ore depths, hunting for food & wool, beds, combat with weapon selection, dropped-item pickup does the actual work — so even a small/free model plays competently instead of flailing. Human-like camera — a custom Fabric mod eases every turn at full FPS. No snapping, no teleport-cam — it reads like a person playing. Grounded commentary — reactions come from the agent's real game state inventory, health, threats, current goal , fused through the same single pipeline as vision and hearing. No screen-guessing, no hallucinated "desert temples". One-click install — Fabric + all mods straight from the dashboard's Play section. Vision/Hearing/Play toggle independently, on any persona. Not just a mouth that opens and closes. Six animation layers run simultaneously over a single WebSocket, so the avatar feels like a person — not a puppet. Viseme lip sync — spectral analysis of PCM audio estimates mouth shape in real-time. Front vowels A/E/I spread the mouth wide, back vowels O/U round it. Combined with RMS amplitude for jaw openness, attack/release envelope, noise floor gating, and a speaking smile baseline. The result: the avatar's mouth actually forms different shapes per sound, not just open/close. Blink — periodic natural eye blinks ~3.8s interval with random variation and occasional double-blinks. Blink rate adapts to mood: sleepy = more blinks, alert = fewer Body motion — slow torso sway on BodyAngleX/Y/Z, lower amplitude and longer period than head movement, so the avatar breathes even when silent Idle motion — head sway and eye darts when not speaking. Eye-dart frequency increases when the streamer's mood focus drops scattered attention 11 expression slots — happy, surprised, laughing, angry, sad, thinking, smug, eyeroll, confused, hype, deadpan Keyword-driven emotions — sentence content automatically triggers matching expressions 21 regex patterns, first match wins Expression auto-mapping — on connect, Wallie discovers VTS hotkeys and matches them to empty expression slots by name. No manual configuration needed for models with standard hotkey names Mood-reactive avatar — the MoodEngine's arousal/valence/focus feed directly into the avatar. Arousal scales idle and body sway amplitude 0.5×–1.6× . Valence shifts brow position and adds a resting smile when the streamer is in a good mood. Focus modulates eye-dart frequency Event cues — super chat → hype face, screen change → surprised look + upward glance, thinking pause → thinking expression, chat message → head turn toward "the chat" Everything is configured from the browser. No YAML files. No terminal commands after setup. ┌──────────────────────────────────────────────────────────────┐ │ ◤ WALLIE Profile: marlow ▾ ⊕ ⎘ 🗑 ● LIVE Stop ‹ │ ├──────────┬───────────────────────────────┬───────────────────┤ │ │ │ LIVE STATUS │ │ Identity │ Section editor │ ┌───────────────┐ │ │ Personal.│ │ │ Current topic │ │ │ Voice │ each section has its own │ │ Now saying... │ │ │ Avatar │ editor with live controls │ │ Open threads │ │ │ Topics │ │ │ Recent angles │ │ │ Vision │ Test ▶ monologue / chat / vis│ │ Session memory │ │ │ Chat │ Output preview + ▶ speak this│ │ Mood state │ │ │ Engine │ │ │ WebSocket log │ │ │ API Keys │ │ └───────────────┘ │ └──────────┴───────────────────────────────┴───────────────────┘ Three columns: navigation, section editor, live status. Test any configuration change instantly with the preview buttons before going live. No Python knowledge, no terminal, no setup steps. Download → double-click → done. and unzip it or Download the ZIP https://github.com/Alradyin/wallie-V2/archive/refs/heads/main.zip git clone https://github.com/Alradyin/wallie-V2.git . Double-click First run installs everything it needs, then the dashboard opens at start.bat . http://127.0.0.1:8765 . Paste your API key in the dashboard → pick a model → hit Start . That's the whole setup. Everything else is configured in the browser. macOS / Linux:run ./start.sh instead chmod +x start.sh once . | Path | LLM | TTS | Cost/hour | Quality | |---|---|---|---|---| Free | Gemini 2.5 Flash | Piper local | $0 | Good for testing | Cheap & fast | Groq Llama 3.3 70B | Fish Audio | ~$1.50 | Best balance | Premium | Claude Sonnet | ElevenLabs | ~$6.50 | Best quality + vision | All configured from the dashboard. Add your API keys → pick provider → pick model → Start. Piper free TTS setup Piper runs locally — no API key needed. One extra terminal command to download a voice: In your wallie directory, with venv active: pip install piper-tts onnxruntime python scripts/download piper voice.py en US-amy-medium Then in the dashboard: Voice → provider: piper , path: voices/en US-amy-medium.onnx . Vision setup screen reactions Requires a vision-capable LLM. In the dashboard: Engine → toggle "Vision capable" ON- Use a vision model: claude-sonnet-4-5 , gpt-4o , gemini-2.5-pro , or llama-4-scout on Groq Vision section → toggle ON, adjust frame interval and sensitivity The SKIP escape hatch is always active — no configuration needed. Chat platform setup Twitch: Add OAuth token from twitchtokengenerator.com https://twitchtokengenerator.com chat:read scope . Or leave empty for anonymous read-only. YouTube: Drop client secret.json in scripts/ . First run opens browser for Google OAuth consent. Kick: Just enter the channel slug. No auth needed public Pusher WebSocket . VTube Studio avatar - Run VTube Studio with API enabled Settings → API → Enable - In the dashboard: Avatar → toggle ON - First connect triggers a plugin approval popup in VTS — click Allow - Expression slots are auto-mapped from your model's hotkeys on connect. Override manually if needed - Adjust lipsync gain/ceiling for your voice. Viseme lip sync spectral mouth shape is enabled by default — disable it in Avatar config if your model doesn't have a ParamMouthForm parameter - Blink, body motion, and mood-reactive behaviour are enabled by default — tweak or disable per-feature in config Works with any Live2D model that has standard parameters MouthOpen , EyeOpenLeft/Right , FaceAngleX/Y/Z , BodyAngleX/Y/Z . OBS output routing Wallie outputs audio through your system's default audio device. To route it into OBS: Windows: Install VB-CABLE https://vb-audio.com/Cable/ . Set CABLE Input as default playback. In OBS: Audio Input Capture → CABLE Output. macOS: Use BlackHole https://existential.audio/blackhole/ the same way. Linux: PipeWire / PulseAudio loopback pw-loopback . Screen ────► Vision ───┐ mss + pHash │ ▼ Chat ─────────► Orchestrator ◄──── Persona + Topics + Mood YT/Twitch/Kick │ │ intent → system prompt + user message ▼ LLM streaming ← 5 providers │ │ token stream ▼ SentenceStreamer ← splits tokens into TTS-ready sentences │ ▼ TTS pipeline ← 3 providers, parallel pre-fire │ │ PCM16 audio ▼ AudioPlayer ──────► VTube Studio viseme lipsync + blink + body + expressions │ ▲ │ Mood Engine arousal/valence/focus │ ▼ speakers / OBS / virtual cable One pipeline. One conversation history. No competing buffers. This is the defining design choice. Early prototypes had parallel generation paths and went insane — the streamer would repeat itself, contradict itself, and lose all continuity. Everything goes through one orchestrator, one set of messages, one output path. Intent priority: highlight chat barge in → vision event → ordinary chat → monologue. Higher-priority intents preempt lower ones. Continuity machinery: - Rolling summary of older turns → injected into system prompt - Open-thread tracker → forces the next segment to resolve dangling questions and teases - Phrase cooldown → prevents catchphrase spam - Theme tracker → blocks repeated angles wallie-v2/ ├── wallie.py entrypoint ├── config.py pydantic models, profile management ├── core/ │ ├── orchestrator.py the single pipeline │ ├── persona.py prompt engineering │ ├── context.py conversation history + rolling summary │ ├── attention.py vision reaction decisions │ ├── mood.py slow-evolving emotional state │ └── memory store.py cross-session persistent memory ├── llm/ 5 LLM provider adapters ├── tts/ 3 TTS provider adapters ├── audio/ sounddevice player with alignment safety ├── vision/ screen capture + change detection + activity classification ├── chat/ YouTube, Twitch, Kick monitors ├── avatar/ VTube Studio WebSocket client ├── dashboard/ FastAPI + Alpine.js no build step ├── profiles/ saved persona profiles YAML └── scripts/ setup utilities | Provider | Streaming | Vision | Notes | |---|---|---|---| Groq | ✅ | ✅ Llama-4 | Fastest inference. Free tier. | OpenAI | ✅ | ✅ GPT-4o | Strong vision. Premium pricing. | OpenRouter | ✅ | ✅ varies | One key, many models. | Anthropic | ✅ | ✅ Claude 4 | Best character/IP recognition. | Gemini | ✅ | ✅ 2.5 family | Free tier 50 req/min . | Ollama | ✅ | ✅ llava, etc. | Fully local. No API key. | | Provider | Streaming | Cloning | Cost | |---|---|---|---| Fish Audio | ✅ | ✅ | ~$15/M chars | ElevenLabs | ✅ | ✅ | ~$30/M chars | Piper | ✅ local | ❌ | $0 | - API keys stored in .env with restricted permissions chmod 600 on POSIX - Dashboard never exposes raw keys — only masked previews sk-•••xyz - Dashboard binds to 127.0.0.1 only — not accessible from the network - Atomic writes prevent .env corruption - Provider error messages scrubbed of key-shaped strings before display - Allowed env variables are hard-coded — the UI cannot write arbitrary keys Do not expose the dashboard to a public network without a reverse proxy with authentication. Audio becomes static Hit the reset audio button in the dashboard top bar. If it recurs, check logs — usually a TTS provider returning non-PCM data detected and aborted automatically in most cases . AI describes generic UI "I see a YouTube page" The SKIP escape hatch depends on the model. Smaller models Llama-4 Scout, Gemini Flash are worse at following it. Upgrade to Claude Sonnet or GPT-4o for better vision, or set commentary density to sparse . AI keeps asking questions to chat The question throttle is automatic forces statements after one question-ending . If it persists, raise frequency penalty to 0.5–0.7 in Engine settings. Avatar mouth doesn't move Check that the MouthOpen parameter name matches your model some use ParamMouthOpen or MouthOpenY . Override in Avatar → Parameter mapping. Avatar mouth moves but shape looks wrong Viseme lip sync drives ParamMouthForm for mouth shape wide ↔ round . If your model uses a different parameter name, update it in Avatar → Parameters → Mouth form. If your model doesn't support mouth form at all, disable "Spectral mouth shape" in the Viseme section — lipsync will fall back to volume-only open/close . Avatar doesn't blink Check that your model has EyeOpenLeft / EyeOpenRight parameters some use ParamEyeLOpen / ParamEyeROpen . Override the parameter names in Avatar config. Blink can be disabled with enable blink: false . Expressions don't fire Expression slots need to match VTS hotkey names or IDs. If auto-mapping didn't find your hotkeys naming mismatch , map them manually in the dashboard or set expr happy , expr sad , etc. in your profile. Use the dashboard's "Discover hotkeys from VTS" to see available names. TTS returns 401 Verify the key in API Keys — the masked preview should match your provider dashboard. Hit "test" to confirm. What's coming next: Streaming avatar backend HeyGen — realistic-looking avatars as an alternative to Live2D First-run wizard — guided setup that walks you through provider choice → key entry → first stream Docker image — docker run wallie with a volume for config Cost meter — running spend tally in the live drawer OBS WebSocket integration — scene switching tied to stream events Voice cloning UI — upload reference audio, create a voice from the dashboard If any of these matter to you — open an issue, or better yet, a PR. Issues and PRs welcome. Ground rules: Code is English-only. Comments, logs, variables — all English. UI can be localized. Single pipeline, single history. Don't add parallel generation paths. That road has been walked. Lazy imports for optional deps. Groq users shouldn't need anthropic installed.- Run python scripts/ import check.py before submitting. MIT. See LICENSE /Alradyin/wallie-V2/blob/main/LICENSE . Wallie's own code is MIT. The Minecraft Play mode uses several third-party mods, which are installed and, for convenience, bundled under dist/mods/ and remain under their own licenses: | Mod | License | Source | |---|---|---| Baritone pathfinding/mining | LGPL-3.0 | | Meteor Client loads Baritone https://github.com/MeteorDevelopment/meteor-client https://github.com/MeteorDevelopment/meteor-client Fabric API https://github.com/FabricMC/fabric https://github.com/FabricMC/fabric Fabric Loader https://github.com/FabricMC/fabric-loader https://github.com/FabricMC/fabric-loader These mods are used unmodified and distributed as separate jars mere aggregation ; they are the property of their respective authors. Each is freely available at the source links above. Wallie is built for people who want their AI streamer to feel like a real show, not a tech demo. Star the repo. Try the $0 path. Tell us what broke.