Wallie_main_demo.mp4 #
The open-source AI streamer that sees, hears & reacts — and actually feels alive.
Quick Start · Features · Dashboard · Architecture · Roadmap
⭐
If you think an AI that watchesTwo seconds, and it genuinely helps a solo dev keep shipping.and hearsyour screen should exist — give it a star.
Every AI streamer you've seen has the same problem.
It says two sentences. s. Says two more. s. Describes your YouTube homepage like a screen reader. Forgets what it said thirty seconds ago. Ends every thought with "what do you guys think?" into a chat that doesn't exist. It's not a show — it's a tech demo on loop.
Wallie is different. It's an AI that actually streams — develops thoughts across minutes, reacts to your screen like a person who's been using that computer all day, remembers what it covered an hour ago, takes opinions and sticks with them, drifts between topics the way real conversations drift, and shuts up when there's nothing worth saying.
You design the streamer. Wallie runs the show.
persona ──┐
topics ──┤
chat ──┤
vision ──┼──► Wallie ──► Voice (TTS) ──► OBS / virtual cable ──► your stream
hearing ──┤ ──► Live2D avatar (VTube Studio)
schedule ──┘
Pick the personality. Pick the voice. Pick the LLM. Pick the platform. Everything is swappable, everything is configurable from the browser, and the whole thing runs on your machine with your keys.
Wallie doesn't just watch your screen anymore — it picks up the game and plays it. Right now that's Minecraft: it gathers, crafts, builds, fights, and explores entirely on its own, narrating every moment in character. Smooth, human-looking camera. No human at the keyboard, no script — a real playthrough with a personality.
https://www.youtube.com/shorts/NEfsmppOXJk
And here's the part that makes it believable: the commentary is grounded in what Wallie is actually doing — its real inventory, health, and current goal — not guessed from a blurry frame. So it says
*"heading down to mine diamonds"*because it is, and brings up
*"that one time I dug straight into a lava pit"*because that's its run.
It actually plays— a planning brain sets the goal;Baritonehandles pathfinding & mining, and a custom Fabric mod handles crafting, combat, item pickup, and a buttery-smooth camera that looks human, not robotic.Grounded commentary— the streamer core stays in sync with the playing brain, so what itsaysmatches what itdoes.One-click setup— the dashboard installs Fabric + every mod for you (and backs up your current ones first).** Any persona, any model**— Play is just another mode. Vision, Hearing, and Play each toggle on/off independently.
Flip on
Playin the dashboard, pick a goal, and go — or jump straight in:
python scripts/run_wallie_live.py
Because the existing ones are toys.
They work for a 2-minute demo and fall apart on a real stream. Here's what actually goes wrong after 10 minutes:
| The problem | What usually happens | What Wallie does |
|---|---|---|
| Repetition | ||
| Says "that's interesting" every 30 seconds | Dedupe engine + phrase cooldown + rolling summary that tracks everything already said | |
| Short memory | ||
| Forgets the topic from 5 minutes ago | Rolling summarizer compresses old turns into bullet notes, injected into every prompt | |
| Robotic vision | ||
| "I can see a YouTube homepage with several videos" | First-person ownership + SKIP escape hatch — narrates nothing, reacts to what matters | |
| Question loops | ||
| Every segment ends with "what do you think, chat?" | Question detector throttles after one; next prompt forces a statement ending | |
| No personality | ||
| Generic helpful assistant voice | Full persona system: energy, humor style, catchphrases, backstory, opinions, taboo topics | |
| Choppy pacing | ||
| 2 sentences → long → 2 sentences | Pipeline overlap + longer thought development eliminates dead air | |
| Broken threads | ||
| Sets up a story, never delivers it | Open-thread tracker forces the next segment to pay off teases and answer its own questions | |
| Topic whiplash | ||
| Jumps from AI to cooking to space with no bridge | Association-based topic drift with configurable style (rigid / natural / freeform) |
These aren't edge cases. They're the default behavior of most AI streamer projects. Wallie was built by hitting every one of these problems and refusing to ship until they were fixed.
Six LLM providers. Three TTS engines. Three chat platforms. Mix and match per profile.
| LLM | TTS | Chat | Avatar |
|---|---|---|---|
| OpenAI | Fish Audio | Twitch | VTube Studio (Live2D) |
| Anthropic (Claude) | ElevenLabs | YouTube | — |
| Google (Gemini) | Piper (local, free) | Kick | — |
| Groq | — | — | — |
| OpenRouter | — | — | — |
| Ollama (local, free) | — | — | — |
Swap providers without changing code. Run a fully offline stream with Ollama + Piper, or go premium with Claude + ElevenLabs.
Not just a name and a system prompt. You design a character:
Identity— name, handle, pronouns, age range, origin, archetype, backstory** Voice**— energy level (chill → unhinged), humor style (pick multiple: ironic, deadpan, absurd, observational, roast...), profanity level, formalityFlavor— catchphrases, running gags, banned words, favorite topics, taboo topics** Opinions**— strong opinions toggle, admit uncertainty, break the fourth wall** Extra notes**— free text for anything the structured fields miss
Save multiple personas. Switch between them with a dropdown.
The single biggest technical challenge. Most AI chatbots break down after 20 minutes because the context window fills up.
Wallie's approach:
Rolling summarizer— every ~14 segments, a background LLM call compresses older history into tight bullet notes** Session notes**— that compressed memory is injected into every system prompt, so the streamer knows what it already said** Cross-session memory**— key facts and viewer interactions persist across streams** Dedupe engine**— paraphrase-aware similarity check (bigram + trigram Jaccard) catches the model repeating itself in different words
Real streamers don't talk at a constant rate. They get excited, they to think, they go on tangents, they settle into a flow.
Mood engine— arousal, valence, focus, and talkativity drift slowly over the stream. High arousal = faster, warmer output. Low talkativity = silence beats. Mood state feeds into the avatar for reactive animation.Attention engine— vision events are filtered through a probabilistic decision layer. Not every screen change gets a deep reaction. Some get a glance. Some get ignored. Some trigger a personal tangent.Silence beats— the streamer holds natural s when the mood says to. Dead air handled by design, not by accident.** Pipeline overlap**— next segment starts generating while current audio is still playing. No awkward 3-second gaps between thoughts.
wallie_minecraft.mp4 #
Screen reactions are the hardest part to get right. Wallie's approach:
First-person ownership— Wallie owns whatever is on screen. Gaming: "I just got bodied by that boss", not "the character is fighting a boss". Browsing: "let me pull this up", not "the user is browsing". Never third-person, never narration.SKIP escape hatch— if there's nothing specific to name, the model outputsSKIP
and stays quiet. No more narrating generic UIs.Activity adaptation— detects scrolling, typing, app-switching, video playback and adjusts reactions accordingly. Typing → ignore. App switch → react. Rapid browsing → wait until user settles. Each activity type gets context-aware prompting.Attention engine— not every screen change gets the same treatment. DEEP reactions (22%), quick GLANCEs (28%), personal TANGENTs (5%), deliberate IGNOREs (27%), and SILENCE beats (18%). Streak fatigue prevents reacting the same way twice in a row.Scene memory— remembers what it last said about the current screen. Dedupe threshold at 0.65 with per-sentence comparison catches paraphrased repetition.
Wallie captures your system audio (WASAPI loopback) and reacts to it live, fused with vision into a single reaction — sight and sound, one voice. Just like you organize your senses, not two narrators talking over each other.
Speech & lyrics— local speech-to-text (faster-whisper) transcribes videos, voice chat, and song lyrics. Auto-uses your GPU if available, clean CPU fallback if not.Real music understanding— pure-numpy DSP, no extra ML weight. It reads** major/minor key → mood**, tempo + beat strength, instrumentation texture (bass-heavy / acoustic / airy / lo-fi), and production quality (crisp / muddy / harsh). A sad song reads as melancholy; a beat drop reads as energy. It'll call a track a banger or roast a muddy mix.Multimodal fusion— what it hears is fused with what it sees into one coherent reaction, never two competing ones.** No self-echo**— content-based guard so Wallie never reacts to its own TTS bleeding back through the loopback.** Pure-hearing mode**— run it with vision off and it reacts to audio the way it reacts to a screen: stays quiet, listens, then reacts to what's playing.
Enable it in the dashboard's Hearing section. Needs two extra deps: pip install soundcard faster-whisper
.
Play mode hands Wallie the controls. First game: Minecraft, survival, live and unscripted.
Reliable autonomy— a planning brain picks high-level goals; a deterministic skill library (crafting, smelting, full tool/armour sets, mining to ore depths, hunting for food & wool, beds, combat with weapon selection, dropped-item pickup) does the actual work — so even a small/free model plays competently instead of flailing.Human-like camera— a custom Fabric mod eases every turn at full FPS. No snapping, no teleport-cam — it reads like a person playing.** Grounded commentary**— reactions come from the agent's real game state (inventory, health, threats, current goal), fused through the same single pipeline as vision and hearing. No screen-guessing, no hallucinated "desert temples".One-click install— Fabric + all mods straight from the dashboard's Play section. Vision/Hearing/Play toggle independently, on any persona.
Not just a mouth that opens and closes. Six animation layers run simultaneously over a single WebSocket, so the avatar feels like a person — not a puppet.
Viseme lip sync— spectral analysis of PCM audio estimates mouth shape in real-time. Front vowels (A/E/I) spread the mouth wide, back vowels (O/U) round it. Combined with RMS amplitude for jaw openness, attack/release envelope, noise floor gating, and a speaking smile baseline. The result: the avatar's mouth actually forms different shapes per sound, not just open/close.Blink— periodic natural eye blinks (~3.8s interval) with random variation and occasional double-blinks. Blink rate adapts to mood: sleepy = more blinks, alert = fewerBody motion— slow torso sway on BodyAngleX/Y/Z, lower amplitude and longer period than head movement, so the avatar breathes even when silentIdle motion— head sway and eye darts when not speaking. Eye-dart frequency increases when the streamer's mood focus drops (scattered attention)11 expression slots— happy, surprised, laughing, angry, sad, thinking, smug, eyeroll, confused, hype, deadpan** Keyword-driven emotions**— sentence content automatically triggers matching expressions (21 regex patterns, first match wins)** Expression auto-mapping**— on connect, Wallie discovers VTS hotkeys and matches them to empty expression slots by name. No manual configuration needed for models with standard hotkey namesMood-reactive avatar— the MoodEngine's arousal/valence/focus feed directly into the avatar. Arousal scales idle and body sway amplitude (0.5×–1.6×). Valence shifts brow position and adds a resting smile when the streamer is in a good mood. Focus modulates eye-dart frequencyEvent cues— super chat → hype face, screen change → surprised look + upward glance, thinking → thinking expression, chat message → head turn toward "the chat"
Everything is configured from the browser. No YAML files. No terminal commands after setup.
┌──────────────────────────────────────────────────────────────┐
│ ◤ WALLIE Profile: marlow ▾ ⊕ ⎘ 🗑 ● LIVE Stop ‹ │
├──────────┬───────────────────────────────┬───────────────────┤
│ │ │ LIVE STATUS │
│ Identity │ Section editor │ ┌───────────────┐ │
│ Personal.│ │ │ Current topic │ │
│ Voice │ (each section has its own │ │ Now saying... │ │
│ Avatar │ editor with live controls) │ │ Open threads │ │
│ Topics │ │ │ Recent angles │ │
│ Vision │ Test ▶ monologue / chat / vis│ │ Session memory │ │
│ Chat │ Output preview + ▶ speak this│ │ Mood state │ │
│ Engine │ │ │ WebSocket log │ │
│ API Keys │ │ └───────────────┘ │
└──────────┴───────────────────────────────┴───────────────────┘
Three columns: navigation, section editor, live status. Test any configuration change instantly with the preview buttons before going live.
No Python knowledge, no terminal, no setup steps. Download → double-click → done.
and unzip it (orDownload the ZIPgit clone https://github.com/Alradyin/wallie-V2.git
).Double-click First run installs everything it needs, then the dashboard opens atstart.bat
.http://127.0.0.1:8765
.Paste your API key in the dashboard → pick a model → hitStart.
That's the whole setup. Everything else is configured in the browser.
macOS / Linux:run./start.sh
instead (chmod +x start.sh
once).
| Path | LLM | TTS | Cost/hour | Quality |
|---|---|---|---|---|
| Free | ||||
| Gemini 2.5 Flash | Piper (local) | $0 | Good for testing | |
| Cheap & fast | ||||
| Groq (Llama 3.3 70B) | Fish Audio | ~$1.50 | Best balance | |
| Premium | ||||
| Claude Sonnet | ElevenLabs | ~$6.50 | Best quality + vision |
All configured from the dashboard. Add your API keys → pick provider → pick model → Start.
Piper (free TTS) setup
Piper runs locally — no API key needed. One extra terminal command to download a voice:
pip install piper-tts onnxruntime
python scripts/download_piper_voice.py en_US-amy-medium
Then in the dashboard: Voice → provider: piper
, path: voices/en_US-amy-medium.onnx
.
Vision setup (screen reactions)
Requires a vision-capable LLM. In the dashboard:
Engine→ toggle "Vision capable" ON- Use a vision model:
claude-sonnet-4-5
,gpt-4o
,gemini-2.5-pro
, orllama-4-scout
on Groq Vision section → toggle ON, adjust frame interval and sensitivity
The SKIP escape hatch is always active — no configuration needed.
Chat platform setup
Twitch: Add OAuth token from twitchtokengenerator.com (chat:read scope). Or leave empty for anonymous read-only.
YouTube: Drop client_secret.json
in scripts/
. First run opens browser for Google OAuth consent.
Kick: Just enter the channel slug. No auth needed (public Pusher WebSocket).
VTube Studio avatar
- Run VTube Studio with API enabled (Settings → API → Enable)
- In the dashboard: Avatar → toggle ON
- First connect triggers a plugin approval popup in VTS — click Allow
- Expression slots are
auto-mapped from your model's hotkeys on connect. Override manually if needed - Adjust lipsync gain/ceiling for your voice. Viseme lip sync (spectral mouth shape) is enabled by default — disable it in Avatar config if your model doesn't have a
ParamMouthForm
parameter - Blink, body motion, and mood-reactive behaviour are enabled by default — tweak or disable per-feature in config
Works with any Live2D model that has standard parameters (MouthOpen
, EyeOpenLeft/Right
, FaceAngleX/Y/Z
, BodyAngleX/Y/Z
).
OBS output routing
Wallie outputs audio through your system's default audio device. To route it into OBS:
Windows: Install VB-CABLE. Set CABLE Input as default playback. In OBS: Audio Input Capture → CABLE Output.
macOS: Use BlackHole the same way.
Linux: PipeWire / PulseAudio loopback (pw-loopback
).
Screen ────► Vision ───┐
(mss + pHash) │
▼
Chat ─────────► Orchestrator ◄──── Persona + Topics + Mood
(YT/Twitch/Kick) │
│ intent → system prompt + user message
▼
LLM (streaming) ← 5 providers
│
│ token stream
▼
SentenceStreamer ← splits tokens into TTS-ready sentences
│
▼
TTS pipeline ← 3 providers, parallel pre-fire
│
│ PCM16 audio
▼
AudioPlayer ──────► VTube Studio (viseme lipsync + blink + body + expressions)
│ ▲
│ Mood Engine (arousal/valence/focus)
│
▼
speakers / OBS / virtual cable
One pipeline. One conversation history. No competing buffers. This is the defining design choice. Early prototypes had parallel generation paths and went insane — the streamer would repeat itself, contradict itself, and lose all continuity. Everything goes through one orchestrator, one set of messages, one output path.
Intent priority: highlight chat (barge in) → vision event → ordinary chat → monologue. Higher-priority intents preempt lower ones.
Continuity machinery:
- Rolling summary of older turns → injected into system prompt
- Open-thread tracker → forces the next segment to resolve dangling questions and teases
- Phrase cooldown → prevents catchphrase spam
- Theme tracker → blocks repeated angles
wallie-v2/
├── wallie.py # entrypoint
├── config.py # pydantic models, profile management
├── core/
│ ├── orchestrator.py # the single pipeline
│ ├── persona.py # prompt engineering
│ ├── context.py # conversation history + rolling summary
│ ├── attention.py # vision reaction decisions
│ ├── mood.py # slow-evolving emotional state
│ └── memory_store.py # cross-session persistent memory
├── llm/ # 5 LLM provider adapters
├── tts/ # 3 TTS provider adapters
├── audio/ # sounddevice player with alignment safety
├── vision/ # screen capture + change detection + activity classification
├── chat/ # YouTube, Twitch, Kick monitors
├── avatar/ # VTube Studio WebSocket client
├── dashboard/ # FastAPI + Alpine.js (no build step)
├── profiles/ # saved persona profiles (YAML)
└── scripts/ # setup utilities
| Provider | Streaming | Vision | Notes |
|---|---|---|---|
| Groq | |||
| ✅ | ✅ (Llama-4) | Fastest inference. Free tier. | |
| OpenAI | |||
| ✅ | ✅ (GPT-4o) | Strong vision. Premium pricing. | |
| OpenRouter | |||
| ✅ | ✅ (varies) | One key, many models. | |
| Anthropic | |||
| ✅ | ✅ (Claude 4) | Best character/IP recognition. | |
| Gemini | |||
| ✅ | ✅ (2.5 family) | Free tier (50 req/min). | |
| Ollama | |||
| ✅ | ✅ (llava, etc.) | Fully local. No API key. |
| Provider | Streaming | Cloning | Cost |
|---|---|---|---|
| Fish Audio | |||
| ✅ | ✅ | ~$15/M chars | |
| ElevenLabs | |||
| ✅ | ✅ | ~$30/M chars | |
| Piper | |||
| ✅ (local) | ❌ | $0 |
- API keys stored in
.env
with restricted permissions (chmod 600
on POSIX) - Dashboard never exposes raw keys — only masked previews (
sk-•••xyz
) - Dashboard binds to
127.0.0.1
only — not accessible from the network - Atomic writes prevent
.env
corruption - Provider error messages scrubbed of key-shaped strings before display
- Allowed env variables are hard-coded — the UI cannot write arbitrary keys
Do not expose the dashboard to a public network without a reverse proxy with authentication.
Audio becomes static
Hit the reset audio button in the dashboard top bar. If it recurs, check logs — usually a TTS provider returning non-PCM data (detected and aborted automatically in most cases).
AI describes generic UI ("I see a YouTube page")
The SKIP escape hatch depends on the model. Smaller models (Llama-4 Scout, Gemini Flash) are worse at following it. Upgrade to Claude Sonnet or GPT-4o for better vision, or set commentary density to sparse
.
AI keeps asking questions to chat
The question throttle is automatic (forces statements after one question-ending). If it persists, raise frequency penalty to 0.5–0.7 in Engine settings.
Avatar mouth doesn't move
Check that the MouthOpen
parameter name matches your model (some use ParamMouthOpen
or MouthOpenY
). Override in Avatar → Parameter mapping.
Avatar mouth moves but shape looks wrong
Viseme lip sync drives ParamMouthForm
for mouth shape (wide ↔ round). If your model uses a different parameter name, update it in Avatar → Parameters → Mouth form. If your model doesn't support mouth form at all, disable "Spectral mouth shape" in the Viseme section — lipsync will fall back to volume-only (open/close).
Avatar doesn't blink
Check that your model has EyeOpenLeft
/ EyeOpenRight
parameters (some use ParamEyeLOpen
/ ParamEyeROpen
). Override the parameter names in Avatar config. Blink can be disabled with enable_blink: false
.
Expressions don't fire
Expression slots need to match VTS hotkey names or IDs. If auto-mapping didn't find your hotkeys (naming mismatch), map them manually in the dashboard or set expr_happy
, expr_sad
, etc. in your profile. Use the dashboard's "Discover hotkeys from VTS" to see available names.
TTS returns 401
Verify the key in API Keys — the masked preview should match your provider dashboard. Hit "test" to confirm.
What's coming next:
Streaming avatar backend (HeyGen)— realistic-looking avatars as an alternative to Live2D** First-run wizard**— guided setup that walks you through provider choice → key entry → first stream** Docker image**—docker run wallie
with a volume for configCost meter— running spend tally in the live drawer** OBS WebSocket integration**— scene switching tied to stream events** Voice cloning UI**— upload reference audio, create a voice from the dashboard
If any of these matter to you — open an issue, or better yet, a PR.
Issues and PRs welcome. Ground rules:
Code is English-only. Comments, logs, variables — all English. UI can be localized.Single pipeline, single history. Don't add parallel generation paths. That road has been walked.Lazy imports for optional deps. Groq users shouldn't needanthropic
installed.- Run
python scripts/_import_check.py
before submitting.
MIT. See LICENSE.
Wallie's own code is MIT. The Minecraft Play mode uses several third-party mods, which are
installed (and, for convenience, bundled under dist/mods/
) and remain under their own licenses:
| Mod | License | Source |
|---|---|---|
| Baritone (pathfinding/mining) | ||
| LGPL-3.0 | ||
Meteor Client(loads Baritone)https://github.com/MeteorDevelopment/meteor-client** Fabric API**https://github.com/FabricMC/fabric** Fabric **https://github.com/FabricMC/fabric-These mods are used unmodified and distributed as separate jars (mere aggregation); they are the property of their respective authors. Each is freely available at the source links above.
Wallie is built for people who want their AI streamer to feel like a real show, not a tech demo.
Star the repo. Try the $0 path. Tell us what broke.