{"slug": "mana-2-3-seconds-to-feeling-human", "title": "Mana: 2-3 Seconds to Feeling Human", "summary": "A developer has built Mana, a voice AI assistant that runs entirely on a local machine with an 8GB VRAM GPU, achieving 2-3 second response times by using a single 4B model for reasoning, code generation, and explanation in one pass. The system, inspired by fictional AIs like Alice from Sword Art Online and JARVIS, has been running 24/7 for three months without crashes.", "body_md": "so I shipped a voice AI assistant that runs entirely on my machine. no cloud, no APIs, no latency nightmares.\n\nthe original idea came from Alice in Sword Art Online — an AI that feels like an actual person, not a chatbot. mixed with JARVIS's anticipation and Neuro-sama's quirky personality. here's what actually went into getting from \"wouldn't it be cool\" to \"this runs 24/7 without issues.\"\n\nmost voice assistants are cloud-first: you speak → sent to server → processed → response → back to you. each hop adds latency. you're looking at 3-6 seconds before you hear anything. for a voice interaction, that's dead. it kills the feeling of talking to something intelligent.\n\nI wanted something faster. something that *responds*.\n\nthe constraint: do it locally. use an 8GB VRAM GPU, run everything on-device, no external APIs except for the live2d avatar bits (because that's hard to render locally and still look good).\n\nhere's the reality: I have a GPU with 8GB VRAM. no budget to experiment with better cards or more models. so every architecture decision was forced by what actually fits.\n\nnaive approach: chain multiple specialized models.\n\n```\nUser speaks\n  → Transcription model (Whisper)\n  → Planning model (3B: what should I do?)\n  → Coding model (7B: generate implementation)\n  → Verification model (4B: is this correct?)\n  → TTS (speak the answer)\n```\n\nmath: 1s + 2s + 3s + 1.5s = 7.5s of latency before the user hears anything. nope.\n\nthe problem isn't just that each model is slow. it's *model loading overhead*. every time you swap from one model to another, you:\n\nwith only 8GB, this gets gnarly fast.\n\nthe constraint was hardware. 8GB VRAM. no more, no less. that forced clarity: pick one model that does everything, or pick nothing.\n\nso I went with a single model (4B by default, with 7B/8B quality modes available) that does reasoning + code generation + explanation in *one pass*.\n\nlatency: ~2-3 seconds total. actually conversational.\n\nthis is the difference between a chatbot and a companion. JARVIS doesn't pause for 6 seconds before responding. neither does Mana. turns out, when you're forced to optimize for latency (because you only have 8GB to work with), you accidentally build something that feels human.\n\nthe tradeoff: a 4B model is weaker than larger models, but fits in 8GB VRAM and keeps latency down. for voice queries, that accuracy loss is negligible. I can bump to 7B or 8B for quality mode when latency isn't critical.\n\n**context preservation** — the LLM reasons internally (\"user wants me to find X in their data\"), then codes, then explains. no information loss at model boundaries.\n\n**VRAM efficiency** — load the 4B model once (~2-3GB in INT8). keep it there. reuse it for every query. upgrade to 7B/8B only when you want quality over speed.\n\n**simple output format** — use XML tags to split the LLM response:\n\n```\n   <reasoning>what I understood</reasoning>\n   <code>implementation</code>\n   <explanation>what to say via TTS</explanation>\n```\n\nthen execute code silently, speak only the explanation.\n\n```\n[Voice input]\n  ↓ Whisper (local transcription)\n[Text]\n  ↓ Qwen 4B (reasoning + code + explanation)\n[Structured output]\n  ├─ Code (execute silently, log results)\n  └─ Explanation (TTS via Kokoro/Chatterbox/Fish Speech)\n  ↓\n[Audio output to user]\n```\n\ntotal latency from \"hey Mana\" to hearing the response: ~2-3 seconds. feels like talking to something intelligent.\n\nwith an 8GB GPU, the budget is tight:\n\ntotal: actually fits (barely). I quantize aggressively, drop smaller models, and flush unused ones. the tradeoff is worth it — every millisecond of startup or response latency costs the feeling of talking to something alive.\n\nit's been running 24/7 for about 3 months now. no crashes. no \"processing never finished\" hangs. it just works.\n\n**earlier profiling** — I spent weeks optimizing things that didn't matter (TTS buffer sizes), then found the real bottleneck (model loading) in one afternoon with a profiler.\n\n**accept quantization earlier** — INT8 quantization costs ~5-10% accuracy on reasoning. for voice queries, that's negligible. I fought it for weeks.\n\n**screen context is expensive** — OCR-ing the screen every query tanks latency. I ended up making it opt-in (\"hey Mana, look at my screen\").\n\ncurrently working on:\n\nthe long-term goal (as hardware allows):\n\nthat's it. local-first voice AI that ships on consumer hardware. the constraint (8GB VRAM) forced good decisions: use one model, quantize aggressively, separate concerns (code vs. explanation), measure everything.\n\nthe rest is just execution.\n\nat the core, the goal was Alice. an AI that feels like talking to a real person. turns out, the technical constraints *create* that feeling. instant response. screen awareness. a quirky personality through code decisions. JARVIS's anticipation through context preservation. Neuro-sama's realness through being unfiltered and responsive.\n\nyou can't fake that with prompts. you have to build it into the architecture.\n\nwant to try it? it's open source: [github.com/Yuuzulight/Mana](https://github.com/Yuuzulight/Mana)", "url": "https://wpnews.pro/news/mana-2-3-seconds-to-feeling-human", "canonical_source": "https://dev.to/yuuzulight/mana-2-3-seconds-to-feeling-human-3e0b", "published_at": "2026-08-04 23:42:09+00:00", "updated_at": "2026-08-05 00:44:21.035846+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure", "developer-tools"], "entities": ["Mana", "Whisper", "Qwen", "Kokoro", "Chatterbox", "Fish Speech", "Alice", "JARVIS"], "alternates": {"html": "https://wpnews.pro/news/mana-2-3-seconds-to-feeling-human", "markdown": "https://wpnews.pro/news/mana-2-3-seconds-to-feeling-human.md", "text": "https://wpnews.pro/news/mana-2-3-seconds-to-feeling-human.txt", "jsonld": "https://wpnews.pro/news/mana-2-3-seconds-to-feeling-human.jsonld"}}