From a desk toy to a hotel room: what it took to ship a local voice assistant to real buildings

A developer shipped v0.4.0 of Aice, a fully local voice assistant, moving it from a single desktop setup to a multi-service architecture for hotels, care homes and hospital wards. The release adds a per-room ESP32-S3 pod, a pod-gateway handling echo cancellation and Piper TTS, a backend running Whisper STT and a local LLM classifier, and a facilitator service, all running on-premises so voice never leaves the building. Security work included HttpOnly session cookies, CSRF tokens, bearer-token auth and a property certificate authority, while guest privacy is enforced by scoping long-term memory to individual stays with configurable retention and purge policies.

A voice assistant that runs on your own machine is a nice weekend story. You talk, Whisper transcribes, a local LLM works out what you meant, a skill answers, Piper speaks. No cloud, no account, nothing leaves the room. Then you try to put it in a hotel room and every assumption breaks. On your desk there's one user, one microphone, one trusted network, and one person who knows what the assistant can and can't do. A hotel has a hundred rooms, a guest who checks out on Tuesday and a stranger who checks in on Wednesday, a Wi-Fi network shared with every phone in the building, staff who need to see requests instead of hearing them, and nobody who will tell the assistant to "try again" when it goes quiet. Our last release, v0.4.0, is 29 commits and a few thousand lines of Rust and C++ that deal with that gap. This post walks through what changed and why, including the bugs we only found once real hardware sat on a real table. Before v0.4.0 the system was one backend and a desktop frontend. Now a property runs three services plus one small device per room: room pod ESP32-S3 pod-gateway aice-backend facilitator aice-hotels / -care / -ward mic + speaker + LEDs ──WSS──▶ speech detection ──WSS──▶ Whisper STT ──HTTPS──▶ staff desk, tickets, stays, 3 buttons echo cancellation LLM classifier device registry, alerts, Piper TTS, pacing skills, memory property MCP tools aice-hotels , aice-care and aice-ward . All of it runs on hardware in the building. The voice never leaves the property. The honest first entry in the changelog's Security section: before this release, the desk, the ticket API and the MCP endpoint were reachable without credentials, and a room's turn stream accepted whatever room the client said it was in. Fine on a laptop. Unacceptable on a hotel LAN. So the first commits are all about trust: HttpOnly SameSite=Strict session cookies, CSRF tokens on every form and API call, and a five-minute lockout after five bad passwords. Every ticket change goes to an append-only audit log that supervisors can read, and only a supervisor can reopen a closed ticket. service.token on first run, and the backend presents it as a bearer token. core-tls crate creates a certificate authority for the property, issues server certificates from it and builds rustls configs. On any non-loopback bind the facilitator defaults to TLS, serves its CA certificate for pods to pin, and refuses plain HTTP. The backend and bridge also refuse plaintext on a network bind unless you set allow plaintext lan explicitly. Verification is cached for 30 seconds. If the facilitator is down, a recent answer is kept for ten minutes so a desk restart doesn't take every room offline. Aice has long-term memory Memory Palace, our palace-rs crate : it can recall what you told it last week. In a home that's useful. In a hotel it's a privacy incident waiting to happen. The guest in room 214 tonight must not hear about the previous guest's allergies. The answer is the stay. Staff check guests and residents in and out on the desk or with stay open|close|list . Each stay gets its own memory wing, and a turn from that room reads and writes only that wing: wake-up context, recall, ingestion, and knowledge-graph facts, which are namespaced per wing. A few details that took more thought than the headline: memory retention is keep a returning guest can continue an old stay's memory , archive after N days, or wipe on close the default for wards . A backend job deletes due wings, including their facts, and confirms each purge back to the facilitator. One thing we learned the hard way while upgrading palace-rs : newer versions record wing names in several new tables a wing registry, usage events, mined-file records, feedback . A purge that only deleted the memories would have left the names of past stays behind. Purges now clear all of them, so a wiped stay leaves nothing. We started with an M5Stack ATOM Echo because it's tiny and cheap. It taught us something important: its microphone and speaker share one clock pin, so it cannot listen while it speaks. That makes it half-duplex. A guest can't interrupt it by voice, which for a voice assistant in a room is a real limitation. So we wrote down what a room pod actually has to do full-duplex audio, an echo reference, buttons, LEDs and moved to the Waveshare ESP32-S3-AUDIO-Board. It has an ES8311 codec driving the speaker and an ES7210 ADC for the mics, both on one full-duplex I2S port. The best part: one ES7210 channel records what the speaker is playing. That loopback is the echo reference that makes everything in the next section possible. The codec register drivers are vendored from Espressif's Apache-2.0 esp codec dev . Two bugs only appeared on the board: .local names. A fresh pod derives its id from its Wi-Fi MAC, generates a random nonce and keeps it in flash, and enrols with the facilitator over HTTPS pinned to the property CA. The LED ring pulses slow amber until a supervisor assigns a room on the desk. Then it connects to the room bridge over WSS with its new device token. If the server replies "unauthorized", the pod clears its token and enrols again. If it lost its token, it can get a new one by proving its nonce, and the old token stops working. Holding the button at boot forgets the identity completely. firmware keygen|publish|rollout signs images with a property ECDSA P-256 key and offers them to a stable percentage of the fleet. Pods download with their device token and install only an image whose signature verifies. In fairness: that flow is covered by tests, but at the time of the hardware bring-up we hadn't yet run a signed update end to end on the board. help button ticket for the room, and the bridge speaks a confirmation. No speech recognition or LLM is involved, so it still works when those are down. This was the hardest part of the release. If the assistant starts reading a long answer and the guest says "no, stop, I meant breakfast," it should stop. To do that, the bridge has to hear the guest while the pod's own speaker is playing a few centimetres from the microphone. Without echo cancellation, the loudest thing the mic hears is the assistant itself, and it interrupts itself constantly. A full-duplex pod now streams two signals during an answer: its microphone and its playback loopback. The bridge's echo canceller does two things. First, it finds the lag. The echo arrives later than the reference. On our pod the lag measured one 40 ms frame plus 0.8 ms, but whether that extra frame was there changed from session to session. So the canceller runs a correlation search over up to 128 ms every quarter second and keeps the lag only when the correlation is confident. Second, it learns the echo path with a 256-tap normalised LMS filter 16 ms at 16 kHz and subtracts the predicted echo from the mic. The core loop is short: js let predicted: f32 = self.weights.iter .zip window .map | w, x | w x .sum ; let error = heard - predicted; let energy: f32 = window.iter .map |x| x x .sum ; if self.adapt && energy ACTIVE REFERENCE ENERGY { let step = MU error.clamp -clip, clip / energy + EPSILON ; for w, x in self.weights.iter mut .zip window { w += step x; } } The interesting parts are around that loop: Gain staging mattered as much as the maths. The reference input runs at 18 dB so it never clips, the mic has no digital gain, and the speaker is at -20 dB. Measured on the Waveshare pod: 11–14 dB of echo removed, no self-interruption across 27 seconds of answers, and about 60% of normal-volume speech caught within a second. That isn't perfect, but it's the difference between "you can interrupt it" and "you can't." One last detail: a full-duplex pod still has a little audio buffered when the answer ends. That tail was being heard as a new request. The bridge now waits until the pod's speaker is actually quiet. Several fixes in this release share one theme: people don't talk in single, clean commands. wake word.cooldown secs after playback ends. The bridge reports playback started / playback finished , so the window runs from when the guest stopped hearing the answer, not from when the text was generated. We first shipped 2 seconds, which was far too short for a natural follow-up. The default is now 8. playback interrupted event with the spoken ratio cuts the remembered answer down to what they actually heard. Two classifier bugs were especially sneaky: In a voice interface, latency is the UX. A few changes made a large difference: stt.workers contexts. service.max concurrent turns , service.turn queue max wait ms gets a spoken "busy" reply instead of silence. ollama urls adds hosts next to the main one. Calls rotate across them, and a failing host is skipped for 30 seconds. room-loadtest enrols simulated pods on a facilitator, then drives them at real-time pace through the real bridge, backend and facilitator, and reports p50, p95 and max from end of speech to first answer audio. In CI it runs eight rooms against a backend with only two turn permits, and requires that every turn is answered, admission holds, and p95 stays under five seconds. If a change makes the system slower under load, CI goes red. CI also now compiles the pod firmware with PlatformIO, and releases ship Linux x86 64 and aarch64 archives for appliance installs next to the macOS build. A request that becomes a ticket nobody reads is worse than no assistant at all. Each pack now has alert tiers and per-tool acknowledgement windows: sla , audited and paged to the next tier, and this repeats on the last tier until someone acknowledges it. Defaults differ by pack: care homes escalate falls and distress after 60 seconds, wards escalate "get a nurse" after 120, hotels use 15 minutes for everything and the runbook tells you to tighten that for maintenance and complaints . Pods are watched too. The backend reports which pods have an open stream every 30 seconds. An active pod that goes silent past device offline after secs raises one escalated device offline ticket and shows OFFLINE on the desk, because a room that can't call for help is itself something staff need to know about. Once real people are talking to it, "it felt slow" is the most common bug report and the least useful one. So the last big change is end-to-end timing of the guest journey. First, an embarrassing one: every duration metric was exported as a Prometheus summary, so every dashboard panel built on histogram quantile over bucket series had been quietly empty. Every seconds metric is now a histogram with fixed buckets from 5 ms to 4 hours. Then the tracing. The bridge times backend wait, speech synthesis, and end-of-speech to first answer audio; the desk times ticket resolution; stage, dependency and LLM durations attach to the turn's trace without touching call sites. Any turn whose wait after speech passes service.slow turn threshold ms 3 seconds by default is logged once as slow turn , with: That last number turned out to be the most useful. When the slowest stage only explains half the wait, the problem is in the gaps: a queue, a lock, a reconnect. It's counted in voice turn unaccounted duration seconds , and a new "Guest Journey" Grafana dashboard puts it all on one screen. The classifier is now the part that matters most: every request goes through the LLM, and a misclassified request is a missed towel or a missed nurse call. Next we're working on scoring candidate classifier models against a labelled dataset, and running a second model in shadow next to the live one in production to record where they disagree before we switch anything. Aice is an open-source Apache-2.0 , local-first voice assistant written in Rust. Speech recognition, the LLM and memory all run on hardware you own. With v0.4.0 it runs in hotels, care homes and wards: guests ask out loud in their room, and staff and property systems get the request, with no cloud for the voice. Source: github.com/AncientiCe/aice https://github.com/AncientiCe/aice v0.4.0 release macOS arm64, Linux x86 64 and aarch64 : github.com/AncientiCe/aice/releases/tag/v0.4.0 https://github.com/AncientiCe/aice/releases/tag/v0.4.0 If you're running a property and want to pilot it, or you've solved echo cancellation on a microcontroller better than we have, I'd love to hear from you in the comments.