{"slug": "rebuilding-my-local-voice-assistant-pipeline", "title": "Rebuilding my local voice assistant pipeline", "summary": "Platypush developer Fabio Manganiello replaced the assistant.openwakeword plugin with a new assistant.microwakeword plugin built on microWakeWord via the pymicro-wakeword Python bindings, after months of false positives and false negatives from openWakeWord's v0.1 models from 2023. The new plugin ships the okay_nabu, hey_jarvis, hey_mycroft and alexa models in-package, keeps the same API (models, detection_threshold, frame_duration, patience_frames, input_device, input_volume, list_models and HotwordDetectedEvent), and defaults patience_frames to 0 because the models already smooth output over a sliding window. Switching requires only pointing the configuration at assistant.microwakeword, and custom models can be added by dropping a microWakeWord JSON manifest and .tflite file into the plugin's models directory.", "body_md": "# \n              \n                Rebuilding my voice assistant pipeline\n                [https://blog.platypush.tech/article/Rewriting-my-voice-assistant-pipeline](https://blog.platypush.tech/article/Rewriting-my-voice-assistant-pipeline)\n\n### Swapping openwakeword for microwakeword and making voice detection smarter\n\nI [wrote an article](https://blog.platypush.tech/article/Local-voice-assistant) a couple of months ago describing how I rebuilt my old voice assistant pipeline in [Platypush](https://platypush.tech).\n\nAnd I wrote earlier articles describing my journey through self-hosted voice assistants - [in 2024](https://blog.platypush.tech/article/The-state-of-voice-assistant-integrations-in-2024) (with [Picovoice](https://picovoice.ai)), [in 2020](https://blog.platypush.tech/article/Build-custom-voice-assistants) (diving into the legacy [`assistant.google`](https://docs.platypush.tech/platypush/plugins/assistant.google.html) integration, the now defunct `assistant.alexa` integration, the now defunct Snowboy integration, the now defunct DeepSpeech integration and Picovoice) and the [original article in 2019](https://blog.platypush.tech/article/Build-your-customizable-voice-assistant-with-Platypush) describing the Google Assistant integration.\n\nFor a couple of months I used the voice pipeline described in my latest article as my main driver, but after getting used to the robustness of commercial voice assistants I was unhappy with the high rates of false positives/negatives of my self-hosted pipeline, as well as its lack of robustness in detecting start/end of speech, especially in noisy environments and using cheap microphones.\n\nAfter a bit of experimentation, I decided to swap and fine tune a few parts, and after a couple of days of trying the results are satisfying enough for me to write a short article about it.\n\nThe resulting pipeline is:\n\n## Wake word detection: a new `assistant.microwakeword` plugin\n\nUntil recently I used [`assistant.openwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.openwakeword.html), which wraps [openWakeWord](https://github.com/dscripka/openWakeWord).\n\nIt works, but the project has been quiet for a while and all of its pre-trained models are `v0.1` files from 2023, mostly trained on synthetic speech.\n\nIn practice I was getting both false positives (the hotword triggering on background noise) and false negatives (the hotword not detected unless I was close to the microphone).\n\nI added a new plugin, [`assistant.microwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.microwakeword.html), on top of [microWakeWord](https://github.com/kahrendt/microWakeWord) through the [`pymicro-wakeword`](https://pypi.org/project/pymicro-wakeword/) Python bindings.\n\n`microWakeWord` is the engine used by ESPHome and Home Assistant Voice, its models are actively maintained, and the built-in models (`okay_nabu`, `hey_jarvis`, `hey_mycroft`, `alexa`) ship with the package — no download required.\n\nThe current official models are provided under the [ESPHome micro-wake-word-models repository](https://github.com/esphome/micro-wake-word-models).\n\nCustom models are supported too: drop a `microWakeWord` JSON manifest (with its `.tflite` model) in the plugin's models directory and reference it by name.\n\nThe plugin exposes the same API as `assistant.openwakeword` (same `models`, `detection_threshold`, `frame_duration`, `patience_frames`, `input_device`, `input_volume` parameters, same `list_models` action, same `HotwordDetectedEvent`), so switching is a matter of just changing the configuration to point to the new plugin:\n\n```\nassistant.microwakeword:\n  models:\n    - alexa\n```\n\nA few design notes:\n\n- Without an explicit `detection_threshold` , each model uses the`probability_cutoff` from its own manifest (0.9 for`alexa` ), which is the value the model was validated against.\n- Models already smooth their output over a sliding window, so unlike openwakeword, `patience_frames` defaults to 0 here.\n- The audio thread never touches the TFLite models directly: it queues frames (and a reset marker) to the main thread, because `MicroWakeWord.reset()` reloads the interpreter and isn't safe to call concurrently.\n- Like `assistant.openwakeword` , the plugin releases the audio device when another assistant plugin starts a conversation, and re-acquires it when the conversation ends.\n\n## Speech-to-text: `assistant.openai`, with a much better ear\n\n`assistant.openai` records audio when a conversation starts and transcribes it through Whisper.\n\nThe tricky part has always been *end-of-speech detection*: knowing when the user has finished talking. The previous implementation used a per-frame boolean from WebRTC VAD, and any single frame classified as speech would reset the silence timer. With even a bit of background noise, phrases would only end when the maximum conversation duration was hit.\n\nThe audio preprocessing layer (`AudioPreprocessor`, shared by the STT integrations) has been reworked:\n\n- **[Silero VAD](https://github.com/snakers4/silero-vad) as the primary backend** (through[`onnxruntime`](https://onnxruntime.ai) , no[PyTorch](https://pytorch.org) required), with WebRTC VAD and a plain RMS-based detector as fallbacks. To get an idea of Silero VAD's performance compared against other solutions for speech activity detection, take a look at the insightful article[*One voice detector to rule them all*](https://thegradient.pub/one-voice-detector-to-rule-them-all/) . The Silero model is downloaded on first use into the Platypush workdir and checksum-verified:\n\n``` php\nassistant.openai:\n  # auto = silero -> webrtc -> energy, depending on what's installed\n  vad_backend: auto\n```\n\n- \n**Hysteresis + sliding window** : a frame becomes speech above the VAD threshold and stays speech until its score drops by a margin, and the per-frame decisions are smoothed by majority vote over a 0.5s window. A single noise burst can no longer keep a conversation alive.\n- \n**Adaptive noise floor** : the background noise level is tracked over time, and frames that aren't louder than`noise_floor_factor` (default 2.0, ~6 dB) times the floor are never considered as speech.\n\nOn a synthetic benchmark (Piper-generated phrases mixed with babble and burst noise), the old settings kept conversations open for 6–10 seconds after the speech had ended; the new pipeline ends them 1.0–1.6 seconds after the last word, which is the configured [`conversation_end_timeout`](https://docs.platypush.tech/platypush/plugins/assistant.openai.html#platypush.plugins.assistant.openai.AssistantOpenaiPlugin.start_conversation) plus the smoothing lag.\n\nSince the changes on Platypush have been performed directly in the [`AudioPreprocessor`](https://git.platypush.tech/platypush/platypush/src/branch/master/platypush/plugins/assistant/_audio.py) also used by other native speech-to-text engines, other speech-to-text plugins also benefit from them - e.g. if you use [`assistant.vosk`](https://docs.platypush.tech/platypush/plugins/assistant.vosk.html) for full on-device detection.\n\n## Responses and context: the `openai` plugin\n\nThe transcribed phrase is processed by the `openai` plugin, which handles both the API key and the conversation context. By default the assistant renders the response through the configured `tts_plugin`; if the response ends with a question mark, the conversation continues with a follow-up.\n\nCustom hooks can intercept phrases before the default response kicks in — for example, to handle home automation commands locally without an LLM round-trip:\n\n``` python\nfrom platypush import run, when\nfrom platypush.message.event.assistant import SpeechRecognizedEvent\n\n@when(SpeechRecognizedEvent, phrase='play (the)? music')\ndef play_music():\n    run('music.mpd.play')\n```\n\nAnd the hotword event is wired to the conversation start with a similar hook:\n\n``` python\nfrom platypush import run, when\nfrom platypush.message.event.assistant import HotwordDetectedEvent\n\n@when(HotwordDetectedEvent)\ndef on_hotword(**_):\n    run('assistant.openai.start_conversation')\n```\n\n## Text-to-speech: `tts.openai`\n\nResponses are rendered by [`tts.openai`](https://docs.platypush.tech/platypush/plugins/tts.openai.html), with volume and voice configurable. In my tests however [`tts.piper`](https://docs.platypush.tech/platypush/plugins/tts.piper.html) also yielded good results, and it runs entirely locally. In my tests on a RaspberryPi 5 however it added ~3-4s of latency to process the response to audio compared to `tts.openai`.\n\nSince the TTS output goes to the speakers and the wake word detector listens on the microphone, the assistant plugins pause hotword detection while a conversation is active — otherwise the assistant would hear itself.\n\n## The full configuration\n\n```\nopenai:\n  api_key: YOUR_API_KEY\n  # Or, for a custom setup using a model provider compatible with the OpenAI API,\n  # e.g. a local Ollama instance:\n  # base_url: http://localhost:8000/v1\n  # model: gpt-5\n  # Optional context that can be passed to initialize the assistant\n  # context:\n  #   role: system\n  #   content: You are a useful assistant.\n\nassistant.microwakeword:\n  models:\n    - alexa\n\nassistant.openai:\n  tts_plugin: tts.openai\n  conversation_start_sound: ~/sounds/wake_word_detected.wav\n  input_volume: 100\n\n# assistant.vosk:\n#   model: /path/to/vosk/model\n#   conversation_start_sound: ~/sounds/wake_word_detected.wav\n#   input_volume: 100\n\ntts.openai:\n  output_volume: 80.0\n\n# Or, for a fully local setup that uses tts.piper:\n# tts.piper:\n#   output_volume: 80.0\n```\n\n## What changed on the code side\n\n- **New [`assistant.microwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.microwakeword.html) plugin** : microWakeWord hotword detection through`pymicro-wakeword` , same API as`assistant.openwakeword` , built-in models bundled with the package, custom models via JSON manifests.\n- **[`assistant.openwakeword`](https://docs.platypush.tech/platypush/plugins/assistant.openwakeword.html) fixes** : frames are now 80ms (1280 samples, the size openWakeWord recommends) instead of 500ms — longer chunks only return the highest score over their sub-frames and can miss hotwords — plus a`patience_frames` parameter (consecutive frames above threshold) and an explicit`detection_threshold` parameter (default 0.5).\n- **Shared [`AudioPreprocessor`](https://git.platypush.tech/platypush/platypush/src/branch/master/platypush/plugins/assistant/_audio.py) rework** : Silero/WebRTC/energy VAD backends with`vad_backend: auto` selection, threshold hysteresis, sliding-window majority vote, and an adaptive noise floor. Used by both[`assistant.openai`](https://docs.platypush.tech/platypush/plugins/assistant.openai.html) and[`assistant.vosk`](https://docs.platypush.tech/platypush/plugins/assistant.vosk.html) .\n- **New [Silero VAD wrapper](https://git.platypush.tech/platypush/platypush/src/branch/master/platypush/plugins/assistant/_vad.py)** : ONNX inference through[`onnxruntime`](https://onnxruntime.ai) , recurrent state carried across frames, checksum-verified model download into the workdir.\n\n## Caveats\n\nThe microphone on my RPi doesn't support 16 kHz natively, so audio is captured at 44.1 kHz and software-resampled — it works, but if detection feels sluggish software resampling is usually the first suspect.\n\nThe VAD and threshold defaults were tuned on synthetic audio plus a couple of my audio recordings.\n\nDifferent rooms, microphones and acoustics may yield different results. You may want to adjust `detection_threshold` on the wake word plugin and `noise_floor_factor` on the STT plugins if you don't feel satisfied with your mileage.\n\n## Reactions\n\n## How to interact with this page\n\n#### Webmentions\n\n          To interact via [Webmentions](https://indieweb.org/Webmention),\n          send an activity that references [this URL](https://blog.platypush.tech/article/Rewriting-my-voice-assistant-pipeline) from a platform that supports Webmentions,\n          such as [Lemmy](https://joinlemmy.org/),\n          [WordPress with Webmention plugins](https://wordpress.org/plugins/webmention/),\n          or any [IndieWeb-compatible site](https://indieweb.org/Webmention#Publishing_Software).\n        \n\n#### ActivityPub\n\n- \n            Follow [@blog@platypush.tech](https://blog.platypush.tech/@blog) on your ActivityPub platform (e.g. Mastodon, Misskey, Pleroma, Lemmy).\n- \n            Mention [@blog@platypush.tech](https://blog.platypush.tech/@blog) in a post to feature on the[Guestbook](https://blog.platypush.tech/guestbook) .\n- \n            Search for [this URL](https://blog.platypush.tech/article/Rewriting-my-voice-assistant-pipeline) on your instance to find and interact with the post.\n- Like, boost, quote, or reply to the post to feature your activity here.\n\nIt's 2026. You deserve a good local-first or local-only voice assistant on your [#RaspberryPi](https://manganiello.tech/tags/raspberrypi)\n\n[blog.platypush.tech/article/Rewriting-my-voice-assistant-pipeline](https://blog.platypush.tech/article/Rewriting-my-voice-assistant-pipeline)", "url": "https://wpnews.pro/news/rebuilding-my-local-voice-assistant-pipeline", "canonical_source": "https://blog.platypush.tech/article/Rewriting-my-voice-assistant-pipeline", "published_at": "2026-11-19 20:57:14+00:00", "updated_at": "2026-11-19 21:14:57.215560+00:00", "lang": "en", "topics": ["ai-tools", "natural-language-processing", "ai-products"], "entities": ["Platypush", "microWakeWord", "openWakeWord", "pymicro-wakeword", "ESPHome", "Home Assistant Voice", "Fabio Manganiello"], "also_reported_by": [], "takeaway": "Swapping openWakeWord's 2023 v0.1 models for microWakeWord's maintained models via the pymicro-wakeword bindings cut false positives and negatives, and patience_frames can default to 0 because microWakeWord models already smooth output over a sliding window.", "recommendation": {"advice": "Replace stale wake-word models with actively maintained ones like microWakeWord's, and set patience_frames to 0 since the models already smooth detections over a sliding window.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/rebuilding-my-local-voice-assistant-pipeline", "markdown": "https://wpnews.pro/news/rebuilding-my-local-voice-assistant-pipeline.md", "text": "https://wpnews.pro/news/rebuilding-my-local-voice-assistant-pipeline.txt", "jsonld": "https://wpnews.pro/news/rebuilding-my-local-voice-assistant-pipeline.jsonld"}}