{"slug": "aquila-voice-assistant-test-suite-for-home-assistant", "title": "Aquila Voice Assistant Test Suite for Home Assistant", "summary": "An open-source test suite for voice assistants, developed by Aquila, is now available, with reproducible tests and a leaderboard at https://git.cicero.sh/aquila/ha-voice-test-suite/. The creator, who has only a 4GB vRAM GPU, has tested small LLMs like Qwen3 4B Instruct and is seeking community help to run the suite on larger local models (20B+) and frontier cloud models, expecting scores in the low 90s with 60-90 minute durations.", "body_md": "Put together an extensive open source test suite for voice assistants. You can view it including current leaderboard at: https://git.cicero.sh/aquila/ha-voice-test-suite/\n\nTests are reproduceable, with clear instructions on how to run them on your machine there.\n\nI only have a GPU with 4GB vRAM, hence only capable of testing the small LLMs like Qwen3 4B Instruct. Have Gemma 4 running right now, but it's insanely slow and probably another 24 - 48 hours before it finishes.\n\nTried cloud models like Claude Sonnet 5 and Grok 4.5, but was rate limited each time, and got fed up so dropped it. Maybe someone out there uses AI more than me and has proper rate limits and wouldn't mind running the test suite on some frontier models as I'm assuming folks would appreciate seeing the results. I'm already confident they'll get low 90s with 60 - 90 minute duration, so not overly worried about it.\n\nIf anyone has larger GPU and wouldn't mind giving it a spin on larger 20B+ local LLMs that would be awesome. Running the tests against any model is quite straight forward, instructions in the readme.\n\nEnjoy!\n\nComments URL: [https://news.ycombinator.com/item?id=49150636](https://news.ycombinator.com/item?id=49150636)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/aquila-voice-assistant-test-suite-for-home-assistant", "canonical_source": "https://news.ycombinator.com/item?id=49150636", "published_at": "2026-08-03 02:41:54+00:00", "updated_at": "2026-08-03 02:52:27.380293+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools"], "entities": ["Aquila", "Qwen3 4B Instruct", "Gemma 4", "Claude Sonnet 5", "Grok 4.5"], "alternates": {"html": "https://wpnews.pro/news/aquila-voice-assistant-test-suite-for-home-assistant", "markdown": "https://wpnews.pro/news/aquila-voice-assistant-test-suite-for-home-assistant.md", "text": "https://wpnews.pro/news/aquila-voice-assistant-test-suite-for-home-assistant.txt", "jsonld": "https://wpnews.pro/news/aquila-voice-assistant-test-suite-for-home-assistant.jsonld"}}