Thanks for the suggestions as I was questioning if I should run the test against written or spoken input and realized my release was missing speech input. Since the openclaw browser interface doesn’t allow speech or audio, I put that feature in the console. I’d like to optimize the release to have a casual voice conversation with the agent in telegram, but that will take some further optimizations. Below is the AI assisted response.
I ran it, and the results are in docs/VOICE.md in the repo.
Setup: 12 questions about facts stored in the memory service, among about 250 other memories, each with a near-miss distractor (my sister’s birthday vs a coworker’s with the same name, the cabin wifi password vs the home one). Retrieval is held fixed, and every answer is spoken with the local voice. The numbers are the median time from the end of the question to the first sound, on an RTX 5080.
| Model | Typed: right | Typed: first sound | Spoken: right | Spoken: first sound |
|---|---|---|---|---|
| 27B, 64k context | 12/12 | 3.6 s | 12/12 | 4.1 s |
| 27B with MTP | 12/12 | 4.0 s | 12/12 | 4.4 s |
| 14B | 12/12 | 3.3 s | 12/12 | 4.3 s |
| 8B | 12/12 | 3.4 s | 11/12 | 4.2 s |
Memory retrieval was on for every run: the questions went through the real memory service and the memory proxy code from the repo (BGE-M3 search plus a reranker), and the same retrieved memories were given to every model. For typed questions the right fact was retrieved 12/12 times, with the near-miss distractor alongside it in 10/12, and the lookup took about 0.25 s (median), so it is a small part of the wait. For spoken questions I ran the lookup again on the words that were heard: the right fact came back 12/12 times on three of the models and 11/12 on the 8B. That one miss is the unexplained case below.
Since my first post, the repo has gained: