Vellum has introduced Voice Mode for its personal assistant, enabling spoken conversation to be processed through the same agent loop that powers text chat. Speech is transcribed using Deepgram, processed by the assistant with full access to its tools, memory, and skills, and then spoken back through ElevenLabs. The assistant can browse the web, read files, run code, send messages, and manage a calendar while the conversation is ongoing, maintaining the same identity across both text and voice interfaces.
Vellum opted to retain the classic cascade architecture instead of transitioning to a native speech-to-speech model. This decision was based on the belief that a closed audio-in, audio-out model does not allow for an agent loop, tools, memory, or approvals. The engineering effort focused on seamlessly integrating these components. A speculative launch mechanism initiates the agent's response as soon as the voice detector identifies trailing silence, running the endpoint decision concurrently with the time to first token, and rolling back the turn if the speaker was merely pausing mid-thought. Each turn first engages a fast front model that either responds directly, remains silent if the speaker seems unfinished, or escalates to a more robust model while delivering a brief holding phrase audibly. During lengthy tool calls, the assistant narrates progress in audio only, ensuring that this narration does not persist in the transcript.
Interruption handling is a feature Vellum describes as custom engineering rather than an API flag. Interruptions such as cutting in mid-sentence, redirecting, or adding information while the assistant is processing are managed effectively, with the assistant yielding and maintaining context. A guard mechanism waits for a quarter second of sustained speech before yielding, preventing the assistant's audio from being misinterpreted as an interruption due to browser echo cancellation. Synthesis operates seamlessly, with the opening clause spoken as soon as it is ready and a prefetch pump ensuring the next segment is ready behind the current playback.
Voices are managed through a Vellum Managed Voice option that automatically handles ElevenLabs voices, with the option to provide an ElevenLabs key for custom voices. If no key is provided, the system defaults to managed voices. Voice continuity is maintained across devices, allowing a task started on a phone to continue on a Mac with the same assistant and memory. Complex multi-step processes automatically transition to text chat when text becomes the more suitable medium.
Start testing Vellum AI
Learn more Vellum's personal assistant is designed with its own identity, memory, and skills, operating on macOS, iOS, web browsers, Slack, Telegram, and more! There are also plans for future Android and Windows support. The assistant comes equipped with over 60 skills, a permission system ranging from strict to full access, and credentials stored in the macOS Keychain when self-hosted or in an isolated vault on the managed platform. The core assistant is open source, and the company is led by Chief Executive Akash Sharma, alongside Co-Chief Technology Officers Noa Flaherty and Sidd Seethepalli.