Adding Voice to a Java AI Assistant — Whisper, TTS, and the Voice Conversation Loop

A developer added voice capabilities to the Jarvis AI Platform by integrating Whisper for speech transcription and native OS text-to-speech engines. The voice pipeline wraps the existing chat pipeline, allowing the AI to hear and speak without modifying the core orchestration logic. Two backends for Whisper are supported: Groq's API and local whisper.cpp, switchable via a single configuration flag.

How we gave Jarvis the ability to hear and speak — Phase 5 of the Jarvis AI Platform After Phase 4 , Jarvis could answer questions using real tools. You: What is the weather in Kathmandu? Jarvis: calls WeatherTool It is 22°C and sunny. You: What is 2847 × 391? Jarvis: calls CalculatorTool 1,113,177 But every interaction required typing. Phase 5 changed that. BEFORE Phase 5: You type → Jarvis types back AFTER Phase 5: You speak → Whisper transcribes → AI responds → TTS speaks back Simple to describe. Surprisingly nuanced to build correctly. The original plan was to run Whisper locally via Ollama. ollama pull whisper Error: pull model manifest: file does not exist Ollama is excellent for language models. It does not support audio transcription models. This forced a rethink. We designed WhisperTranscriptionService to support two backends. Groq provides Whisper large-v3-turbo through an OpenAI-compatible API. The free tier offers 6,000 requests/day with no credit card required. Set GROQ API KEY in .env ↓ Works immediately For users who want completely local transcription: git clone https://github.com/ggerganov/whisper.cpp cd whisper.cpp make bash ./models/download-ggml-model.sh base.en ./server -m models/ggml-base.en.bin --port 8178 Both implementations expose the same OpenAI-compatible multipart API. Switching between them is a single configuration flag . The most important architectural decision of Phase 5 was this: Voice is only a wrapper around the existing chat pipeline. AiOrchestrator does not change. ❌ WRONG Voice Pipeline ↓ Different AI Pipeline ↓ Different Memory ↓ Different Tools ✅ CORRECT Audio ↓ Whisper ↓ Text ↓ AiOrchestrator.chat ↓ Existing Memory Existing RAG Existing Tools ↓ Text ↓ Text-to-Speech Everything built in Phases 1–4 continues working automatically. @Service public class WhisperTranscriptionService { private final WebClient webClient; private final String apiKey; private final String model; private final boolean isLocalMode; public Mono<String transcribe byte audioBytes { if audioBytes == null || audioBytes.length == 0 { return Mono.error VoiceException.emptyAudio ; } if isConfigured { return Mono.error new VoiceException "WHISPER NOT CONFIGURED", "Set GROQ API KEY in .env or start whisper.cpp server", HttpStatus.SERVICE UNAVAILABLE ; } return Mono.fromCallable - callWhisperApi audioBytes, null .subscribeOn Schedulers.boundedElastic ; } } Two design choices are worth highlighting. Schedulers.boundedElastic Calling Groq or whisper.cpp is blocking I/O. Running it on the WebFlux event loop would block every request. boundedElastic keeps the reactive event loop free. isLocalMode Local whisper.cpp requires no API key. One boolean changes the backend without changing any business logic. Instead of adding another dependency, we chose native OS speech engines. Why? Platform support: Windows → PowerShell + System.Speech.Synthesis macOS → say Linux → espeak / text2wave The service detects the platform once at startup. private static final boolean IS WINDOWS = OS.contains "win" ; private static final boolean IS MAC = OS.contains "mac" ; private static final boolean IS LINUX = OS.contains "nux" || OS.contains "nix" ; Voice configuration is entirely environment-driven. Windows JARVIS VOICE NAME=Microsoft Zira Desktop macOS JARVIS VOICE NAME=Samantha Linux JARVIS VOICE NAME=en+f3 Speed JARVIS VOICE SPEED=1.2 A code review caught this subtle issue. // ❌ Wrong TimeZone.getTimeZone zoneId .getDisplayName false, TimeZone.LONG, Locale.ENGLISH ; That always reports Standard Time . The correct implementation derives the current DST state. boolean isDst = TimeZone.getTimeZone zoneId .inDaylightTime Date.from now.toInstant ; TimeZone.getTimeZone zoneId .getDisplayName isDst, TimeZone.LONG, Locale.ENGLISH ; Without this fix, users in DST regions would see incorrect timezone names for half the year. LLMs stream tokens. "The" "weather" "in" "London" "is" "22" "°" "C" "and" "sunny" "." Reading individual tokens aloud sounds terrible. The solution was sentence buffering. private void startTtsPipeline Flux<String tokenStream { StringBuilder buffer = new StringBuilder ; tokenStream .flatMap token - { buffer.append token ; boolean isSentenceEnd = isSentenceBoundary token ; boolean isBufferFull = buffer.toString .split "\\s+" .length = MAX BUFFER TOKENS; if isSentenceEnd || isBufferFull { String sentence = buffer.toString .trim ; buffer.setLength 0 ; if sentence.isBlank { return Flux.just sentence ; } } return Flux.<String empty ; } .concatMap textToSpeechService::speakAndPlay .subscribeOn Schedulers.boundedElastic .subscribe ; } Three implementation details matter. concatMap Sentences must play sequentially . Using flatMap would overlap multiple audio streams. MAX BUFFER TOKENS Some AI responses contain no punctuation. After 50 words we flush automatically. Speech generation happens on boundedElastic . The browser continues receiving streamed tokens immediately. The first implementation blocked streaming. ❌ Wrong Token ↓ TTS ↓ Next Token Terrible user experience. The final architecture separates streaming from speech. Token Stream │ ┌─────────────┴─────────────┐ │ │ ▼ ▼ Browser SSE Sentence Buffer Immediate Background │ │ ▼ ▼ Real-time UI Text-to-Speech Implementation: public Flux<VoiceChatEvent voiceChat ... { Flux<String tokenStream = orchestrator.chat request ; // Background TTS startTtsPipeline tokenStream ; // Immediate SSE return sessionEvent.concatWith tokenStream.map VoiceChatEvent::token ; } The browser updates instantly. Speech begins as soon as the first sentence is complete. Neither blocks the other. The SSE stream emits strongly typed events. public record VoiceChatEvent EventType type, String data { public enum EventType { SESSION, TOKEN, DONE } public static VoiceChatEvent session UUID id { return new VoiceChatEvent EventType.SESSION, id.toString ; } public static VoiceChatEvent token String text { return new VoiceChatEvent EventType.TOKEN, text ; } } The initial SESSION event solves a practical problem. If the server creates a brand-new conversation, the frontend immediately receives the generated session ID for future requests. Five endpoints power the voice system. POST /api/v1/voice/transcribe POST /api/v1/voice/speak POST /api/v1/voice/speak/bytes POST /api/v1/voice/chat GET /api/v1/voice/status Two speech endpoints exist for different use cases. /speak /speak/bytes The original plan was simply wrong. Community feedback caught this before implementation. Every Whisper request is blocking. Every TTS process is blocking. Everything runs on boundedElastic . festival --tts cannot generate files It only plays audio. Linux audio generation requires: text2wave -o output.wav or Festival's Scheme interface. if process.waitFor TIMEOUT SECONDS, TimeUnit.SECONDS { process.destroyForcibly ; log.warn "TTS generation process timed out" ; } Ignoring waitFor leaves orphaned child processes. Timezone names depend on the actual instant, not simply the timezone itself. Before enabling voice, clients can verify availability. curl http://localhost:8080/api/v1/voice/status \ -H "Authorization: Bearer $TOKEN" { "success": true, "data": { "transcriptionAvailable": true, "ttsAvailable": true, "voiceReady": true, "transcriptionMode": "groq-cloud", "ttsEngine": "system-macos" } } transcriptionMode groq-cloud local-whisper ttsEngine system-windows system-macos system-linux User speaks "What is the weather in Kathmandu?" │ ▼ Whisper Groq / whisper.cpp │ ▼ "What is the weather in Kathmandu?" │ ▼ AiOrchestrator.chat ├── Session History ├── Long-Term Memory ├── RAG Context └── Tool Calling │ ▼ WeatherTool "Kathmandu" │ ▼ "The weather in Kathmandu is 22°C and clear." │ ├──────────────► Browser SSE │ └──────────────► Text-to-Speech Nothing in the AI pipeline changes. Voice simply wraps the architecture built during Phases 1–4. Phase 6 introduced the Agent System , allowing Jarvis to plan and execute multi-step tasks autonomously. Phase 7 brings a complete web interface over everything we've built. The backend is now complete. Phases 1–6 are merged, tested, and production-ready. Jarvis can now hear. Jarvis can now speak. Jarvis is open source under the Apache 2.0 License. Current contributor-friendly issues include: 69 CLI voice commands voice, listen, speak New Voice integration tests GitHub: github.com/sujankim/jarvis-ai-platform Your AI. Your Data. Your Machine.