# Adding Voice to a Java AI Assistant — Whisper, TTS, and the Voice Conversation Loop > Source: > Published: 2026-07-07 06:14:19+00:00 How we gave Jarvis the ability to hear and speak — Phase 5 of the Jarvis AI Platform After **Phase 4**, Jarvis could answer questions using real tools. ``` You: What is the weather in Kathmandu? Jarvis: [calls WeatherTool] It is 22°C and sunny. You: What is 2847 × 391? Jarvis: [calls CalculatorTool] 1,113,177 ``` But every interaction required typing. Phase 5 changed that. ``` BEFORE Phase 5: You type → Jarvis types back AFTER Phase 5: You speak → Whisper transcribes → AI responds → TTS speaks back ``` Simple to describe. Surprisingly nuanced to build correctly. The original plan was to run Whisper locally via Ollama. ``` ollama pull whisper ## Error: pull model manifest: file does not exist ``` Ollama is excellent for language models. It does **not** support audio transcription models. This forced a rethink. We designed `WhisperTranscriptionService` to support two backends. Groq provides **Whisper large-v3-turbo** through an OpenAI-compatible API. The free tier offers **6,000 requests/day** with no credit card required. ``` Set GROQ_API_KEY in .env ↓ Works immediately ``` For users who want completely local transcription: ``` git clone https://github.com/ggerganov/whisper.cpp cd whisper.cpp make bash ./models/download-ggml-model.sh base.en ./server -m models/ggml-base.en.bin --port 8178 ``` Both implementations expose the same OpenAI-compatible multipart API. Switching between them is a **single configuration flag**. The most important architectural decision of Phase 5 was this: **Voice is only a wrapper around the existing chat pipeline.** `AiOrchestrator` does **not** change. ``` ❌ WRONG Voice Pipeline ↓ Different AI Pipeline ↓ Different Memory ↓ Different Tools ✅ CORRECT Audio ↓ Whisper ↓ Text ↓ AiOrchestrator.chat() ↓ Existing Memory Existing RAG Existing Tools ↓ Text ↓ Text-to-Speech ``` Everything built in Phases 1–4 continues working automatically. ``` @Service public class WhisperTranscriptionService { private final WebClient webClient; private final String apiKey; private final String model; private final boolean isLocalMode; public Mono transcribe(byte[] audioBytes) { if (audioBytes == null || audioBytes.length == 0) { return Mono.error(VoiceException.emptyAudio()); } if (!isConfigured()) { return Mono.error(new VoiceException( "WHISPER_NOT_CONFIGURED", "Set GROQ_API_KEY in .env or start whisper.cpp server", HttpStatus.SERVICE_UNAVAILABLE)); } return Mono.fromCallable(() -> callWhisperApi(audioBytes, null)) .subscribeOn(Schedulers.boundedElastic()); } } ``` Two design choices are worth highlighting. `Schedulers.boundedElastic()` Calling Groq or whisper.cpp is blocking I/O. Running it on the WebFlux event loop would block every request. `boundedElastic()` keeps the reactive event loop free. `isLocalMode` Local `whisper.cpp` requires no API key. One boolean changes the backend without changing any business logic. Instead of adding another dependency, we chose native OS speech engines. **Why?** Platform support: ``` Windows → PowerShell + System.Speech.Synthesis macOS → say Linux → espeak / text2wave ``` The service detects the platform once at startup. ``` private static final boolean IS_WINDOWS = OS.contains("win"); private static final boolean IS_MAC = OS.contains("mac"); private static final boolean IS_LINUX = OS.contains("nux") || OS.contains("nix"); ``` Voice configuration is entirely environment-driven. ``` ## Windows JARVIS_VOICE_NAME=Microsoft Zira Desktop ## macOS JARVIS_VOICE_NAME=Samantha ## Linux JARVIS_VOICE_NAME=en+f3 ## Speed JARVIS_VOICE_SPEED=1.2 ``` A code review caught this subtle issue. ``` // ❌ Wrong TimeZone.getTimeZone(zoneId) .getDisplayName(false, TimeZone.LONG, Locale.ENGLISH); ``` That always reports **Standard Time**. The correct implementation derives the current DST state. ``` boolean isDst = TimeZone.getTimeZone(zoneId) .inDaylightTime(Date.from(now.toInstant())); TimeZone.getTimeZone(zoneId) .getDisplayName( isDst, TimeZone.LONG, Locale.ENGLISH); ``` Without this fix, users in DST regions would see incorrect timezone names for half the year. LLMs stream tokens. ``` "The" "weather" "in" "London" "is" "22" "°" "C" "and" "sunny" "." ``` Reading individual tokens aloud sounds terrible. The solution was sentence buffering. ``` private void startTtsPipeline(Flux tokenStream) { StringBuilder buffer = new StringBuilder(); tokenStream .flatMap(token -> { buffer.append(token); boolean isSentenceEnd = isSentenceBoundary(token); boolean isBufferFull = buffer.toString() .split("\\s+").length >= MAX_BUFFER_TOKENS; if (isSentenceEnd || isBufferFull) { String sentence = buffer.toString().trim(); buffer.setLength(0); if (!sentence.isBlank()) { return Flux.just(sentence); } } return Flux.empty(); }) .concatMap(textToSpeechService::speakAndPlay) .subscribeOn(Schedulers.boundedElastic()) .subscribe(); } ``` Three implementation details matter. `concatMap()` Sentences must play **sequentially**. Using `flatMap()` would overlap multiple audio streams. `MAX_BUFFER_TOKENS` Some AI responses contain no punctuation. After 50 words we flush automatically. Speech generation happens on `boundedElastic()` . The browser continues receiving streamed tokens immediately. The first implementation blocked streaming. ``` ❌ Wrong Token ↓ TTS ↓ Next Token ``` Terrible user experience. The final architecture separates streaming from speech. ``` Token Stream │ ┌─────────────┴─────────────┐ │ │ ▼ ▼ Browser SSE Sentence Buffer Immediate Background │ │ ▼ ▼ Real-time UI Text-to-Speech ``` Implementation: ``` public Flux voiceChat(...) { Flux tokenStream = orchestrator.chat(request); // Background TTS startTtsPipeline(tokenStream); // Immediate SSE return sessionEvent.concatWith( tokenStream.map(VoiceChatEvent::token)); } ``` The browser updates instantly. Speech begins as soon as the first sentence is complete. Neither blocks the other. The SSE stream emits strongly typed events. ``` public record VoiceChatEvent( EventType type, String data ) { public enum EventType { SESSION, TOKEN, DONE } public static VoiceChatEvent session(UUID id) { return new VoiceChatEvent( EventType.SESSION, id.toString()); } public static VoiceChatEvent token(String text) { return new VoiceChatEvent( EventType.TOKEN, text); } } ``` The initial `SESSION` event solves a practical problem. If the server creates a brand-new conversation, the frontend immediately receives the generated session ID for future requests. Five endpoints power the voice system. ``` POST /api/v1/voice/transcribe POST /api/v1/voice/speak POST /api/v1/voice/speak/bytes POST /api/v1/voice/chat GET /api/v1/voice/status ``` Two speech endpoints exist for different use cases. `/speak` `/speak/bytes` The original plan was simply wrong. Community feedback caught this before implementation. Every Whisper request is blocking. Every TTS process is blocking. Everything runs on `boundedElastic()` . `festival --tts` cannot generate files It only plays audio. Linux audio generation requires: ``` text2wave -o output.wav ``` or Festival's Scheme interface. ``` if (!process.waitFor( TIMEOUT_SECONDS, TimeUnit.SECONDS)) { process.destroyForcibly(); log.warn("TTS generation process timed out"); } ``` Ignoring `waitFor()` leaves orphaned child processes. Timezone names depend on the actual instant, not simply the timezone itself. Before enabling voice, clients can verify availability. ``` curl http://localhost:8080/api/v1/voice/status \ -H "Authorization: Bearer $TOKEN" { "success": true, "data": { "transcriptionAvailable": true, "ttsAvailable": true, "voiceReady": true, "transcriptionMode": "groq-cloud", "ttsEngine": "system-macos" } } ``` `transcriptionMode` `groq-cloud` `local-whisper` `ttsEngine` `system-windows` `system-macos` `system-linux` ``` User speaks "What is the weather in Kathmandu?" │ ▼ Whisper (Groq / whisper.cpp) │ ▼ "What is the weather in Kathmandu?" │ ▼ AiOrchestrator.chat() ├── Session History ├── Long-Term Memory ├── RAG Context └── Tool Calling │ ▼ WeatherTool("Kathmandu") │ ▼ "The weather in Kathmandu is 22°C and clear." │ ├──────────────► Browser (SSE) │ └──────────────► Text-to-Speech ``` Nothing in the AI pipeline changes. Voice simply wraps the architecture built during Phases 1–4. Phase 6 introduced the **Agent System**, allowing Jarvis to plan and execute multi-step tasks autonomously. Phase 7 brings a complete web interface over everything we've built. The backend is now complete. Phases 1–6 are merged, tested, and production-ready. Jarvis can now hear. Jarvis can now speak. Jarvis is open source under the Apache 2.0 License. Current contributor-friendly issues include: ``` #69 CLI voice commands (voice, listen, speak) New Voice integration tests ``` GitHub: ``` github.com/sujankim/jarvis-ai-platform ``` Your AI. Your Data. Your Machine.