{"slug": "adding-voice-to-a-java-ai-assistant-whisper-tts-and-the-voice-conversation-loop", "title": "Adding Voice to a Java AI Assistant — Whisper, TTS, and the Voice Conversation Loop", "summary": "A developer added voice capabilities to the Jarvis AI Platform by integrating Whisper for speech transcription and native OS text-to-speech engines. The voice pipeline wraps the existing chat pipeline, allowing the AI to hear and speak without modifying the core orchestration logic. Two backends for Whisper are supported: Groq's API and local whisper.cpp, switchable via a single configuration flag.", "body_md": "How we gave Jarvis the ability to hear and speak — Phase 5 of the Jarvis AI Platform\n\nAfter **Phase 4**, Jarvis could answer questions using real tools.\n\n```\nYou: What is the weather in Kathmandu?\nJarvis: [calls WeatherTool] It is 22°C and sunny.\n\nYou: What is 2847 × 391?\nJarvis: [calls CalculatorTool] 1,113,177\n```\n\nBut every interaction required typing.\n\nPhase 5 changed that.\n\n```\nBEFORE Phase 5:\nYou type  → Jarvis types back\n\nAFTER Phase 5:\nYou speak → Whisper transcribes → AI responds → TTS speaks back\n```\n\nSimple to describe.\n\nSurprisingly nuanced to build correctly.\n\nThe original plan was to run Whisper locally via Ollama.\n\n```\nollama pull whisper\n\n## Error:\npull model manifest: file does not exist\n```\n\nOllama is excellent for language models.\n\nIt does **not** support audio transcription models.\n\nThis forced a rethink.\n\nWe designed `WhisperTranscriptionService`\n\nto support two backends.\n\nGroq provides **Whisper large-v3-turbo** through an OpenAI-compatible API.\n\nThe free tier offers **6,000 requests/day** with no credit card required.\n\n```\nSet GROQ_API_KEY in .env\n\n↓\n\nWorks immediately\n```\n\nFor users who want completely local transcription:\n\n```\ngit clone https://github.com/ggerganov/whisper.cpp\n\ncd whisper.cpp\n\nmake\n\nbash ./models/download-ggml-model.sh base.en\n\n./server -m models/ggml-base.en.bin --port 8178\n```\n\nBoth implementations expose the same OpenAI-compatible multipart API.\n\nSwitching between them is a **single configuration flag**.\n\nThe most important architectural decision of Phase 5 was this:\n\n**Voice is only a wrapper around the existing chat pipeline.**\n\n`AiOrchestrator`\n\ndoes **not** change.\n\n```\n❌ WRONG\n\nVoice Pipeline\n   ↓\nDifferent AI Pipeline\n   ↓\nDifferent Memory\n   ↓\nDifferent Tools\n\n✅ CORRECT\n\nAudio\n   ↓\nWhisper\n   ↓\nText\n   ↓\nAiOrchestrator.chat()\n   ↓\nExisting Memory\nExisting RAG\nExisting Tools\n   ↓\nText\n   ↓\nText-to-Speech\n```\n\nEverything built in Phases 1–4 continues working automatically.\n\n```\n@Service\npublic class WhisperTranscriptionService {\n\n    private final WebClient webClient;\n    private final String apiKey;\n    private final String model;\n    private final boolean isLocalMode;\n\n    public Mono<String> transcribe(byte[] audioBytes) {\n\n        if (audioBytes == null || audioBytes.length == 0) {\n            return Mono.error(VoiceException.emptyAudio());\n        }\n\n        if (!isConfigured()) {\n            return Mono.error(new VoiceException(\n                    \"WHISPER_NOT_CONFIGURED\",\n                    \"Set GROQ_API_KEY in .env or start whisper.cpp server\",\n                    HttpStatus.SERVICE_UNAVAILABLE));\n        }\n\n        return Mono.fromCallable(() ->\n                        callWhisperApi(audioBytes, null))\n                .subscribeOn(Schedulers.boundedElastic());\n    }\n}\n```\n\nTwo design choices are worth highlighting.\n\n`Schedulers.boundedElastic()`\n\nCalling Groq or whisper.cpp is blocking I/O.\n\nRunning it on the WebFlux event loop would block every request.\n\n`boundedElastic()`\n\nkeeps the reactive event loop free.\n\n`isLocalMode`\n\nLocal `whisper.cpp`\n\nrequires no API key.\n\nOne boolean changes the backend without changing any business logic.\n\nInstead of adding another dependency, we chose native OS speech engines.\n\n**Why?**\n\nPlatform support:\n\n```\nWindows → PowerShell + System.Speech.Synthesis\n\nmacOS   → say\n\nLinux   → espeak / text2wave\n```\n\nThe service detects the platform once at startup.\n\n```\nprivate static final boolean IS_WINDOWS = OS.contains(\"win\");\nprivate static final boolean IS_MAC = OS.contains(\"mac\");\nprivate static final boolean IS_LINUX =\n        OS.contains(\"nux\") || OS.contains(\"nix\");\n```\n\nVoice configuration is entirely environment-driven.\n\n```\n## Windows\nJARVIS_VOICE_NAME=Microsoft Zira Desktop\n\n## macOS\nJARVIS_VOICE_NAME=Samantha\n\n## Linux\nJARVIS_VOICE_NAME=en+f3\n\n## Speed\nJARVIS_VOICE_SPEED=1.2\n```\n\nA code review caught this subtle issue.\n\n```\n// ❌ Wrong\nTimeZone.getTimeZone(zoneId)\n        .getDisplayName(false,\n                TimeZone.LONG,\n                Locale.ENGLISH);\n```\n\nThat always reports **Standard Time**.\n\nThe correct implementation derives the current DST state.\n\n```\nboolean isDst =\n        TimeZone.getTimeZone(zoneId)\n                .inDaylightTime(Date.from(now.toInstant()));\n\nTimeZone.getTimeZone(zoneId)\n        .getDisplayName(\n                isDst,\n                TimeZone.LONG,\n                Locale.ENGLISH);\n```\n\nWithout this fix, users in DST regions would see incorrect timezone names for half the year.\n\nLLMs stream tokens.\n\n```\n\"The\"\n\n\"weather\"\n\n\"in\"\n\n\"London\"\n\n\"is\"\n\n\"22\"\n\n\"°\"\n\n\"C\"\n\n\"and\"\n\n\"sunny\"\n\n\".\"\n```\n\nReading individual tokens aloud sounds terrible.\n\nThe solution was sentence buffering.\n\n```\nprivate void startTtsPipeline(Flux<String> tokenStream) {\n\n    StringBuilder buffer = new StringBuilder();\n\n    tokenStream\n            .flatMap(token -> {\n\n                buffer.append(token);\n\n                boolean isSentenceEnd =\n                        isSentenceBoundary(token);\n\n                boolean isBufferFull =\n                        buffer.toString()\n                              .split(\"\\\\s+\").length\n                              >= MAX_BUFFER_TOKENS;\n\n                if (isSentenceEnd || isBufferFull) {\n\n                    String sentence =\n                            buffer.toString().trim();\n\n                    buffer.setLength(0);\n\n                    if (!sentence.isBlank()) {\n                        return Flux.just(sentence);\n                    }\n                }\n\n                return Flux.<String>empty();\n            })\n\n            .concatMap(textToSpeechService::speakAndPlay)\n\n            .subscribeOn(Schedulers.boundedElastic())\n\n            .subscribe();\n}\n```\n\nThree implementation details matter.\n\n`concatMap()`\n\nSentences must play **sequentially**.\n\nUsing `flatMap()`\n\nwould overlap multiple audio streams.\n\n`MAX_BUFFER_TOKENS`\n\nSome AI responses contain no punctuation.\n\nAfter 50 words we flush automatically.\n\nSpeech generation happens on `boundedElastic()`\n\n.\n\nThe browser continues receiving streamed tokens immediately.\n\nThe first implementation blocked streaming.\n\n```\n❌ Wrong\n\nToken\n ↓\nTTS\n ↓\nNext Token\n```\n\nTerrible user experience.\n\nThe final architecture separates streaming from speech.\n\n```\n               Token Stream\n                    │\n      ┌─────────────┴─────────────┐\n      │                           │\n      ▼                           ▼\n\nBrowser SSE               Sentence Buffer\n\nImmediate                  Background\n\n      │                           │\n      ▼                           ▼\n\nReal-time UI             Text-to-Speech\n```\n\nImplementation:\n\n```\npublic Flux<VoiceChatEvent> voiceChat(...) {\n\n    Flux<String> tokenStream =\n            orchestrator.chat(request);\n\n    // Background TTS\n    startTtsPipeline(tokenStream);\n\n    // Immediate SSE\n    return sessionEvent.concatWith(\n            tokenStream.map(VoiceChatEvent::token));\n}\n```\n\nThe browser updates instantly.\n\nSpeech begins as soon as the first sentence is complete.\n\nNeither blocks the other.\n\nThe SSE stream emits strongly typed events.\n\n```\npublic record VoiceChatEvent(\n        EventType type,\n        String data\n) {\n\n    public enum EventType {\n        SESSION,\n        TOKEN,\n        DONE\n    }\n\n    public static VoiceChatEvent session(UUID id) {\n        return new VoiceChatEvent(\n                EventType.SESSION,\n                id.toString());\n    }\n\n    public static VoiceChatEvent token(String text) {\n        return new VoiceChatEvent(\n                EventType.TOKEN,\n                text);\n    }\n}\n```\n\nThe initial `SESSION`\n\nevent solves a practical problem.\n\nIf the server creates a brand-new conversation, the frontend immediately receives the generated session ID for future requests.\n\nFive endpoints power the voice system.\n\n```\nPOST /api/v1/voice/transcribe\nPOST /api/v1/voice/speak\nPOST /api/v1/voice/speak/bytes\nPOST /api/v1/voice/chat\nGET  /api/v1/voice/status\n```\n\nTwo speech endpoints exist for different use cases.\n\n`/speak`\n\n`/speak/bytes`\n\nThe original plan was simply wrong.\n\nCommunity feedback caught this before implementation.\n\nEvery Whisper request is blocking.\n\nEvery TTS process is blocking.\n\nEverything runs on `boundedElastic()`\n\n.\n\n`festival --tts`\n\ncannot generate files\nIt only plays audio.\n\nLinux audio generation requires:\n\n```\ntext2wave -o output.wav\n```\n\nor Festival's Scheme interface.\n\n```\nif (!process.waitFor(\n        TIMEOUT_SECONDS,\n        TimeUnit.SECONDS)) {\n\n    process.destroyForcibly();\n\n    log.warn(\"TTS generation process timed out\");\n}\n```\n\nIgnoring `waitFor()`\n\nleaves orphaned child processes.\n\nTimezone names depend on the actual instant, not simply the timezone itself.\n\nBefore enabling voice, clients can verify availability.\n\n```\ncurl http://localhost:8080/api/v1/voice/status \\\n     -H \"Authorization: Bearer $TOKEN\"\n{\n  \"success\": true,\n  \"data\": {\n    \"transcriptionAvailable\": true,\n    \"ttsAvailable\": true,\n    \"voiceReady\": true,\n    \"transcriptionMode\": \"groq-cloud\",\n    \"ttsEngine\": \"system-macos\"\n  }\n}\n```\n\n`transcriptionMode`\n\n`groq-cloud`\n\n`local-whisper`\n\n`ttsEngine`\n\n`system-windows`\n\n`system-macos`\n\n`system-linux`\n\n```\nUser speaks\n\n\"What is the weather in Kathmandu?\"\n\n        │\n        ▼\n\nWhisper\n(Groq / whisper.cpp)\n\n        │\n        ▼\n\n\"What is the weather in Kathmandu?\"\n\n        │\n        ▼\n\nAiOrchestrator.chat()\n\n    ├── Session History\n    ├── Long-Term Memory\n    ├── RAG Context\n    └── Tool Calling\n\n        │\n        ▼\n\nWeatherTool(\"Kathmandu\")\n\n        │\n        ▼\n\n\"The weather in Kathmandu is\n22°C and clear.\"\n\n        │\n        ├──────────────► Browser (SSE)\n        │\n        └──────────────► Text-to-Speech\n```\n\nNothing in the AI pipeline changes.\n\nVoice simply wraps the architecture built during Phases 1–4.\n\nPhase 6 introduced the **Agent System**, allowing Jarvis to plan and execute multi-step tasks autonomously.\n\nPhase 7 brings a complete web interface over everything we've built.\n\nThe backend is now complete.\n\nPhases 1–6 are merged, tested, and production-ready.\n\nJarvis can now hear.\n\nJarvis can now speak.\n\nJarvis is open source under the Apache 2.0 License.\n\nCurrent contributor-friendly issues include:\n\n```\n#69  CLI voice commands (voice, listen, speak)\n\nNew  Voice integration tests\n```\n\nGitHub:\n\n```\ngithub.com/sujankim/jarvis-ai-platform\n```\n\nYour AI. Your Data. Your Machine.", "url": "https://wpnews.pro/news/adding-voice-to-a-java-ai-assistant-whisper-tts-and-the-voice-conversation-loop", "canonical_source": "https://dev.to/sujankim/adding-voice-to-a-java-ai-assistant-whisper-tts-and-the-voice-conversation-loop-5ed7", "published_at": "2026-07-07 06:14:19+00:00", "updated_at": "2026-07-07 06:58:19.123152+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "natural-language-processing", "developer-tools"], "entities": ["Jarvis AI Platform", "Whisper", "Groq", "Ollama", "whisper.cpp", "AiOrchestrator", "WhisperTranscriptionService"], "alternates": {"html": "https://wpnews.pro/news/adding-voice-to-a-java-ai-assistant-whisper-tts-and-the-voice-conversation-loop", "markdown": "https://wpnews.pro/news/adding-voice-to-a-java-ai-assistant-whisper-tts-and-the-voice-conversation-loop.md", "text": "https://wpnews.pro/news/adding-voice-to-a-java-ai-assistant-whisper-tts-and-the-voice-conversation-loop.txt", "jsonld": "https://wpnews.pro/news/adding-voice-to-a-java-ai-assistant-whisper-tts-and-the-voice-conversation-loop.jsonld"}}