How we gave Jarvis the ability to hear and speak — Phase 5 of the Jarvis AI Platform
After Phase 4, Jarvis could answer questions using real tools.
You: What is the weather in Kathmandu?
Jarvis: [calls WeatherTool] It is 22°C and sunny.
You: What is 2847 × 391?
Jarvis: [calls CalculatorTool] 1,113,177
But every interaction required typing.
Phase 5 changed that.
BEFORE Phase 5:
You type → Jarvis types back
AFTER Phase 5:
You speak → Whisper transcribes → AI responds → TTS speaks back
Simple to describe.
Surprisingly nuanced to build correctly.
The original plan was to run Whisper locally via Ollama.
ollama pull whisper
## Error:
pull model manifest: file does not exist
Ollama is excellent for language models.
It does not support audio transcription models.
This forced a rethink.
We designed WhisperTranscriptionService
to support two backends.
Groq provides Whisper large-v3-turbo through an OpenAI-compatible API.
The free tier offers 6,000 requests/day with no credit card required.
Set GROQ_API_KEY in .env
↓
Works immediately
For users who want completely local transcription:
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make
bash ./models/download-ggml-model.sh base.en
./server -m models/ggml-base.en.bin --port 8178
Both implementations expose the same OpenAI-compatible multipart API.
Switching between them is a single configuration flag.
The most important architectural decision of Phase 5 was this:
Voice is only a wrapper around the existing chat pipeline.
AiOrchestrator
does not change.
❌ WRONG
Voice Pipeline
↓
Different AI Pipeline
↓
Different Memory
↓
Different Tools
✅ CORRECT
Audio
↓
Whisper
↓
Text
↓
AiOrchestrator.chat()
↓
Existing Memory
Existing RAG
Existing Tools
↓
Text
↓
Text-to-Speech
Everything built in Phases 1–4 continues working automatically.
@Service
public class WhisperTranscriptionService {
private final WebClient webClient;
private final String apiKey;
private final String model;
private final boolean isLocalMode;
public Mono<String> transcribe(byte[] audioBytes) {
if (audioBytes == null || audioBytes.length == 0) {
return Mono.error(VoiceException.emptyAudio());
}
if (!isConfigured()) {
return Mono.error(new VoiceException(
"WHISPER_NOT_CONFIGURED",
"Set GROQ_API_KEY in .env or start whisper.cpp server",
HttpStatus.SERVICE_UNAVAILABLE));
}
return Mono.fromCallable(() ->
callWhisperApi(audioBytes, null))
.subscribeOn(Schedulers.boundedElastic());
}
}
Two design choices are worth highlighting.
Schedulers.boundedElastic()
Calling Groq or whisper.cpp is blocking I/O.
Running it on the WebFlux event loop would block every request.
boundedElastic()
keeps the reactive event loop free.
isLocalMode
Local whisper.cpp
requires no API key.
One boolean changes the backend without changing any business logic.
Instead of adding another dependency, we chose native OS speech engines.
Why?
Platform support:
Windows → PowerShell + System.Speech.Synthesis
macOS → say
Linux → espeak / text2wave
The service detects the platform once at startup.
private static final boolean IS_WINDOWS = OS.contains("win");
private static final boolean IS_MAC = OS.contains("mac");
private static final boolean IS_LINUX =
OS.contains("nux") || OS.contains("nix");
Voice configuration is entirely environment-driven.
## Windows
JARVIS_VOICE_NAME=Microsoft Zira Desktop
## macOS
JARVIS_VOICE_NAME=Samantha
## Linux
JARVIS_VOICE_NAME=en+f3
## Speed
JARVIS_VOICE_SPEED=1.2
A code review caught this subtle issue.
// ❌ Wrong
TimeZone.getTimeZone(zoneId)
.getDisplayName(false,
TimeZone.LONG,
Locale.ENGLISH);
That always reports Standard Time.
The correct implementation derives the current DST state.
boolean isDst =
TimeZone.getTimeZone(zoneId)
.inDaylightTime(Date.from(now.toInstant()));
TimeZone.getTimeZone(zoneId)
.getDisplayName(
isDst,
TimeZone.LONG,
Locale.ENGLISH);
Without this fix, users in DST regions would see incorrect timezone names for half the year.
LLMs stream tokens.
"The"
"weather"
"in"
"London"
"is"
"22"
"°"
"C"
"and"
"sunny"
"."
Reading individual tokens aloud sounds terrible.
The solution was sentence buffering.
private void startTtsPipeline(Flux<String> tokenStream) {
StringBuilder buffer = new StringBuilder();
tokenStream
.flatMap(token -> {
buffer.append(token);
boolean isSentenceEnd =
isSentenceBoundary(token);
boolean isBufferFull =
buffer.toString()
.split("\\s+").length
>= MAX_BUFFER_TOKENS;
if (isSentenceEnd || isBufferFull) {
String sentence =
buffer.toString().trim();
buffer.setLength(0);
if (!sentence.isBlank()) {
return Flux.just(sentence);
}
}
return Flux.<String>empty();
})
.concatMap(textToSpeechService::speakAndPlay)
.subscribeOn(Schedulers.boundedElastic())
.subscribe();
}
Three implementation details matter.
concatMap()
Sentences must play sequentially.
Using flatMap()
would overlap multiple audio streams.
MAX_BUFFER_TOKENS
Some AI responses contain no punctuation.
After 50 words we flush automatically.
Speech generation happens on boundedElastic()
.
The browser continues receiving streamed tokens immediately.
The first implementation blocked streaming.
❌ Wrong
Token
↓
TTS
↓
Next Token
Terrible user experience.
The final architecture separates streaming from speech.
Token Stream
│
┌─────────────┴─────────────┐
│ │
▼ ▼
Browser SSE Sentence Buffer
Immediate Background
│ │
▼ ▼
Real-time UI Text-to-Speech
Implementation:
public Flux<VoiceChatEvent> voiceChat(...) {
Flux<String> tokenStream =
orchestrator.chat(request);
// Background TTS
startTtsPipeline(tokenStream);
// Immediate SSE
return sessionEvent.concatWith(
tokenStream.map(VoiceChatEvent::token));
}
The browser updates instantly.
Speech begins as soon as the first sentence is complete.
Neither blocks the other.
The SSE stream emits strongly typed events.
public record VoiceChatEvent(
EventType type,
String data
) {
public enum EventType {
SESSION,
TOKEN,
DONE
}
public static VoiceChatEvent session(UUID id) {
return new VoiceChatEvent(
EventType.SESSION,
id.toString());
}
public static VoiceChatEvent token(String text) {
return new VoiceChatEvent(
EventType.TOKEN,
text);
}
}
The initial SESSION
event solves a practical problem.
If the server creates a brand-new conversation, the frontend immediately receives the generated session ID for future requests.
Five endpoints power the voice system.
POST /api/v1/voice/transcribe
POST /api/v1/voice/speak
POST /api/v1/voice/speak/bytes
POST /api/v1/voice/chat
GET /api/v1/voice/status
Two speech endpoints exist for different use cases.
/speak
/speak/bytes
The original plan was simply wrong.
Community feedback caught this before implementation.
Every Whisper request is blocking.
Every TTS process is blocking.
Everything runs on boundedElastic()
.
festival --tts
cannot generate files It only plays audio.
Linux audio generation requires:
text2wave -o output.wav
or Festival's Scheme interface.
if (!process.waitFor(
TIMEOUT_SECONDS,
TimeUnit.SECONDS)) {
process.destroyForcibly();
log.warn("TTS generation process timed out");
}
Ignoring waitFor()
leaves orphaned child processes.
Timezone names depend on the actual instant, not simply the timezone itself.
Before enabling voice, clients can verify availability.
curl http://localhost:8080/api/v1/voice/status \
-H "Authorization: Bearer $TOKEN"
{
"success": true,
"data": {
"transcriptionAvailable": true,
"ttsAvailable": true,
"voiceReady": true,
"transcriptionMode": "groq-cloud",
"ttsEngine": "system-macos"
}
}
transcriptionMode
groq-cloud
local-whisper
ttsEngine
system-windows
system-macos
system-linux
User speaks
"What is the weather in Kathmandu?"
│
▼
Whisper
(Groq / whisper.cpp)
│
▼
"What is the weather in Kathmandu?"
│
▼
AiOrchestrator.chat()
├── Session History
├── Long-Term Memory
├── RAG Context
└── Tool Calling
│
▼
WeatherTool("Kathmandu")
│
▼
"The weather in Kathmandu is
22°C and clear."
│
├──────────────► Browser (SSE)
│
└──────────────► Text-to-Speech
Nothing in the AI pipeline changes.
Voice simply wraps the architecture built during Phases 1–4.
Phase 6 introduced the Agent System, allowing Jarvis to plan and execute multi-step tasks autonomously.
Phase 7 brings a complete web interface over everything we've built.
The backend is now complete.
Phases 1–6 are merged, tested, and production-ready.
Jarvis can now hear.
Jarvis can now speak.
Jarvis is open source under the Apache 2.0 License.
Current contributor-friendly issues include:
#69 CLI voice commands (voice, listen, speak)
New Voice integration tests
GitHub:
github.com/sujankim/jarvis-ai-platform
Your AI. Your Data. Your Machine.