{"slug": "apple-quietly-shipped-everything-you-need-to-build-a-real-time-translator-so-i", "title": "Apple quietly shipped everything you need to build a real-time translator — so I built one", "summary": "A developer built Wakaru, a macOS menu bar app that provides real-time on-device captioning and translation for meetings, using Apple's SpeechAnalyzer, Translation framework, and FoundationModels. The app captures system audio via ScreenCaptureKit and displays subtitles in a transparent overlay, avoiding cloud processing for privacy. The developer shared four key technical gotchas, including sample rate matching and handling retroactive text revisions from partial speech recognition results.", "body_md": "I work at a German company. Meetings are in German, sometimes English — and even though I speak both, there are moments in fast meetings where I zone out for two seconds and think: *wait, what did they just say?*\n\nI couldn't use any of the existing captioning tools, because they all pipe your meeting audio to a cloud server. Sending confidential work calls to a third party was a non-starter.\n\nThen I realized something: **macOS 26 quietly shipped every building block you need for a real-time translator.** On-device speech recognition (SpeechAnalyzer), on-device machine translation (Translation framework), and an on-device LLM (FoundationModels / Apple Intelligence). No servers, no API keys, no per-minute fees.\n\nSo I built [ Wakaru](https://apps.apple.com/us/app/wakaru-live-caption-translate/id6796070776) — a menu bar app that turns\n\nThis post covers how it works and the four gotchas that cost me the most time. If you're planning to build anything on macOS 26's new speech or translation APIs, this might save you a few days.\n\nMy first prototype was Electron. It worked — but transcription + translation took about **2 seconds per sentence**. For subtitles, 2 seconds might not sound like a lot, but it is: by the time the caption showed up, the conversation had moved on.\n\nWhen I saw that macOS 26 had the entire pipeline available natively and on-device, I rebuilt it in Swift. The difference was dramatic — captions now appear while the sentence is still being spoken.\n\n(The Electron version wasn't wasted, though. After a lot of tuning it got reasonably fast, and I'm planning to release it for Windows soon.)\n\n```\nScreenCaptureKit ──▶ SpeechAnalyzer ──▶ Translation framework ──▶ Subtitle overlay\n (system audio)      (speech-to-text)    or FoundationModels        (NSPanel)\n                                          (translation)\n```\n\nThe key decision: capture **system audio** (what the Mac is playing) instead of the microphone. That's what makes Wakaru app-agnostic — it doesn't integrate with Zoom or Teams; it doesn't need to know they exist.\n\nThe subtitles are drawn on a borderless, transparent, click-through `NSPanel`\n\n, so you can click straight through the captions to whatever is underneath.\n\nThe naive version of this app is a window where original text and translations pile up like a chat log. That's easy to build — and unusable as subtitles.\n\nWhat I ended up with:\n\n`text.count / 7 + 2`\n\nseconds (clamped to 4–12s). Short interjections vanish quickly; long sentences stay until you can actually finish them.\n\n``` js\nlet lifetime = min(max(4, Double(text.count) / 7 + 2), 12)\n```\n\nSpeechAnalyzer prefers 16 kHz audio. My first version captured at 48 kHz and resampled with `AVAudioConverter`\n\n— and the first caption took *seconds* to appear. The converter buffers audio internally before it emits anything.\n\nThe fix: ScreenCaptureKit lets you pick the sample rate at capture time.\n\n``` js\nlet cfg = SCStreamConfiguration()\ncfg.capturesAudio = true\ncfg.sampleRate = 16_000   // match SpeechAnalyzer's preferred format\ncfg.channelCount = 1\n```\n\nCapture at 16 kHz mono from the start and the only conversion left is Float32 → Int16, sample by sample. The latency disappeared.\n\nEven after fixing the sample rate, captions were still sluggish. It turned out to be three *separate* problems:\n\n`.fastResults`\n\nin `reportingOptions`\n\nto get them as they happen. For live captions this is non-negotiable.`prepareToAnalyze`\n\nwhen the user hits start, and recognition is instant from the first word.`finalizeAndFinishThroughEndOfInput()`\n\ndrains the entire audio backlog before returning — a stop/restart (e.g. switching languages) took seconds. For subtitles you don't care about queued audio, so use `cancelAndFinishNow()`\n\n.This was the biggest trap of all.\n\nSpeechAnalyzer's partial results don't just grow at the end. **Text you already displayed gets rewritten retroactively** — filler words (\"uh, uh\") get collapsed, words get swapped, punctuation appears late.\n\nWakaru cuts completed sentences out of the growing transcript and translates each one. That means it has to remember *where the already-translated part ends*. If you store that boundary as a character offset, it silently drifts every time the recognizer rewrites history. The symptoms: the same sentence gets translated twice, or fragments go missing.\n\nThe fix: **stop trusting positions, and use content as a bookmark.** Wakaru remembers the last few words of the most recently committed sentence, and on every update, *searches for that anchor* in the rewritten text to re-derive the boundary.\n\nOne wrinkle remains: if the speaker literally says the same thing twice (\"Thank you. Thank you.\"), the anchor appears in two places. So the rule is \"pick the occurrence *closest to the previous boundary estimate*\" — the character offset survives, demoted from source-of-truth to tie-breaker.\n\nThis one surprised me the most. `TranslationSession`\n\ncan only be obtained inside SwiftUI's `.translationTask`\n\nview modifier. There is no \"just give me a session\" API. Wakaru is a menu bar app — there was nowhere natural to put it.\n\nThe workaround: the subtitle overlay is a SwiftUI view anyway, so I attached an invisible `.translationTask`\n\nto it. It receives the session and hands it to a hub object that the rest of the app calls into.\n\n```\n.translationTask(hub.configuration) { session in\n    await hub.serve(session)   // publish the session to the pipeline\n}\n```\n\nBonus trap: if you stop and restart with the *same* language pair, the new configuration compares equal to the old one and the task never restarts — you have to call `invalidate()`\n\nexplicitly.\n\nOn Apple Intelligence Macs, Wakaru has a high-accuracy mode that translates with the on-device LLM instead of the NMT model, passing the previous 3 sentences as context — pronouns, idioms, and short replies come out much more natural.\n\nBut a ~3B on-device model in a real-time loop needs guardrails:\n\nThe rule that matters: **the LLM is never allowed to stall the captions.** Every failure mode falls back per-sentence to the standard engine.\n\nWakaru runs on macOS 26+ (Apple silicon). It's free for 14 days (and stays free for 1 hour/day after that), with a one-time purchase for unlimited time — no subscription:\n\nHappy to answer any questions about the implementation in the comments! 👇", "url": "https://wpnews.pro/news/apple-quietly-shipped-everything-you-need-to-build-a-real-time-translator-so-i", "canonical_source": "https://dev.to/toffy/apple-quietly-shipped-everything-you-need-to-build-a-real-time-translator-so-i-built-one-9ce", "published_at": "2026-08-11 20:25:27+00:00", "updated_at": "2026-08-11 20:46:50.104146+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "developer-tools", "ai-products"], "entities": ["Apple", "Wakaru", "SpeechAnalyzer", "Translation framework", "FoundationModels", "ScreenCaptureKit", "macOS 26"], "alternates": {"html": "https://wpnews.pro/news/apple-quietly-shipped-everything-you-need-to-build-a-real-time-translator-so-i", "markdown": "https://wpnews.pro/news/apple-quietly-shipped-everything-you-need-to-build-a-real-time-translator-so-i.md", "text": "https://wpnews.pro/news/apple-quietly-shipped-everything-you-need-to-build-a-real-time-translator-so-i.txt", "jsonld": "https://wpnews.pro/news/apple-quietly-shipped-everything-you-need-to-build-a-real-time-translator-so-i.jsonld"}}