[AI in Practice] Gemini 3.5 Transcribe: Real-time Transcription and Speaker Diarization in a macOS Meeting Translation App A developer detailed the addition of real-time transcription and speaker diarization to a macOS meeting translation app using Google's Gemini 3.5 models. The project, gemini-live-translate-macos, leverages ScreenCaptureKit to capture audio and the Gemini Live API for streaming transcription, but speaker diarization is only available via the non-streaming Interactions API, limiting real-time use. I have a macOS App I use myself, gemini-live-translate-macos https://github.com/kkdai/gemini-live-translate-macos . It uses ScreenCaptureKit to directly capture audio from a specified App, eliminating the need for virtual sound cards like BlackHole. It then sends the audio to the Gemini Live API for real-time translation, outputting Traditional Chinese subtitles while playing Chinese audio. I've written two posts about the development process: the first one https://dev.to/agy-macos-app/ was about building it from scratch using AGY CLI, and the second one https://dev.to/agy-macos-app-enhance/ was about using Claude Code to take it from "functional" to "user-friendly." The starting point for this new addition was simple: I saw a document for "Real-time Transcription" added to the Live API. Since I was already connected to the Live API, I thought adding a pure transcription mode would just be a matter of changing a few parameters. However, after checking the documentation, I realized that Google released two models with very similar names but very different capabilities at once. The specific feature I actually wanted speaker diarization wasn't available at all on the model I originally thought it was. Let's lay out the differences first; this is the part I spent the most time figuring out: gemini-3.5-transcribe-live | gemini-3.5-transcribe | | |---|---|---| | API Used | Live API WebSocket streaming | Interactions API Standard HTTP request | | Usage Scenario | Transcribe while speaking | Upload the whole file after recording | | Speaker Diarization | Not supported | Up to 8 speakers | | Word-level Timestamps | Not supported | Supported | | Audio Length | 10 minutes per session | 1 hour 30 mins with diarization | | Smart Mode | SMART available | smart is mutually exclusive with diarization | | Interim Subtitles | Has interimInputTranscription | Not applicable | The official documentation on the Live page's limitations section is very blunt: Speaker diarization is not supported in live streaming sessions. For speaker diarization, use the non-streaming Audio transcription endpoint. So, "seeing who is saying what in real-time" is currently impossible. For speaker diarization, you must record it and send the whole thing after the meeting. This limitation determined my entire subsequent architecture. The setup shape is different from the original translation model. responseModalities must be TEXT , and transcription parameters are placed under setup.inputAudioTranscription : { "setup": { "model": "models/gemini-3.5-transcribe-live", "generationConfig": { "responseModalities": "TEXT" }, "inputAudioTranscription": { "languageCodes": , "mode": "SMART" }, "realtimeInputConfig": { "automaticActivityDetection": { "disabled": false } } } } Leaving languageCodes empty enables automatic language detection; filling it with BCP-47 codes like "en-US" gives it a language preference. mode has two values: VERBATIM keeps everything word-for-word, while SMART removes filler words like "uh" and "um" and automatically adds formatting. For meeting minutes, I chose SMART , which is much cleaner to read. There is also a customVocabulary where you can stuff technical terms, with a limit of 1000 items, though the documentation suggests staying under 100 for best results. I didn't implement this yet; I'll wait until I encounter names that are constantly misheard. The response side adds a field that the original translation model didn't have: interimInputTranscription : Tentative results while speaking, which will be overwritten by subsequent content. inputTranscription : The finalized text when the speaker pauses or the turn ends.This distinction affects how the UI is written, which I'll discuss later. This is the easiest place to trip up. gemini-3.5-transcribe doesn't use generateContent ; it uses the Interactions API: POST https://generativelanguage.googleapis.com/v1beta/interactions { "model": "gemini-3.5-transcribe", "input": { "type": "audio", "uri": "YOUR FILE URI", "mime type": "audio/wav" } , "generation config": { "transcription config": { "language codes": , "mode": { "type": "verbatim", "diarization mode": "speaker", "timestamp granularities": "word" } } } } diarization mode: "speaker" is the switch for speaker diarization, but it can only be paired with verbatim . In other words: if you want speaker diarization, you have to give up the benefits of SMART mode's filler word removal ; you can't have both. I didn't include timestamp granularities at first because it wasn't in the short example in the documentation. Without it, the entire feature silently fails—a process I'll describe in the final section. Audio must first be uploaded to the Files API to get a file URI, which is then included in the request. The official documentation doesn't provide a base64 embedding example. The response looks like this: { "steps": { "type": "model output", "content": { "type": "text", "text": "Hello world", "annotations": { "type": "word info", "text": "Hello", "speaker": "spk 1", "start offset": "0.100s", "end offset": "0.450s" } } } } Speaker labels are in the word-level annotations as IDs like spk 1 , spk 2 . The model doesn't know who is who. The documentation also notes a maximum of 8 speakers, and "attribution for 3 or more people is experimental." This part was simpler than I expected because the App was already parsing inputTranscription and outputTranscription the translation model returns both for bilingual subtitles . The real change was in the mode branching. The original connection layer determined it like this: js let isTranslateModel = modelName.contains "live-translate" A boolean split into two paths. Now needing three, I switched to an enum: enum LiveMode { case translate // gemini- -live-translate- : output translated audio + bilingual subtitles case transcribe // gemini- -transcribe-live: text only case general // other Live models: rely on systemInstruction for interpretation static func from modelName: String - LiveMode { if modelName.contains "transcribe" { return .transcribe } if modelName.contains "live-translate" { return .translate } return .general } } I also took the opportunity to extract the setup config generation and server response parsing from GeminiLiveConnection into pure functions. That file was getting a bit bloated; extracting these reduced it by over sixty lines, and the extracted parts can be tested directly. When parsing responses, I made a decision worth noting. Besides inputTranscription , the parts in modelTurn might also carry the same text. If both are collected, the same sentence will appear twice in the subtitles and export file. To be honest, I haven't actually seen this happen , but while reading the documentation and old code, I noticed both paths would lead to didReceiveOutputTranscription . I blocked it to avoid the risk: // Content for pure transcription mode is already provided by inputTranscription. // Receiving text from modelTurn again would cause the same sentence to appear twice. if mode = .transcribe, let text = part "text" as? String, text.isEmpty { events.append .outputTranscription text } The downside of this preventive defense is that if it never actually happens, this line is dead code that no one understands. So I wrote it as a test "transcribe mode ignores modelTurn text" , ensuring the behavior is locked in and the intent is documented. This project doesn't use an Xcode project file; it uses a build app.sh that calls swiftc to compile all .swift files into a .app . No SwiftPM means no swift test and no XCTest. My solution was to use the same trick again: since the extracted parts are pure functions, I wrote a main.swift as an assertion runner. I compile it with those pure functions into an executable and use the exit code to determine success or failure. swiftc -sdk "$SDK PATH" -target "${ARCH}-apple-macos13.0" \ -o "${BUILD DIR}/run tests" \ LiveSetupConfig.swift TranscriptFormatter.swift \ WAVRecorder.swift AudioChunker.swift \ GeminiTranscribeService.swift GeminiSummaryService.swift \ Tests/main.swift The assertion function itself is less than ten lines: func checkEqual