{"slug": "ai-in-practice-gemini-3-5-transcribe-real-time-transcription-and-speaker-in-a", "title": "[AI in Practice] Gemini 3.5 Transcribe: Real-time Transcription and Speaker Diarization in a macOS Meeting Translation App", "summary": "A developer detailed the addition of real-time transcription and speaker diarization to a macOS meeting translation app using Google's Gemini 3.5 models. The project, gemini-live-translate-macos, leverages ScreenCaptureKit to capture audio and the Gemini Live API for streaming transcription, but speaker diarization is only available via the non-streaming Interactions API, limiting real-time use.", "body_md": "I have a macOS App I use myself, [gemini-live-translate-macos](https://github.com/kkdai/gemini-live-translate-macos). It uses ScreenCaptureKit to directly capture audio from a specified App, eliminating the need for virtual sound cards like BlackHole. It then sends the audio to the Gemini Live API for real-time translation, outputting Traditional Chinese subtitles while playing Chinese audio. I've written two posts about the development process: [the first one](https://dev.to/agy-macos-app/) was about building it from scratch using AGY CLI, and [the second one](https://dev.to/agy-macos-app-enhance/) was about using Claude Code to take it from \"functional\" to \"user-friendly.\"\n\nThe starting point for this new addition was simple: I saw a document for \"Real-time Transcription\" added to the Live API. Since I was already connected to the Live API, I thought adding a pure transcription mode would just be a matter of changing a few parameters.\n\nHowever, after checking the documentation, I realized that Google released **two models with very similar names but very different capabilities** at once. The specific feature I actually wanted (speaker diarization) wasn't available at all on the model I originally thought it was.\n\nLet's lay out the differences first; this is the part I spent the most time figuring out:\n\n`gemini-3.5-transcribe-live` |\n`gemini-3.5-transcribe` |\n|\n|---|---|---|\n| API Used | Live API (WebSocket streaming) | Interactions API (Standard HTTP request) |\n| Usage Scenario | Transcribe while speaking | Upload the whole file after recording |\n| Speaker Diarization | Not supported | Up to 8 speakers |\n| Word-level Timestamps | Not supported | Supported |\n| Audio Length | 10 minutes per session | 1 hour (30 mins with diarization) |\n| Smart Mode |\n`SMART` available |\n`smart` is mutually exclusive with diarization |\n| Interim Subtitles | Has `interimInputTranscription`\n|\nNot applicable |\n\nThe official documentation on the Live page's limitations section is very blunt:\n\nSpeaker diarization is not supported in live streaming sessions. For speaker diarization, use the non-streaming Audio transcription endpoint.\n\nSo, \"seeing who is saying what in real-time\" is currently impossible. For speaker diarization, you must record it and send the whole thing after the meeting. This limitation determined my entire subsequent architecture.\n\nThe setup shape is different from the original translation model. `responseModalities`\n\nmust be `TEXT`\n\n, and transcription parameters are placed under `setup.inputAudioTranscription`\n\n:\n\n```\n{\n  \"setup\": {\n    \"model\": \"models/gemini-3.5-transcribe-live\",\n    \"generationConfig\": { \"responseModalities\": [\"TEXT\"] },\n    \"inputAudioTranscription\": {\n      \"languageCodes\": [],\n      \"mode\": \"SMART\"\n    },\n    \"realtimeInputConfig\": {\n      \"automaticActivityDetection\": { \"disabled\": false }\n    }\n  }\n}\n```\n\nLeaving `languageCodes`\n\nempty enables automatic language detection; filling it with BCP-47 codes like `[\"en-US\"]`\n\ngives it a language preference. `mode`\n\nhas two values: `VERBATIM`\n\nkeeps everything word-for-word, while `SMART`\n\nremoves filler words like \"uh\" and \"um\" and automatically adds formatting. For meeting minutes, I chose `SMART`\n\n, which is much cleaner to read.\n\nThere is also a `customVocabulary`\n\nwhere you can stuff technical terms, with a limit of 1000 items, though the documentation suggests staying under 100 for best results. I didn't implement this yet; I'll wait until I encounter names that are constantly misheard.\n\nThe response side adds a field that the original translation model didn't have:\n\n`interimInputTranscription`\n\n: Tentative results while speaking, which will be overwritten by subsequent content.`inputTranscription`\n\n: The finalized text when the speaker pauses or the turn ends.This distinction affects how the UI is written, which I'll discuss later.\n\nThis is the easiest place to trip up. `gemini-3.5-transcribe`\n\ndoesn't use `generateContent`\n\n; it uses the Interactions API:\n\n```\nPOST https://generativelanguage.googleapis.com/v1beta/interactions\n\n{\n  \"model\": \"gemini-3.5-transcribe\",\n  \"input\": [\n    { \"type\": \"audio\", \"uri\": \"YOUR_FILE_URI\", \"mime_type\": \"audio/wav\" }\n  ],\n  \"generation_config\": {\n    \"transcription_config\": {\n      \"language_codes\": [],\n      \"mode\": {\n        \"type\": \"verbatim\",\n        \"diarization_mode\": \"speaker\",\n        \"timestamp_granularities\": [\"word\"]\n      }\n    }\n  }\n}\n```\n\n`diarization_mode: \"speaker\"`\n\nis the switch for speaker diarization, but it can only be paired with `verbatim`\n\n. In other words: **if you want speaker diarization, you have to give up the benefits of SMART mode's filler word removal**; you can't have both.\n\nI didn't include `timestamp_granularities`\n\nat first because it wasn't in the short example in the documentation. Without it, the entire feature silently fails—a process I'll describe in the final section.\n\nAudio must first be uploaded to the Files API to get a file URI, which is then included in the request. The official documentation doesn't provide a base64 embedding example. The response looks like this:\n\n```\n{\n  \"steps\": [{\n    \"type\": \"model_output\",\n    \"content\": [{\n      \"type\": \"text\",\n      \"text\": \"Hello world\",\n      \"annotations\": [\n        { \"type\": \"word_info\", \"text\": \"Hello\", \"speaker\": \"spk_1\",\n          \"start_offset\": \"0.100s\", \"end_offset\": \"0.450s\" }\n      ]\n    }]\n  }]\n}\n```\n\nSpeaker labels are in the **word-level** annotations as IDs like `spk_1`\n\n, `spk_2`\n\n. The model doesn't know who is who. The documentation also notes a maximum of 8 speakers, and \"attribution for 3 or more people is experimental.\"\n\nThis part was simpler than I expected because the App was already parsing `inputTranscription`\n\nand `outputTranscription`\n\n(the translation model returns both for bilingual subtitles). The real change was in the mode branching.\n\nThe original connection layer determined it like this:\n\n``` js\nlet isTranslateModel = modelName.contains(\"live-translate\")\n```\n\nA boolean split into two paths. Now needing three, I switched to an enum:\n\n```\nenum LiveMode {\n    case translate // gemini-*-live-translate-*: output translated audio + bilingual subtitles\n    case transcribe // gemini-*-transcribe-live: text only\n    case general // other Live models: rely on systemInstruction for interpretation\n\n    static func from(modelName: String) -> LiveMode {\n        if modelName.contains(\"transcribe\") { return .transcribe }\n        if modelName.contains(\"live-translate\") { return .translate }\n        return .general\n    }\n}\n```\n\nI also took the opportunity to extract the setup config generation and server response parsing from `GeminiLiveConnection`\n\ninto pure functions. That file was getting a bit bloated; extracting these reduced it by over sixty lines, and the extracted parts can be tested directly.\n\nWhen parsing responses, I made a decision worth noting. Besides `inputTranscription`\n\n, the `parts`\n\nin `modelTurn`\n\nmight also carry the same text. If both are collected, the same sentence will appear twice in the subtitles and export file.\n\nTo be honest, I **haven't actually seen this happen**, but while reading the documentation and old code, I noticed both paths would lead to `didReceiveOutputTranscription`\n\n. I blocked it to avoid the risk:\n\n```\n// Content for pure transcription mode is already provided by inputTranscription.\n// Receiving text from modelTurn again would cause the same sentence to appear twice.\nif mode != .transcribe,\n   let text = part[\"text\"] as? String, !text.isEmpty {\n    events.append(.outputTranscription(text))\n}\n```\n\nThe downside of this preventive defense is that if it never actually happens, this line is dead code that no one understands. So I wrote it as a test (\"transcribe mode ignores modelTurn text\"), ensuring the behavior is locked in and the intent is documented.\n\nThis project doesn't use an Xcode project file; it uses a `build_app.sh`\n\nthat calls `swiftc`\n\nto compile all `.swift`\n\nfiles into a `.app`\n\n. No SwiftPM means no `swift test`\n\nand no XCTest.\n\nMy solution was to use the same trick again: since the extracted parts are pure functions, I wrote a `main.swift`\n\nas an assertion runner. I compile it with those pure functions into an executable and use the exit code to determine success or failure.\n\n```\nswiftc -sdk \"$SDK_PATH\" -target \"${ARCH}-apple-macos13.0\" \\\n  -o \"${BUILD_DIR}/run_tests\" \\\n  LiveSetupConfig.swift TranscriptFormatter.swift \\\n  WAVRecorder.swift AudioChunker.swift \\\n  GeminiTranscribeService.swift GeminiSummaryService.swift \\\n  Tests/main.swift\n```\n\nThe assertion function itself is less than ten lines:\n\n```\nfunc checkEqual<T: Equatable>(_ actual: T?, _ expected: T, _ name: String) {\n    if actual == expected {\n        passedCount += 1\n    } else {\n        failures.append(\"\\(name) (Actual: \\(String(describing: actual)), Expected: \\(expected))\")\n    }\n}\n```\n\nThis obviously can't compete with a real testing framework—no setup/teardown, no parallel execution, and it won't tell you which line failed. But it runs, it prevents regressions, and it doesn't require converting the whole project to SwiftPM just for testing. This phase ended with 37 assertions; after adding speaker diarization, it grew to 98.\n\n\"Adding speaker diarization\" sounds like calling one more API, but it actually involves three more tasks: saving audio to a file, uploading it, and then connecting to a completely different API. And because batch transcription might take several minutes, it can't block the original stop process.\n\nThe final data flow looks like this:\n\n``` php\ngraph TD\n    A[ScreenCaptureKit captures PCM] --> B[Gemini Live API<br/>Real-time Subtitles]\n    A --> C[WAVRecorder<br/>Synchronous writing]\n\n    D[Press Stop] --> E[Phase 1: Immediate Output]\n    E --> E1[Transcript md]\n    E --> E2[AI Meeting Summary]\n    E --> E3[Meeting Record Webpage]\n\n    D --> F[Phase 2: Background Execution]\n    F --> F1[Collect file → Split if > 30 mins]\n    F1 --> F2[Upload to Files API]\n    F2 --> F3[Interactions API Batch Transcription]\n    F3 --> F4[Delete cloud copy]\n    F4 --> F5[Write diarized.md]\n    F5 --> F6[Rerun summary with speaker info + infer names]\n    F6 --> F7[Regenerate webpage]\n    F7 --> F8[Delete local recording]\n```\n\nSplitting it into two phases was intentional. Every step in Phase 2 could fail (upload timeout, quota exhausted, response format mismatch). If it were hooked into the middle of the original process, a failure would mean losing even the basic transcript and summary. Now, Phase 1 remains untouched, and Phase 2 is just appended; if it fails, you just lose one output.\n\nThe recording format was also conveniently available: the audio sent to the Live API is already resampled to 16kHz mono 16-bit PCM. Writing that same data to disk results in valid WAV content—just add a 44-byte header, no re-encoding required.\n\nThe same defensive mindset was applied to parsing: `parseDiarized`\n\nassumes the response structure based on documentation, but if the actual format differs, I left a fallback—if word annotations aren't found, it falls back to the entire `output_text`\n\nso the transcript isn't completely empty. This fallback actually came in handy, though not in the way I expected, as I'll explain in the last section.\n\nThe audio limit for speaker diarization is 30 minutes, but meetings often last over an hour, so chunking is necessary.\n\nMy first idea was the most intuitive: fill 30 minutes for one segment and leave the rest as the last segment. While writing tests, I realized this approach had two holes.\n\nA 60-minute and 5-second meeting would be split into 30 mins, 30 mins, and 5 seconds. Sending that 5-second tail for transcription is meaningless and costs an extra upload and API call. What if I merge the tail into the previous segment? Then that segment becomes 30 mins and 5 seconds, **exceeding the API limit**, and the whole segment gets rejected.\n\n**Cause & Solution**: Switch to even splitting. Calculate how many segments are needed (round up), then distribute the total length evenly:\n\n``` js\nlet count = (usable + maxChunkBytes - 1) / maxChunkBytes\nguard count > 1 else { return [0..<usable] }\n\nvar ranges: [Range<Int>] = []\nvar start = 0\nfor index in 1...count {\n    var end = usable * index / count\n    end -= end % blockAlign // Align to 16-bit sample boundary\n    if index == count { end = usable }\n    ranges.append(start..<end)\n    start = end\n}\n```\n\n60 mins and 5 seconds becomes two segments of 30 mins and 2.5 seconds? No, that would exceed the limit. Actually, it's `ceil(3605 / 1800) = 3`\n\n, split into three segments of 20 minutes each. There will never be a tiny remainder segment, and it will never exceed the limit.\n\nThe `end -= end % blockAlign`\n\nis also necessary. 16-bit mono uses 2 bytes per sample. If you cut on an odd byte, the entire subsequent audio stream's bytes will be shifted by one, resulting in noise when played.\n\nA problem I thought of only after chunking. Each segment is an independent API call. The model doesn't know what happened in the previous segment, so `spk_1`\n\nin Segment 2 has no relation to `spk_1`\n\nin Segment 1; they could be different people.\n\nIf you just concatenate the three segments, the reader will naturally assume `spk_1`\n\nis the same person throughout. This is worse than having no speaker labels: **it gives a false sense of certainty**.\n\n**Cause & Solution**: Whenever chunking occurs, append the segment number to the ID to create a namespace:\n\n``` php\nstatic func qualifiedLabel(chunkIndex: Int, chunkCount: Int, speaker: String) -> String {\n    guard chunkCount > 1 else { return speaker }\n    return \"Seg\\(chunkIndex + 1)-\\(speaker)\"\n}\n```\n\nInsert an explanatory line at the segment boundaries during export:\n\n```\n---\n\n> Segment 2 (Speaker IDs are not continuous with the previous segment)\n```\n\nThese IDs are also used for the text sent to the AI for name inference. By sharing the same vocabulary, the model has a chance to link the same person across segments—if someone is called \"Evan\" in both Segment 1 and Segment 3, it can map them individually rather than being forced to assume the IDs are identical.\n\nThe batch API returns **word-level** annotations, which you have to join into sentences yourself. For English, it's intuitive: join with spaces.\n\nThe problem is Chinese. Gemini's Chinese tokens joined with spaces look like \"Ni hao shi jie wo men jin tian\" (Hello world we today), looking like a word segmentation exercise.\n\n**Cause & Solution**: When joining characters, check the properties of the characters on both sides. If either side is CJK (Chinese, Japanese, Korean), don't add a space; also, don't add a space before punctuation:\n\n```\nprivate static func needsSpace(after previous: Character, before next: Character) -> Bool {\n    if isCJK(previous) || isCJK(next) { return false }\n    if next.isPunctuation { return false }\n    return true\n}\n```\n\n`isCJK`\n\nchecks Unicode blocks, covering CJK Unified Ideographs, Kana, Hangul, and full-width characters.\n\nThe punctuation rule was added later. Originally I only blocked CJK, but testing `[\"Hello\", \",\", \"world\"]`\n\nrevealed it became `Hello , world`\n\n. You'd never notice this without writing tests because it doesn't \"break\"—it's just a bit ugly.\n\nPhase 1 generates an AI meeting summary after stopping. Phase 2 generates another one after getting the diarized transcript (this time with speaker info, so the \"Assignee\" field in action items can be filled).\n\nNormally, Phase 2 is definitely slower—it has to upload over 100MB of audio and wait for transcription. But \"normally slower\" isn't a guarantee. If the Phase 1 summary API happens to hang and retry, while the Phase 2 audio is only one minute long and finishes quickly, the order will reverse. The late-returning old summary from Phase 1 would overwrite the diarized result.\n\n**Cause & Solution**: Phase 2 waits for Phase 1 to finish before writing:\n\n```\n// Wait for Phase 1 summary to land, otherwise it might return after us and overwrite the diarized result\nawait minutesTask?.value\nguard !Task.isCancelled else { return }\n```\n\nA one-line fix, but you have to first realize that \"these two things actually have no guaranteed order.\" This kind of race almost never appears in testing; it only happens on a day with particularly bad network, leaving the user with a confusingly reverted meeting record.\n\nWhile writing tests for the WAV header, I followed TDD rules: write the test first, create a stub returning an empty `Data()`\n\n, and run it to see it fail. Instead of failing, the entire test program crashed:\n\n```\nSwift/arm64e-apple-macos.swiftinterface:41299: Fatal error:\nUnsafeRawBufferPointer.load out of bounds\nTrace/BPT trap: 5\n```\n\nMy test was trying to read the 24th byte to check if the sample rate was 16000, but the stub returned empty `Data`\n\n, causing an out-of-bounds read.\n\n**Cause & Solution**: Zero-pad before reading:\n\n``` js\nlet produced = WAVRecorder.header(dataByteCount: 64000)\ncheckEqual(produced.count, 44, \"WAV header is 44 bytes\")\n\n// Pad with zeros if length is insufficient, so subsequent fields report failure instead of crashing the test\nlet header = produced + Data(repeating: 0, count: max(0, 44 - produced.count))\n```\n\nThis is a small thing, but it highlighted something I usually ignore: **the test program itself must be resilient to the object under test being completely broken.** If a test ends in a crash during the RED phase instead of reporting a failure, you only know \"something broke,\" not which of the twelve fields were wrong. After padding with zeros, a single run lists all twelve expected values, allowing me to implement them by following a list rather than guessing.\n\nThere's a structural difference between real-time transcription and translation modes: translation mode accumulates words into the \"current sentence\" and pushes to history only when punctuation is reached; transcription mode's `inputTranscription`\n\nis a finalized sentence that goes straight to history.\n\nThe original auto-scroll was written like this:\n\n```\nproxy.scrollTo(\"currentLine\", anchor: .bottom)\n```\n\nThe ID `currentLine`\n\nwas attached to the display block for the \"current sentence.\" In transcription mode, the sentence goes to history as soon as it's finalized, and that block immediately disappears—the scroll target no longer exists, so the screen stays put.\n\n**Cause & Solution**: Use a bottom anchor that always exists:\n\n```\nColor.clear\n    .frame(height: 1)\n    .id(\"bottomAnchor\")\n```\n\nThis bug had no error message or crash; the \"new feature just wouldn't scroll,\" and it only became apparent when there was enough content to exceed the screen. I found it by reading the view's conditional branches, not by running it.\n\nOnce you have `spk_1`\n\nand `spk_2`\n\n, the natural desire is to replace the IDs with real names. This is actually feasible—meetings often contain clues like \"Evan, how's the progress on your end?\" or \"I'm Sarah, in charge of frontend.\" If you give the tagged transcript to the model, it can make the connection.\n\nBut this is also where hallucinations are most likely. Models are happy to \"infer\" a name from tone, job content, or speaking frequency and present it with the same confidence as a fact in the meeting minutes.\n\n**Cause & Solution**: Three methods combined.\n\nFirst, the schema explicitly allows null and requires evidence:\n\n```\nproperties[\"speakers\"] = [\n    \"type\": \"ARRAY\",\n    \"items\": [\n        \"type\": \"OBJECT\",\n        \"properties\": [\n            \"label\": [\"type\": \"STRING\"],\n            // Must return null if no clues are found, rather than forcing a name\n            \"name\": [\"type\": \"STRING\", \"nullable\": true],\n            \"evidence\": [\"type\": \"STRING\", \"nullable\": true]\n        ],\n        \"required\": [\"label\"]\n    ]\n]\n```\n\nSecond, the prompt sets strict rules:\n\nOnly fill in the name if there is a clear address, roll call, or self-introduction in the transcript or meeting notes. Explain in 'evidence' which sentence led to this conclusion. Do not speculate on names based on tone, job content, or speaking frequency. If no clear clues are found, both name and evidence must return null.\n\nThird, **remove the confidence score**. I originally designed a `confidence`\n\nfield but later removed it. The reason is that a model's self-assessment of confidence is inherently unreliable, and `evidence`\n\nalready fully serves this role: if there's evidence, the inference succeeded; if not, it didn't. An extra confidence score just makes people think it's more credible than it is—\"It says 0.7 here, so it's probably 70% accurate\"—but that 0.7 doesn't come from any real probability distribution.\n\nThe output looks like this, with evidence for successful inferences and honest admissions for failures:\n\n```\n## Participants\n- **Evan** (spk_1) — Evidence: Addressed as \"Evan, how's the progress on your end?\" in Segment 3\n- spk_2 — Insufficient clues in transcript and notes to identify name\n```\n\nKeeping the ID in parentheses is also intentional. Seeing `**Evan (spk_1)**`\n\ntells you it was inferred; if it's wrong, you can verify it yourself. If it just said `**Evan**`\n\n, it would look like an absolute fact.\n\nAnother source of clues: I also send the meeting notes the user wrote on the spot. Notes often already contain a list of participants, which significantly increases the success rate. But a safeguard is needed here—the prompt must clearly state \"Notes can only be used to map speaker names; do not treat note content as something someone said in the summary or action items,\" otherwise your own memos might turn into someone else's speech.\n\nBatch transcription inherently creates two more pieces of data than real-time streaming: the local recording file and the copy uploaded to Google. Both must have a clear disposal plan.\n\n```\n// Delete the cloud copy immediately after use, don't wait 48 hours for auto-expiration\nawait GeminiFilesUploader.delete(apiKey: apiKey, name: uploaded.name)\n```\n\nThe numbers for both phases combined:\n\n| Pure Transcription Mode | Speaker Diarization | Post-testing Fixes | |\n|---|---|---|---|\n| New Files | 2 | 4 | 0 |\n| Total Assertions | 37 | 98 | 108 |\n| Commit |\n`b0e12c5` |\n\n`486ba8f`\n\n`686899a`\n\n`1cc41c3`\n\n`GeminiLiveConnection.swift`\n\n, the file that originally did everything, was reduced by over sixty lines after setup generation and response parsing were extracted. Those two pure function modules are now guarded by over 20 tests. This was an unexpected benefit: **the decoupling done to make things testable was itself a refactoring I had put off for a long time.**\n\n**First, confirm which model has the feature you want.** I initially assumed \"Real-time Transcription\" would have speaker diarization—they're both transcription models, right? The only difference should be real-time vs. batch. I only found out otherwise after checking the docs, and the difference isn't just a parameter switch; it's architectural: for diarization, you must record, upload, use a different API, accept a 30-minute limit, and give up SMART mode. If I hadn't checked first, I would have hit a whole new subsystem while expecting to just \"change a few parameters.\"\n\n**Limitations often dictate the architecture.** Almost every design decision this time was forced by limitations: the 30-minute limit forced chunking, chunking forced ID namespaces, the slowness of batching forced two-phase export, and the mutual exclusivity of `smart`\n\nand `diarization`\n\nforced me to choose between a clean transcript and speaker labels. Checking limitations before designing is much easier than designing and then hitting a wall.\n\n**The real-time transcription half has been tested in the field.** The screenshot above is the result of feeding it a Japanese video; `gemini-3.5-transcribe-live`\n\nautomatically detected Japanese and output a Japanese transcript directly without translation, and the meeting record webpage was generated as usual after stopping. Leaving `languageCodes`\n\nempty for auto-detection actually works.\n\n**The speaker diarization half had issues during field testing**, and in a way I didn't expect: the file was generated, the transcript was correct, and the program reported no errors, but there was no speaker separation. This part deserves its own section.\n\nAfter posting, I ran speaker diarization on a real conversation. `meeting-2026-08-28-11-54-diarized.md`\n\nwas generated, the content was complete, not a word was missing, but there were no speaker labels from beginning to end—just one continuous block of text.\n\nNo error messages, the status bar showed success, and the file had everything it should. This kind of failure is the hardest to debug because it looks like success.\n\nThe first problem was that I had no evidence—the App didn't save the raw response, and the recording of that meeting was automatically deleted because \"transcription succeeded.\" To re-run it, I'd have to start another meeting with no guarantee of reproduction.\n\nSo instead of guessing what broke, I first found a way to get a raw response. macOS's built-in `say`\n\ncommand can use different voices, so I used it to synthesize a two-person conversation:\n\n```\nsay -v Alex -o a1.aiff \"Hi Samantha, did you finish the quarterly report yesterday?\"\nsay -v Samantha -o a2.aiff \"Yes Alex, I sent it to the whole team this morning.\"\nsay -v Alex -o a3.aiff \"Sure, I will look at the budget section this afternoon.\"\n\nfor f in a1 a2 a3; do afconvert -f WAVE -d LEI16@16000 -c 1 $f.aiff $f.wav; done\n```\n\nThe three segments joined together are 17 seconds long with two speakers, in the exact same format as the App's recordings (16kHz mono 16-bit). Then I used curl to go through the entire upload and transcription process, dumping the full JSON.\n\nThis step took less than five minutes, but it turned \"starting another meeting, running for ten minutes, and not being sure of reproduction\" into \"changing one field, running for ten seconds, and seeing the difference immediately.\" **A minimal reproducible example is worth the time**, especially when the original reproduction path is expensive.\n\nThe dumped response looked like this:\n\n```\n{\n  \"steps\": [{\n    \"content\": [{ \"text\": \"Hi Samantha, did you finish...\", \"type\": \"text\" }],\n    \"type\": \"model_output\"\n  }]\n}\n```\n\n811 bytes, complete transcript, **zero annotations**. So the problem wasn't my parsing; it was the request.\n\nUsing the same audio and the same uploaded file, I changed just one field in the request:\n\n| Request Content | Objects with `speaker` in response |\n|---|---|\nOnly `diarization_mode: \"speaker\"`\n|\n0 |\nRemoved `language_codes: []`\n|\n0 |\nAdded `timestamp_granularities: [\"word\"]`\n|\n41 |\n\nSpeaker IDs are attached to `word_info`\n\nannotations, and word-level annotations must be explicitly requested in the request to be returned. No request, no annotations; no annotations, no speakers.\n\nI didn't include this field originally because I copied the shortest Python example from the documentation—that example only had `type`\n\nand `diarization_mode`\n\n. The full REST example elsewhere in the documentation actually has `timestamp_granularities`\n\n, but by then I already \"knew\" how to write it and didn't look back.\n\nThe most frustrating part is that **it doesn't report an error**. The API returns 200, gives you the full transcript, and the `status`\n\nis `completed`\n\n. If it had returned an error like \"You requested speaker diarization but didn't enable word annotations,\" I would have fixed it in five minutes.\n\nAfter adding the field, annotations appeared, but they looked different from what I expected:\n\n```\n{\"text\":\"Hi,\",\"start_offset\":\"0.100s\",\"end_offset\":\"0.500s\",\"speaker\":\"spk:0\",\"type\":\"word_info\"}\n```\n\n`spk:0`\n\n. **A colon, and starting from 0.** The documentation consistently uses `spk_1`\n\n, `spk_2`\n\n.\n\nMy display logic was written like this:\n\n``` js\nif let range = speaker.range(of: \"spk_\"), let number = Int(speaker[range.upperBound...]) {\n    return \"Speaker \\(number)\"\n}\nreturn speaker // ← Fallback prints the raw ID if no match\n```\n\n`spk:0`\n\ndoesn't match `spk_`\n\n, so it fell straight into the fallback, displaying `**spk:0**:`\n\non the screen.\n\nMy tests completely missed this bug for a simple reason: **the test data was written according to the documentation.** The documentation was wrong, so the tests were wrong, and the green light told me everything was fine. This is the main thing I want to record: for external API tests, you are actually testing \"my understanding of this API,\" not the API itself. If your understanding is wrong, the test will faithfully protect that error.\n\nI didn't fix it by changing `spk_`\n\nto `spk:`\n\n; that would just be betting in a different direction. Instead, I stopped parsing the number in the ID and used the order in which the speaker first appeared in that segment:\n\n```\n/// The actual format of the ID is determined by the API (actual returns spk:0, spk:1, while official docs say spk_1),\n/// so we don't parse the number in the ID. Instead, we number them based on their first appearance in this segment.\nstatic func speakerOrder(in chunk: [DiarizedSegment]) -> [String: Int] {\n    var order: [String: Int] = [:]\n    for segment in chunk where !segment.speaker.isEmpty {\n        if order[segment.speaker] == nil {\n            order[segment.speaker] = order.count + 1\n        }\n    }\n    return order\n}\n```\n\nIt won't break even if the format changes again because it doesn't look at the format at all.\n\nI wrote this earlier and was quite proud of it:\n\n`parseDiarized`\n\nassumes the response structure based on documentation, but if the actual format differs, I left a fallback—if word annotations aren't found, it falls back to the entire`output_text`\n\nso the transcript isn't completely empty.\n\nThis fallback did work, and the effect was as expected: the user got a complete transcript with nothing lost.\n\nBut it also **hid the failure**. If I hadn't had that fallback, `-diarized.md`\n\nwould have been empty or not generated at all, and I would have known immediately that something was wrong. With it, I got a file that looked perfectly normal, just missing the feature I wanted—the very feature that was the sole reason for enabling it.\n\nI haven't fully figured out the balance here. Graceful degradation isn't wrong; the mistake is **not speaking up after degrading**. My current approach is to keep the fallback but change the status bar message when it's triggered, explicitly stating \"Speaker info not obtained this time\" instead of the usual \"Diarized transcript saved.\"\n\nRunning the full parsing with a real response, both segments spoken by Alex correctly returned to Speaker 1:\n\n```\n**Speaker 1**: Hi, Samantha. Did you finish the quarterly report yesterday?\n**Speaker 2**: Yes, Alex. I sent it to the whole team this morning. Could you review the budget section?\n**Speaker 1**: Sure. I will look at the budget section this afternoon and get back to you.\n```\n\nThe quality of the speaker diarization itself is good. Tests increased from 98 to 106, with the extra eight being regression tests for these two root causes—this time, the test data wasn't copied from the docs but clipped from real responses.\n\nThree more details learned from field testing:\n\n`generationConfig`\n\nis explicitly rejected with `Unknown parameter 'generationConfig'. Did you mean 'generation_config'?`\n\n. This is a great error message, a hundred times more useful than the silent failure above.`start_index`\n\nand `end_index`\n\n, mapping directly to positions in `content.text`\n\n. Using these to split strings is much more accurate than my heuristic for joining CJK/English words, meaning the entire `joinWords`\n\ncould be removed. I haven't done this yet.`state`\n\nof uploaded files is immediately `ACTIVE`\n\n; 17 seconds of audio didn't go through a `PROCESSING`\n\nphase. My polling logic seems redundant for short audio, but I don't know if it's needed for long audio, so I'll keep it for now.**This article thus has two endings, and I've decided to keep both.** The first part said \"Speaker diarization hasn't been verified yet; I'll update after I run it,\" and then I actually ran it, and it broke. If I had edited the first part and only kept the fixed version, this would have been a smooth \"I did X, and it worked\" post—but what actually happened was \"I did X according to the docs, it failed silently, and it took me half an hour to find out why.\" The latter is much more useful to readers.\n\nThe code is at [kkdai/gemini-live-translate-macos](https://github.com/kkdai/gemini-live-translate-macos). The two official documents are [Live transcription](https://ai.google.dev/gemini-api/docs/live-api/live-transcribe) and [Audio transcription](https://ai.google.dev/gemini-api/docs/transcribe). I recommend reading the limitations section on the speaker diarization page thoroughly before starting.", "url": "https://wpnews.pro/news/ai-in-practice-gemini-3-5-transcribe-real-time-transcription-and-speaker-in-a", "canonical_source": "https://dev.to/gde/ai-in-practice-gemini-35-transcribe-real-time-transcription-and-speaker-diarization-in-a-macos-152h", "published_at": "2026-08-28 15:53:34+00:00", "updated_at": "2026-08-28 16:20:19.780299+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-products"], "entities": ["Google", "Gemini 3.5", "gemini-live-translate-macos", "ScreenCaptureKit", "Gemini Live API", "Interactions API"], "alternates": {"html": "https://wpnews.pro/news/ai-in-practice-gemini-3-5-transcribe-real-time-transcription-and-speaker-in-a", "markdown": "https://wpnews.pro/news/ai-in-practice-gemini-3-5-transcribe-real-time-transcription-and-speaker-in-a.md", "text": "https://wpnews.pro/news/ai-in-practice-gemini-3-5-transcribe-real-time-transcription-and-speaker-in-a.txt", "jsonld": "https://wpnews.pro/news/ai-in-practice-gemini-3-5-transcribe-real-time-transcription-and-speaker-in-a.jsonld"}}