# [AI in Practice] Gemini 3.8 Flash TTS Launch: I built a "Learn Japanese with MVs" Web App and burned through my daily quota. > Source: > Published: 2026-09-26 17:25:33+00:00 Every time I see a new Gemini feature, my first thought is "Can I connect it to my LINE Bot?" The [Gemini API release notes](https://ai.google.dev/gemini-api/docs/changelog) from 9/22 stated that **Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS are officially launched (GA)**, and the official blog simultaneously posted [Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-8-text-to-speech/). As usual, I listed a bunch of LINE Bot ideas: bedtime stories in parents' voices, turning group chats into radio dramas, morning dual-host podcasts... Halfway through the list, I realized that what actually struck me most wasn't a bot, but the fact that "**directing tone sentence-by-sentence**" is particularly suitable for teaching pronunciation. So the topic changed to a Web App: As it turned out, there was nothing to complain about regarding the quality of the TTS itself; what really took time were the things surrounding it that weren't in the documentation. Two models were released in this GA: | Model | Positioning | |---|---| | `gemini-3.8-flash-tts` | Flagship model, emphasizes vocal performance and character creation, allows sentence-by-sentence performance control | | `gemini-3.8-flash-lite-tts` | Cheap, fast, suitable for high-volume generation | Compared to the previous generation, there are three things I find truly useful: `voice_id` for repeated use.`style` (e.g., "speak slowly, pronounce every syllable clearly"), and it supports tags like ``, ``, as well as dialogue between up to two people. The actual call structure is different from the past `generate_content`, moving to two sets of APIs: `interactions` and `voices`: ``` # Design a voice using text description (one-time, save for reuse) voice = client.voices.create( store=True, voice={ "model": "gemini-3.8-flash-tts", "type": "prompted", "display_name": "Song Lingo Japanese Teacher", "gender": "female", "language_code": "ja-JP", "prompted": {"input": "A warm, patient Japanese language teacher in her early 30s from Tokyo..."}, }, ) # Use this voice to read a sentence, with style controlling tone and speed interaction = client.interactions.create( model="gemini-3.8-flash-tts", input=[{ "type": "user_input", "content": [{ "type": "text", "text": "こんにちは。今日はいい天気ですね。", "annotations": [{"type": "speech_metadata", "style": "speaking slowly and clearly"}], }], }], response_format={"type": "audio"}, generation_config={"speech_config": [{"voice": voice.id}]}, ) ``` A few technical details to know beforehand that will save a lot of trouble: | Item | Specification | |---|---| | Output Format | WAV, 16-bit PCM, mono, 24 kHz; also supports streaming | | Languages | Over 100 types, including Simplified Chinese, Traditional Chinese, Cantonese; **Taiwanese was not seen** | | Custom Voice Limit | 200 per project, kept for 1 year; or choose self-managed `voicekey_` , valid for 7 days | | Voice replication Regional Restrictions | Not available in Illinois, Texas, EEA, UK, Switzerland, India | | Price | Not mentioned in announcements or docs | | **Tier 1 Quota** | **100 requests per day** (this becomes the main character later) | "Enter song name, automatically crawl lyrics" is the most intuitive way, but it gets stuck in two places: `finishReason: RECITATION`. Then I thought of another way: **Throw the YouTube MV URL directly to Gemini and ask it to transcribe the lyrics.** Technically feasible, as the Gemini API accepts public YouTube URLs. But let's be clear: **Obtaining lyrics in a different way doesn't make it legal.** Whether crawled from a site, typed manually, or transcribed by AI from an MV, it's the same protected text. What determines the risk is how you use it. So my decision was: This is a personal learning tool; lyrics only exist in the local `output/` folder, and this folder was added to `.gitignore` from the first commit, never appearing on GitHub. During development, I also required Claude Code to **only print statistics** when verifying data, never printing lyrics to the terminal or writing them to any logs. ``` YouTube URL ─transcribe.py─▶ Lyrics and timeline ─annotate.py──▶ Phonetics, translation, word breakdown, grammar points ─speak.py────▶ Teacher demonstration (normal / slow) ─Next.js─────▶ Learn sentence-by-sentence with MV ``` `transcribe.py` sends the YouTube URL as `file_data` to `gemini-3.8-flash`, using structured output to require start/end times, original text, Hiragana for Japanese, and an "unclear" flag for each sentence. The most critical line in the prompt: **If there are lyric subtitles on the MV screen, prioritize using the subtitles.** Transcribing singing is much harder than speech due to accompaniment, elongated notes, and harmonies; Japanese also has homophone issues (hearing "kimi" without knowing if it's "君" or "きみ"). With subtitles, Gemini looks at the screen and listens simultaneously, greatly improving accuracy. I tested with three songs: | Song | Language | Lyrics Source | Sentence Count (Unique) | |---|---|---|---| | Yuuri 〈Betelgeuse〉 | Japanese | Audio | 40 (23) | | Yuuri 〈Christmas Eve〉 | Japanese | Screen Subtitles | 45 (35) | | Take That 〈Back for Good〉 | English | Audio | 37 (26) | The structure was complete: no empty sentences, no backward time, no continuous repetition of the same sentence (a common symptom of model errors). The only thing that made me uneasy was that **every sentence was marked as "certain"**, even for the two based on audio. The model is too confident; I'll fix this later with another method. `annotate.py` adds Traditional Chinese translation, word-by-word breakdown (reading, POS, meaning), a grammar point, and a pronunciation tip to each sentence. Choruses repeat, so Gemini is only called for **unique sentences**, saving about 30-40%. Phonetics is the most interesting part of this section. My original idea was to use existing libraries, which are more reliable than LLMs, but both failed: `ha`, when the correct pronunciation is `wa`. Just looking at Hiragana, it can't tell if "は" is a particle.`gamsahapnida`. The official Revised Romanization follows actual pronunciation, which should be `gamsahamnida`; this package doesn't handle nasalization. The final division of labor: **Reason and Solution**: Libraries are good at "deterministic conversion" but bad at "judgment requiring context"; LLMs are the opposite. Handing the parts requiring understanding (segmentation, POS) to the LLM and the deterministic parts (Kana to Romaji) to the program uses both where they excel. As mentioned, the transcription results marked everything as "certain," which isn't very believable. My fix: Gemini already gave a full-sentence reading during transcription, and then gave individual word readings during annotation. **These two were generated separately; sentences that don't match are likely Kanji misreadings.** In practice, each of the two Japanese songs had 1 mismatch (1/23, 1/35). The program adds `needs_review` to them, so the webpage can remind the user to proofread. This trick doesn't cost any extra API calls; it just compares two existing results. `speak.py` designs a teacher for each language, e.g., for Japanese: "A gentle and patient Japanese teacher from Tokyo in her early 30s." Created once, `voice_id` saved for reuse. Two audio clips per sentence: | | Normal Speed | Slow Speed | |---|---|---| | Length | 2.5–6.1s, avg 4.3s | 4.4–9.7s, avg 7.1s | | Slow / Normal | | At least 1.24x, up to 2.34x | Slow speed relies solely on a `style` description without adjusting playback speed, and it's "speaking slowly and clearly," not stretching a normal speed clip. This is what impressed me most about this TTS. The webpage uses Next.js, with the server-side reading data directly from `../output/`. The screen shows the embedded MV on the left, teaching cards below (Kana/Romaji/Translation/Breakdown/Grammar/Tips above Kanji, plus "Teacher," "Slow," "Original" buttons); the right side is the sentence list, auto-switching with the MV. Upon running `create-next-app`, I encountered something interesting. The generated `AGENTS.md` wrote in the first line: This is NOT the Next.js you know Meaning this version has breaking changes, and agents should read `node_modules/next/dist/docs/` before coding. Claude Code followed suit; `params` became Promises, and `RouteContext` types for route handlers were found there. Documentation written specifically for AI by frameworks is a practice I think will become increasingly common. This is the most important constraint of the project, so I'll mention it first. `gemini-3.8-flash-tts` in Tier 1 has **100 requests per day**. A song with 30+ sentences, each with normal and slow versions, takes about 50–70 requests. **At this rate, only one or two songs can be processed per day.** So I changed the batch generation in `speak.py` to **generate only when the user clicks play**, then cache it: | Action | Consumes Gemini? | |---|---| | Viewing lyrics, translation, breakdown | No | | Playing MV, clicking "Original" | No, that's YouTube | | Playing cached demo | No | | **First time** playing a demo | 1 TTS call (Normal and Slow counted separately) | | Adding new song | ~2 Flash calls, no TTS | Audio files are named using the hash of the sentence content. Repeated choruses naturally share the same audio. Quota is only spent on sentences actually practiced. While batch generating for the second song, progress stopped at 56/70. The status was strange: Python process alive, CPU 0%, 4 workers waiting on Google servers. First guess was no timeout, so I added 60s timeout and retries. After restarting, only 1 segment was added in ten minutes, and this time **there wasn't even a network connection**. Turning off SDK auto-retry and looking at the response revealed the answer: ``` 429 Rate limit exceeded for model gemini-3.8-flash-tts (limit: 100 requests per day on Tier 1). Please retry in 7h12m16s retry-after: 25936 ``` Daily quota was exhausted, and the SDK, upon receiving a 429, **obediently waited 25,936 seconds to retry according to `Retry-After`**. From the outside, it looked like a program that consumed no CPU, had no network, and would never end. Worse, my initial attempt to use `HttpRetryOptions(attempts=1)` to disable retries failed completely. The reason is that the `interactions` API uses a different HTTP client in the SDK (module name `_gaos`), which ignores `HttpRetryOptions`. I had to change its own config: ``` client = genai.Client() # This API doesn't respect HttpRetryOptions; if not disabled, it sleeps for hours on daily quota limits client.interactions.sdk_configuration.retry_config.max_retries = 0 ``` **Reason and Solution**: "Respecting `Retry-After`" is good for per-minute limits but a disaster for daily quotas. Now the program handles retries itself: transient errors retry up to 3 times; if a 429 contains "per day," all workers stop immediately, print the wait time, and keep existing audio files. This was the most expensive lesson. After switching to "generate on play," the work moved from Python to Next.js server-side using `fetch` for REST. Python version: ``` audio = base64.b64decode(interaction.output_audio.data) ``` The TS version followed suit with `json.output_audio?.data`. The problem: **The `output_audio` field doesn't exist in the REST response at all.** Checking the SDK source code revealed that `output_audio` is a **convenience field calculated by the Python SDK** in a pydantic validator: it looks for a step with `type: "model_output"` in the `steps` array, then extracts the item with `type: "audio"` from that step's `content`. The real REST response looks like this: ``` steps[] → { type: "model_output", content[] → { type: "audio", data: "" } } ``` So this happened: **Within about an hour, the 100-request quota was exhausted, and not a single new file was in `output/audio/`.** To make matters worse, when this code went live, the daily quota was already exhausted, so the success format couldn't be verified. Claude Code noted "success path unverified" in the report, but we let it go live anyway. **Reason and Solution**: Fixed in two layers. `steps`, then the old `outputs` format, and finally `output_audio`. No quota to verify via API, so I fed the same mock response to the Python SDK parser and the new TS function to ensure consistency. Later, when a quota slot opened, the first English audio was generated and saved (5.04s, 24 kHz), finally verifying it.`Retry-After` time. ``` js // A failed generation may still have been billed, so don't let repeated clicks or the // browser's parallel range requests retry it: replay the error for a while instead. const recentFailures = new Map(); ``` What this bug taught me: **Failure doesn't mean no cost.** As long as the request reaches the model and the model runs, even if your program fails at the last parsing step, the bill still counts. Programs that consume quota must see a real success response before going live, and failure paths should default to "this might have already been billed." `.env` Next.js first API call returned: ``` 400 API key not valid. Please pass a valid API key. ``` The same key worked fine in Python. Checking the `.env` format (length and start/end chars only) revealed my key was `GEMINI_API_KEY="..."` **with double quotes**. Python's `python-dotenv` automatically strips quotes; the Node-side reader didn't, sending the quotes as part of the key. Also caught another issue: Gemini's error response is sometimes an array `[{ "error": ... }]` instead of an object, so the original program couldn't even read the error message, showing a blank "400:" on screen. **Reason and Solution**: Changed `.env` reading logic to match `python-dotenv` (allow `export` prefix, strip quotes), and handled both error formats. A small thing, but these "one side handles it for you" differences are easy to miss when sharing a config file between languages. Transcription errors are inevitable, so I added a proofreading interface: edit text, reading, or translation directly, or click "✓ OK"; after editing text, "Re-analyze whole song" reruns `annotate.py` (1 Flash request) while preserving manual edits. Before starting, I found a problem: Audio files were named by "index of unique sentence," e.g., `028_normal.wav`. **If a lyric line is edited, subsequent sentence numbers might shift, causing existing audio to map to the wrong sentence.** **Reason and Solution**: Changed to naming by the hash of the sentence content: ``` php def clip_name(text: str, speed: str) -> str: """Clips are keyed by line content so editing a lyric only invalidates that line's audio.""" return f"{hashlib.sha1(text.encode()).hexdigest()[:16]}_{speed}.wav" ``` Editing one line only invalidates that line's audio; others are unaffected. All 103 generated audio files were moved to new filenames, so no quota was wasted. Python and TS both calculated the hash for the same Japanese segment to ensure consistency before going live. Found some videos wouldn't play. YouTube returns error 101 or 150 when "owner does not allow playback on other sites." Now when these errors are detected, the video area swaps to a thumbnail with an explanation and an "Open on YouTube (start from here)" button; "Original" also opens a new tab at that timestamp. Teacher demos and teaching cards remain available. A small pitfall with thumbnails: High-res `maxresdefault.jpg` isn't available for every video, but YouTube **doesn't return 404; it returns a 120x90 gray default image**, so `` doesn't trigger, resulting in a large gray area. **Reason and Solution**: Check image width after loading; if under 121 pixels, swap to the guaranteed `hqdefault.jpg`. The whole project was done with Claude Code, from checking changelogs and discussing product direction to coding and deployment planning. A few things I think we did right: There were mistakes too, as mentioned in the pitfalls: | | Number | |---|---| | commit | 11 | | Songs added | 3 (2 Japanese, 1 English) | | Demo clips generated | 104 | | Single demo generation time | ~5s | | Adding a new song | ~48s (Transcribe 14s, Analyze 34s) | | API usage per song | ~2 Flash calls, demos generated on demand | Current features: Currently only runs locally. Plan to deploy to GCP: `uv`. **SDK convenience fields are not part of the API.** `output_audio` is Python-only. Check real responses when switching languages or using REST. **"Respecting Retry-After" depends on the limit type.** Good for per-minute, bad for daily. Fail fast and explain why for quota errors. **Failed requests might have been billed.** Assume cost on failure for paid operations and prevent retry storms. A 60s failure cache would have saved a day's quota. **Use LLMs and deterministic programs where they excel.** Gemini for segmentation/POS, pykakasi for Kana-to-Romaji. Cross-check independent results to find errors for free. **Think through copyright before the first line of code.** Different acquisition methods don't change legality. Use `.gitignore`, stats-only verification, and IAP to keep it a personal tool. Code at [kkdai/song-lingo](https://github.com/kkdai/song-lingo) (lyrics data not in repo). Official docs for [Speech generation](https://ai.google.dev/gemini-api/docs/speech-generation) and [Voice design](https://ai.google.dev/gemini-api/docs/voice-design). Reminder for REST users: audio is in `steps[].content[]`, not `output_audio`.