[AI in Practice] Gemini 3.8 Flash TTS Launch: I built a "Learn Japanese with MVs" Web App and burned through my daily quota.

A developer built a Japanese-language learning web app that uses Google's newly generally available Gemini 3.8 Flash TTS and Flash-Lite TTS models to generate sentence-by-sentence pronunciation audio, hitting the Tier 1 quota of 100 requests per day. The writeup details the new interactions and voices API structure, custom voice limits of 200 per project, WAV 16-bit PCM mono 24 kHz output, and regional restrictions on voice replication, while noting the app's lyrics-sourcing approach raises copyright considerations.

Every time I see a new Gemini feature, my first thought is "Can I connect it to my LINE Bot?" The Gemini API release notes https://ai.google.dev/gemini-api/docs/changelog from 9/22 stated that Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS are officially launched GA , and the official blog simultaneously posted Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-8-text-to-speech/ . As usual, I listed a bunch of LINE Bot ideas: bedtime stories in parents' voices, turning group chats into radio dramas, morning dual-host podcasts... Halfway through the list, I realized that what actually struck me most wasn't a bot, but the fact that " directing tone sentence-by-sentence " is particularly suitable for teaching pronunciation. So the topic changed to a Web App: As it turned out, there was nothing to complain about regarding the quality of the TTS itself; what really took time were the things surrounding it that weren't in the documentation. Two models were released in this GA: | Model | Positioning | |---|---| | gemini-3.8-flash-tts | Flagship model, emphasizes vocal performance and character creation, allows sentence-by-sentence performance control | | gemini-3.8-flash-lite-tts | Cheap, fast, suitable for high-volume generation | Compared to the previous generation, there are three things I find truly useful: voice id for repeated use. style e.g., "speak slowly, pronounce every syllable clearly" , and it supports tags like <laughs , <sigh , as well as dialogue between up to two people. The actual call structure is different from the past generate content , moving to two sets of APIs: interactions and voices : Design a voice using text description one-time, save for reuse voice = client.voices.create store=True, voice={ "model": "gemini-3.8-flash-tts", "type": "prompted", "display name": "Song Lingo Japanese Teacher", "gender": "female", "language code": "ja-JP", "prompted": {"input": "A warm, patient Japanese language teacher in her early 30s from Tokyo..."}, }, Use this voice to read a sentence, with style controlling tone and speed interaction = client.interactions.create model="gemini-3.8-flash-tts", input= { "type": "user input", "content": { "type": "text", "text": "こんにちは。今日はいい天気ですね。", "annotations": {"type": "speech metadata", "style": "speaking slowly and clearly"} , } , } , response format={"type": "audio"}, generation config={"speech config": {"voice": voice.id} }, A few technical details to know beforehand that will save a lot of trouble: | Item | Specification | |---|---| | Output Format | WAV, 16-bit PCM, mono, 24 kHz; also supports streaming | | Languages | Over 100 types, including Simplified Chinese, Traditional Chinese, Cantonese; Taiwanese was not seen | | Custom Voice Limit | 200 per project, kept for 1 year; or choose self-managed voicekey , valid for 7 days | | Voice replication Regional Restrictions | Not available in Illinois, Texas, EEA, UK, Switzerland, India | | Price | Not mentioned in announcements or docs | | Tier 1 Quota | 100 requests per day this becomes the main character later | "Enter song name, automatically crawl lyrics" is the most intuitive way, but it gets stuck in two places: finishReason: RECITATION . Then I thought of another way: Throw the YouTube MV URL directly to Gemini and ask it to transcribe the lyrics. Technically feasible, as the Gemini API accepts public YouTube URLs. But let's be clear: Obtaining lyrics in a different way doesn't make it legal. Whether crawled from a site, typed manually, or transcribed by AI from an MV, it's the same protected text. What determines the risk is how you use it. So my decision was: This is a personal learning tool; lyrics only exist in the local output/ folder, and this folder was added to .gitignore from the first commit, never appearing on GitHub. During development, I also required Claude Code to only print statistics when verifying data, never printing lyrics to the terminal or writing them to any logs. YouTube URL ─transcribe.py─▶ Lyrics and timeline ─annotate.py──▶ Phonetics, translation, word breakdown, grammar points ─speak.py────▶ Teacher demonstration normal / slow ─Next.js─────▶ Learn sentence-by-sentence with MV transcribe.py sends the YouTube URL as file data to gemini-3.8-flash , using structured output to require start/end times, original text, Hiragana for Japanese, and an "unclear" flag for each sentence. The most critical line in the prompt: If there are lyric subtitles on the MV screen, prioritize using the subtitles. Transcribing singing is much harder than speech due to accompaniment, elongated notes, and harmonies; Japanese also has homophone issues hearing "kimi" without knowing if it's "君" or "きみ" . With subtitles, Gemini looks at the screen and listens simultaneously, greatly improving accuracy. I tested with three songs: | Song | Language | Lyrics Source | Sentence Count Unique | |---|---|---|---| | Yuuri 〈Betelgeuse〉 | Japanese | Audio | 40 23 | | Yuuri 〈Christmas Eve〉 | Japanese | Screen Subtitles | 45 35 | | Take That 〈Back for Good〉 | English | Audio | 37 26 | The structure was complete: no empty sentences, no backward time, no continuous repetition of the same sentence a common symptom of model errors . The only thing that made me uneasy was that every sentence was marked as "certain" , even for the two based on audio. The model is too confident; I'll fix this later with another method. annotate.py adds Traditional Chinese translation, word-by-word breakdown reading, POS, meaning , a grammar point, and a pronunciation tip to each sentence. Choruses repeat, so Gemini is only called for unique sentences , saving about 30-40%. Phonetics is the most interesting part of this section. My original idea was to use existing libraries, which are more reliable than LLMs, but both failed: ha , when the correct pronunciation is wa . Just looking at Hiragana, it can't tell if "は" is a particle. gamsahapnida . The official Revised Romanization follows actual pronunciation, which should be gamsahamnida ; this package doesn't handle nasalization. The final division of labor: Reason and Solution : Libraries are good at "deterministic conversion" but bad at "judgment requiring context"; LLMs are the opposite. Handing the parts requiring understanding segmentation, POS to the LLM and the deterministic parts Kana to Romaji to the program uses both where they excel. As mentioned, the transcription results marked everything as "certain," which isn't very believable. My fix: Gemini already gave a full-sentence reading during transcription, and then gave individual word readings during annotation. These two were generated separately; sentences that don't match are likely Kanji misreadings. In practice, each of the two Japanese songs had 1 mismatch 1/23, 1/35 . The program adds needs review to them, so the webpage can remind the user to proofread. This trick doesn't cost any extra API calls; it just compares two existing results. speak.py designs a teacher for each language, e.g., for Japanese: "A gentle and patient Japanese teacher from Tokyo in her early 30s." Created once, voice id saved for reuse. Two audio clips per sentence: | | Normal Speed | Slow Speed | |---|---|---| | Length | 2.5–6.1s, avg 4.3s | 4.4–9.7s, avg 7.1s | | Slow / Normal | | At least 1.24x, up to 2.34x | Slow speed relies solely on a style description without adjusting playback speed, and it's "speaking slowly and clearly," not stretching a normal speed clip. This is what impressed me most about this TTS. The webpage uses Next.js, with the server-side reading data directly from ../output/ . The screen shows the embedded MV on the left, teaching cards below Kana/Romaji/Translation/Breakdown/Grammar/Tips above Kanji, plus "Teacher," "Slow," "Original" buttons ; the right side is the sentence list, auto-switching with the MV. Upon running create-next-app , I encountered something interesting. The generated AGENTS.md wrote in the first line: This is NOT the Next.js you know Meaning this version has breaking changes, and agents should read node modules/next/dist/docs/ before coding. Claude Code followed suit; params became Promises, and RouteContext types for route handlers were found there. Documentation written specifically for AI by frameworks is a practice I think will become increasingly common. This is the most important constraint of the project, so I'll mention it first. gemini-3.8-flash-tts in Tier 1 has 100 requests per day . A song with 30+ sentences, each with normal and slow versions, takes about 50–70 requests. At this rate, only one or two songs can be processed per day. So I changed the batch generation in speak.py to generate only when the user clicks play , then cache it: | Action | Consumes Gemini? | |---|---| | Viewing lyrics, translation, breakdown | No | | Playing MV, clicking "Original" | No, that's YouTube | | Playing cached demo | No | | First time playing a demo | 1 TTS call Normal and Slow counted separately | | Adding new song | ~2 Flash calls, no TTS | Audio files are named using the hash of the sentence content. Repeated choruses naturally share the same audio. Quota is only spent on sentences actually practiced. While batch generating for the second song, progress stopped at 56/70. The status was strange: Python process alive, CPU 0%, 4 workers waiting on Google servers. First guess was no timeout, so I added 60s timeout and retries. After restarting, only 1 segment was added in ten minutes, and this time there wasn't even a network connection . Turning off SDK auto-retry and looking at the response revealed the answer: 429 Rate limit exceeded for model gemini-3.8-flash-tts limit: 100 requests per day on Tier 1 . Please retry in 7h12m16s retry-after: 25936 Daily quota was exhausted, and the SDK, upon receiving a 429, obediently waited 25,936 seconds to retry according to Retry-After . From the outside, it looked like a program that consumed no CPU, had no network, and would never end. Worse, my initial attempt to use HttpRetryOptions attempts=1 to disable retries failed completely. The reason is that the interactions API uses a different HTTP client in the SDK module name gaos , which ignores HttpRetryOptions . I had to change its own config: client = genai.Client This API doesn't respect HttpRetryOptions; if not disabled, it sleeps for hours on daily quota limits client.interactions.sdk configuration.retry config.max retries = 0 Reason and Solution : "Respecting Retry-After " is good for per-minute limits but a disaster for daily quotas. Now the program handles retries itself: transient errors retry up to 3 times; if a 429 contains "per day," all workers stop immediately, print the wait time, and keep existing audio files. This was the most expensive lesson. After switching to "generate on play," the work moved from Python to Next.js server-side using fetch for REST. Python version: audio = base64.b64decode interaction.output audio.data The TS version followed suit with json.output audio?.data . The problem: The output audio field doesn't exist in the REST response at all. Checking the SDK source code revealed that output audio is a convenience field calculated by the Python SDK in a pydantic validator: it looks for a step with type: "model output" in the steps array, then extracts the item with type: "audio" from that step's content . The real REST response looks like this: steps → { type: "model output", content → { type: "audio", data: "<base64 " } } So this happened: Within about an hour, the 100-request quota was exhausted, and not a single new file was in output/audio/ . To make matters worse, when this code went live, the daily quota was already exhausted, so the success format couldn't be verified. Claude Code noted "success path unverified" in the report, but we let it go live anyway. Reason and Solution : Fixed in two layers. steps , then the old outputs format, and finally output audio . No quota to verify via API, so I fed the same mock response to the Python SDK parser and the new TS function to ensure consistency. Later, when a quota slot opened, the first English audio was generated and saved 5.04s, 24 kHz , finally verifying it. Retry-After time. js // A failed generation may still have been billed, so don't let repeated clicks or the // browser's parallel range requests retry it: replay the error for a while instead. const recentFailures = new Map<string, { error: TtsError; until: number } ; What this bug taught me: Failure doesn't mean no cost. As long as the request reaches the model and the model runs, even if your program fails at the last parsing step, the bill still counts. Programs that consume quota must see a real success response before going live, and failure paths should default to "this might have already been billed." .env Next.js first API call returned: 400 API key not valid. Please pass a valid API key. The same key worked fine in Python. Checking the .env format length and start/end chars only revealed my key was GEMINI API KEY="..." with double quotes . Python's python-dotenv automatically strips quotes; the Node-side reader didn't, sending the quotes as part of the key. Also caught another issue: Gemini's error response is sometimes an array { "error": ... } instead of an object, so the original program couldn't even read the error message, showing a blank "400:" on screen. Reason and Solution : Changed .env reading logic to match python-dotenv allow export prefix, strip quotes , and handled both error formats. A small thing, but these "one side handles it for you" differences are easy to miss when sharing a config file between languages. Transcription errors are inevitable, so I added a proofreading interface: edit text, reading, or translation directly, or click "✓ OK"; after editing text, "Re-analyze whole song" reruns annotate.py 1 Flash request while preserving manual edits. Before starting, I found a problem: Audio files were named by "index of unique sentence," e.g., 028 normal.wav . If a lyric line is edited, subsequent sentence numbers might shift, causing existing audio to map to the wrong sentence. Reason and Solution : Changed to naming by the hash of the sentence content: php def clip name text: str, speed: str - str: """Clips are keyed by line content so editing a lyric only invalidates that line's audio.""" return f"{hashlib.sha1 text.encode .hexdigest :16 } {speed}.wav" Editing one line only invalidates that line's audio; others are unaffected. All 103 generated audio files were moved to new filenames, so no quota was wasted. Python and TS both calculated the hash for the same Japanese segment to ensure consistency before going live. Found some videos wouldn't play. YouTube returns error 101 or 150 when "owner does not allow playback on other sites." Now when these errors are detected, the video area swaps to a thumbnail with an explanation and an "Open on YouTube start from here " button; "Original" also opens a new tab at that timestamp. Teacher demos and teaching cards remain available. A small pitfall with thumbnails: High-res maxresdefault.jpg isn't available for every video, but YouTube doesn't return 404; it returns a 120x90 gray default image , so <img onError doesn't trigger, resulting in a large gray area. Reason and Solution : Check image width after loading; if under 121 pixels, swap to the guaranteed hqdefault.jpg . The whole project was done with Claude Code, from checking changelogs and discussing product direction to coding and deployment planning. A few things I think we did right: There were mistakes too, as mentioned in the pitfalls: | | Number | |---|---| | commit | 11 | | Songs added | 3 2 Japanese, 1 English | | Demo clips generated | 104 | | Single demo generation time | ~5s | | Adding a new song | ~48s Transcribe 14s, Analyze 34s | | API usage per song | ~2 Flash calls, demos generated on demand | Current features: Currently only runs locally. Plan to deploy to GCP: uv . SDK convenience fields are not part of the API. output audio is Python-only. Check real responses when switching languages or using REST. "Respecting Retry-After" depends on the limit type. Good for per-minute, bad for daily. Fail fast and explain why for quota errors. Failed requests might have been billed. Assume cost on failure for paid operations and prevent retry storms. A 60s failure cache would have saved a day's quota. Use LLMs and deterministic programs where they excel. Gemini for segmentation/POS, pykakasi for Kana-to-Romaji. Cross-check independent results to find errors for free. Think through copyright before the first line of code. Different acquisition methods don't change legality. Use .gitignore , stats-only verification, and IAP to keep it a personal tool. Code at kkdai/song-lingo https://github.com/kkdai/song-lingo lyrics data not in repo . Official docs for Speech generation https://ai.google.dev/gemini-api/docs/speech-generation and Voice design https://ai.google.dev/gemini-api/docs/voice-design . Reminder for REST users: audio is in steps .content , not output audio .