[AI in Practice] Shadowing Assessment with Gemini 3.5 Transcribe: Let Song Lingo Listen to You Read Lyrics

In the first post, I used Gemini 3.8 Flash TTS to create Song Lingo: paste a YouTube MV URL, Gemini transcribes the lyrics, adds furigana, translation, and grammar, and then a teacher designed with voice design demonstrates pronunciation sentence by sentence. In the second post, I deployed it to Cloud Run and locked it with IAP so only I can use it.

After deployment, I opened a roadmap on GitHub with three items: mobile layout, shadowing score, and flashcards. The mobile layout is finished; this post is about the second item.

The teacher demonstrates, but I never know if I'm pronouncing it correctly. Listening to the teacher ten times is not as good as saying it once and being told where you're wrong. This is a key step in Song Lingo's evolution from "listening to songs and reading lyrics" to "practicing pronunciation."

Google released Gemini 3.5 Transcribe on 8/26, which consists of two models specialized for speech-to-text:

| Model | Usage | | --- | --- | | gemini-3.5-transcribe | Recorded audio files, transcribed in full | | gemini-3.5-transcribe-live | Real-time streaming via Live API WebSocket, transcribing while speaking |

Key features: 85+ languages, with auto-detection and support for code-switching. Word-level timestamps: Each word has a start and end time (only available in the full transcription version). Speaker diarization: Identifies who is speaking each sentence. Custom vocabulary: Up to 1,000 proper nouns to improve recognition of specialized terms. Two modes: smart handles self-corrections, removes filler words, and auto-formats; verbatim transcribes word-for-word.

The official blog cites Artificial Analysis: average Word Error Rate (WER) of 2.6% for full transcription and 4.0% for streaming; compared to the previous generation Chirp 3, the time to get final results is 70% faster. Pricing has not been announced.

The actual call method is the same as TTS via the interactions API, but audio files must first be uploaded using the Files API:

Many formats are supported; audio/webm happens to be the format for Chrome recordings, and audio/m4a corresponds to AAC recorded by iPhone Safari.

| Model | Suitable for | Use in Shadowing | | --- | --- | --- | | gemini-3.5-transcribe | Transcribing after recording the whole segment | ✅ Has word-level timestamps, takes about 5s in testing | | gemini-3.5-transcribe-live | Real-time streaming | A line of lyrics is only a few seconds; transcribing after finishing is enough. Maintaining WebSockets behind Cloud Run and IAP is much more troublesome. | | gemini-3.8-flash and other multimodal models | Listening to audio directly to provide text feedback | Results vary every time, no guaranteed word-level timestamps, not suitable as the core for scoring. |

I finally chose gemini-3.5-transcribe. As a specialized speech-to-text model, the results are stable, it provides word-level info, and it only uses about 40 tokens per call, not consuming the 100-call daily quota for TTS.

The most expensive lesson from the first post was: the web version of TTS followed the Python SDK's way of reading outputaudio, but that field didn't exist in the REST response, burning a whole day's quota in an hour.

So this time, I wrote a rule directly in the roadmap issue: Verify the parsing logic with real API responses before going live.

Test audio was generated using macOS's built-in say command with sentences I wrote myself: one saying "今日は晴れです" (Today is sunny) and another intentionally saying "今日は雨です" (Today is rainy):

Then I intercepted the raw HTTP content sent and received by the SDK and printed the response structure. I discovered two things not clearly explained in the documentation that would have caused errors if followed blindly.

Documentation says transcription results are in interaction.outputtext. The actual REST response looks like this:

outputtext, like outputaudio last time, is a convenience field the Python SDK assembles from steps. Knowing this beforehand, I read directly from steps[].content[].

The startindex for the word "今日" is 0 and endindex is 6, not 0 to 2. Positions are calculated using UTF-8 bytes, where one CJK character takes 3 bytes. If sliced according to JavaScript string indices, Japanese text would be completely misaligned.