"Building an HSK Speaking Test AI: Real-time Tone Grading with Gemini

I built a free Mandarin speaking assessment tool that grades tone + grammar in real time. Here's the engineering behind it.

HSK (Chinese proficiency test) has a speaking component (HSKK), but most learners can't self-assess their level. Online tutors are expensive. Generic AI conversation tools don't grade tones.

So I built ToneTutor: a 3-minute spoken-HSK test that estimates your speaking level and identifies weak points.

Frontend:

Backend:

The Challenge:

Web Audio API records as WebM. Gemini expects LINEAR16 (WAV). iOS Safari doesn't support WebM. So:

python
async def grade_session(transcript: str):
    prompt = """
    Rate this Mandarin response on HSK 1-6 scale.
    Assess: tone accuracy, grammar, vocabulary range.
    Provide: level estimate + weak points.
    """
    response = await gemini.generate_content(prompt, stream=True)
    return parse_hsk_level(response)

Results

- 3-min test
- Real-time feedback
- Shareable HSK score card
- Free (limited sessions)

Open source coming soon. Built because I'm a native speaker + voice actor frustrated with generic tools.

Try it: tonetutor.tefusiang.com (free for 3 sessions)

Curious about the speech-to-text pipeline or tone grading logic? Ask below.