{"slug": "my-japanese-tutor-agent-works-except-its-brain-calls-everything-a-particle", "title": "My Japanese Tutor Agent Works — Except Its Brain Calls Everything a Particle", "summary": "A developer built Kotoba-Tutor, a voice-first Japanese tutoring agent that chains faster-whisper transcription, a local Swallow-8B correction model, pitch-accent scoring, and a cross-session mistake store on a single RTX 3060. Pre-registered evaluation on 300 script-injected Tatoeba sentences found the correction model collapsed to one error type, predicting \"particle\" on 285 of 300 rows with zero recall on conjugations, politeness, and word choice, and a replication on 1,288 human-verified TEC-JL sentences pushed the false-correction rate up to 0.95. A LoRA fix trained on 50k real learner corrections inverted the failure, cutting false corrections to 0.09 but dropping exact-match to 0.01, while a later run with 10k correct examples and 4x loss on changed tokens reached 0.17 false-correction and 0.03 exact-match.", "body_md": "*I built a voice-first Japanese tutor that listens, corrects, scores pitch accent, and remembers your mistakes. The pipeline works end to end. The evaluation says the correcting brain collapses to one error type out of four. Publishing the failure is the point.*\n\nScope note, first paragraph as promised: 300 sentences from Tatoeba (CC-BY) with script-injected errors — no fluent reader checked any label, disclosed throughout; one local Swallow-8B model, greedy decoding; simulated mistake histories, never real learners; no LLM judge anywhere.\n\nKotoba-Tutor is a LangGraph agent: push-to-talk CLI → faster-whisper transcription → Swallow-8B correction analysis (structured JSON) → pitch-accent scoring with my earlier Akusento model → short explanation → a cross-session store counting your mistake types → a scheduler drilling your weakest one. The explanation is spoken back with a local TTS voice. Everything runs on one RTX 3060; a VRAM spike up front (M0) decided the layout — LLM plus ASR on GPU, TTS on CPU — before anything else was built.\n\nThe loop genuinely works. I spoke an error sentence through TTS, fed the audio in: whisper transcribed it, Swallow flagged が→は with the exact gold restoration, accent scored 0.5, and the explanation came back as audio. That part is real.\n\nWithout a fluent checker I couldn't trust human labels, so I inverted the problem: take 150 correct Tatoeba sentences as gold, inject one error each into 150 more (37 particles, 37 conjugations, 37 politenesses, 39 word choices) with a morphological script. Every gold tag is exact because the script put it there. The limitation is stated in the repo: some injections are defensible variants, which biases *against* the tutor. All 300 are on LangSmith and Hugging Face.\n\nPre-registered eval, memory vs memoryless, same backend: false-correction 0.92–0.94, recall 0.92+ on particles and **0.0 on the other three types**. The confusion matrix is the whole story — the model predicts \"particle\" on 285 of 300 rows and never emits the other types. The memory reminder (your history: 12 particle mistakes…) changed nothing: a valid null. And the false-correction rate is an upper bound, because the \"correct\" gold rows are unverified crowdsource — on 4 trivially correct sentences the model passes 4/4, so it's harshness on real text plus unverified gold, not total inability.\n\nThis is the honest outcome and the reason for the pre-registration: the loop works, the brain needs a v1.1 (constrained type output or a classifier head, not free-form JSON). I'd rather ship a measured failure than an unmeasured demo.\n\nAfter this draft was queued, the corpus access came through (same-day replies from both labs — thank you Komachi lab and NAIST). I rebuilt the eval on 1,288 TEC-JL sentences with dual human references: 812 verified-correct plus single-type errors, same frozen prompt, same model. False-correction went *up* to 0.95 [0.93, 0.97] — the \"unverified gold\" excuse is dead; the model flags correct Japanese ~95% of the time. The collapse replicates exactly (particle ~1,240/1,288, other recalls zero, stray types like \"structure\" leaking out). Row texts are research-only and stay off Hugging Face; only the rates are published. The v1.1 brain fix is still the next step.\n\nI trained the obvious fix — LoRA on 50k real learner corrections — and it failed in the opposite direction: false-correction 0.95 → 0.09, but exact-match 0.14 → 0.01. The adapter copies the input nearly every time. Of course it does: every training example was has_error:true with tiny edits, so copying minimizes loss, and it never saw a correct sentence. Two measured failure modes now bracket the problem (trigger-happy base, copy-everything adapter); the fix needs correct examples plus loss that weights changed tokens. That experiment is queued, not claimed.\n\nRan it: 10k correct examples plus 4x loss on changed tokens. False-correction 0.95 → 0.17, exact-match 0.01 → 0.03 — passes my pre-registered bar on paper, but the model mostly learned to flag nothing at all. Three adapters, three policies (flag everything, copy everything, flag nothing), none of which corrects. The correction problem is harder than the plumbing around it.\n\nHuman-corrected gold (TEC-JL / NAIST Lang-8, both request-gated) is the v1.1 eval path; the v1.1 brain fix; the owner's recorded demo session. Code, data, and the pre-registration are linked below.", "url": "https://wpnews.pro/news/my-japanese-tutor-agent-works-except-its-brain-calls-everything-a-particle", "canonical_source": "https://dev.to/raihan-js/my-japanese-tutor-agent-works-except-its-brain-calls-everything-a-particle-3ohj", "published_at": "2026-10-21 14:30:59+00:00", "updated_at": "2026-10-21 14:40:05.463363+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools", "natural-language-processing"], "entities": ["Kotoba-Tutor", "Swallow-8B", "faster-whisper", "LangGraph", "Tatoeba", "TEC-JL", "Komachi lab", "NAIST"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-japanese-tutor-agent-works-except-its-brain-calls-everything-a-particle", "markdown": "https://wpnews.pro/news/my-japanese-tutor-agent-works-except-its-brain-calls-everything-a-particle.md", "text": "https://wpnews.pro/news/my-japanese-tutor-agent-works-except-its-brain-calls-everything-a-particle.txt", "jsonld": "https://wpnews.pro/news/my-japanese-tutor-agent-works-except-its-brain-calls-everything-a-particle.jsonld"}}