{"slug": "whisper-s-zero-shot-performance-on-nepali-financial-commands-is", "title": "Whisper's zero-shot performance on Nepali financial commands is", "summary": "Researchers at an unnamed institution created NepFinSpeech-403, a dataset of 403 Nepali utterances for wallet operations, and used LoRA to fine-tune OpenAI's Whisper model, boosting transaction success rate from 1.67% zero-shot to 33.33% and Devanagari numeral accuracy from 0.0% to 73.9%, while reducing word error rate from ~130% to 42.58%. The study shows that just 100 domain-specific utterances halved the zero-shot WER, with gains plateauing around 300 examples, and highlights persistent numerical errors like prefix hallucination and zero insertions/deletions, emphasizing the need for constrained decoding in financial ASR. The dataset and model weights are publicly available on GitHub.", "body_md": "# Whisper's zero-shot performance on Nepali financial commands is\n\nThe researchers tackled this by creating a specialized dataset called NepFinSpeech-403, which consists of 403 utterances covering core wallet operations like sending money, loading funds, and checking balances. What makes this interesting for anyone working on LLM agent deployment or specialized ASR (Automatic Speech Recognition) is how they used LoRA (Low-Rank Adaptation) to fine-tune the model without needing a massive compute cluster.\n\n## The impact of domain-adaptive fine-tuning\n\nThe jump in performance after applying LoRA was massive, but the most telling metric isn't actually the WER. While they managed to bring the WER down from ~130% to 42.58%, the real win is in the \"Transaction Success Rate.\"\n\n**Zero-shot Success Rate:** 1.67%**LoRA Fine-tuned Success Rate:** 33.33%**Relative improvement:**~20x gain in actual usability** Devanagari numeral accuracy:**Jumped from 0.0% to 73.9%\n\nThe study shows that for low-resource languages, general-purpose models are almost useless for high-precision tasks like banking. Even with a 42% error rate, the model went from being a complete brick to actually being able to facilitate transactions.\n\n## Data efficiency and the \"plateau\" effect\n\nOne of the most practical takeaways for developers building custom AI workflows is how much data you actually need. You don't need thousands of hours of audio to see a massive shift.\n\n1. **The 100-utterance threshold:** Just 100 domain-specific utterances were enough to cut the zero-shot WER in half.\n\n2. **The 300-utterance plateau:** Performance gains started to level off once they hit roughly 300 examples.\n\nThis is a huge hint for anyone trying to implement a practical tutorial for domain adaptation on a budget. If you're working with a niche dialect or a specific industry jargon, you can get significant results with a very small, high-quality dataset.\n\n## Why it still fails\n\nEven with the fine-tuning, the model isn't perfect. The error analysis pointed to systematic issues with how the model handles numbers. Specifically, they saw \"prefix hallucination\" and frequent zero insertions or deletions. In a financial context, if the model hallucinates a digit or misses a zero, the entire transaction is ruined. This is a classic example of why ASR for fintech requires more than just a good language model—it requires strict post-processing or constrained decoding to ensure numerical integrity.\n\nThe researchers have made everything available, including the dataset and the model weights, which is great for anyone wanting to run a hands-on guide on fine-tuning Whisper for other low-resource languages.\n\n```\nhttps://github.com/subedibiraj/speakpay\n```\n\n[I don't have the actual article content to rewrite — only the 13d ago](/en/news/7268/)\n\n[Table Canon tries to solve TTRPG session memory with an AI 13d ago](/en/news/7259/)\n\n[Training image models on a laptop actually works now 15d ago](/en/news/6930/)\n\n[Voice cloning just turned grandparent scams into a nightmare 15d ago](/en/news/6907/)\n\n[How Much VRAM to Fine-Tune an LLM? 12 to 120 GB 8/2/2026](/en/news/4751/)\n\n[Next Sanders wants to ban artificial superintelligence and pause AI →](/en/news/8774/)", "url": "https://wpnews.pro/news/whisper-s-zero-shot-performance-on-nepali-financial-commands-is", "canonical_source": "https://promptcube3.com/en/news/8783/", "published_at": "2026-09-04 03:51:16+00:00", "updated_at": "2026-09-04 04:23:07.488572+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-research"], "entities": ["Whisper", "OpenAI", "NepFinSpeech-403", "LoRA", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/whisper-s-zero-shot-performance-on-nepali-financial-commands-is", "markdown": "https://wpnews.pro/news/whisper-s-zero-shot-performance-on-nepali-financial-commands-is.md", "text": "https://wpnews.pro/news/whisper-s-zero-shot-performance-on-nepali-financial-commands-is.txt", "jsonld": "https://wpnews.pro/news/whisper-s-zero-shot-performance-on-nepali-financial-commands-is.jsonld"}}