Hrant Davtyan's lab tested nearly 30 systems on 20.7 hours of Armenian audio across five speech datasets, with closed systems leading the combined ranking.
By RuntimeWire Staff · Published
Primary source: Hugging Face Newsroom
Why it matters #
ArmBench-ASR gives Armenian developers a shared test across real speech domains, while showing that closed-model leads shrink or disappear on specific audio types.
Hrant Davtyan, founder and CEO of Metric, released ArmBench-ASR on August 20, giving Armenian speech recognition a common test across read speech, poetry, movie dialogue and narrated news.
The initial v0.1 benchmark evaluates nearly 30 open-weight and closed systems using 10,113 audio clips totaling about 20.7 hours. Alexander Shahramanyan and Mariam Avetisyan are among the Metric researchers credited on the release alongside Davtyan.
Google's Gemini 2.5 Pro placed first with a strict combined word error rate, or WER, of 14.31%. HiSpeech's Armenian conversational model followed at 16.81%, and Gemini 2.5 Flash placed third at 17.55%. Lower scores are better. The eight best combined WER results came from closed systems, while NVIDIA's Armenian FastConformer was the highest-ranked open model in ninth place at 20.21%.
Metric's benchmark premise is straightforward: a single clean read-speech score cannot show how Armenian ASR systems handle poetry, produced media, or narrated news. Public evaluations often cover a single speech style, use inconsistent text processing or lack standardized procedures. ArmBench-ASR applies the same evaluation framework across five domains.
Davtyan has spent much of his career moving between applied data science, teaching and company building. The American University of Armenia identifies him as an assistant professor and a co-founder of the Metric research center. His personal site says he previously grew an AI software operation from one employee to 21 and from one project to 30 projects over two years without marketing spending.
Metric's Armenian-language work occupies a self-funded research track. On its research page, Metric says its researchers invest their own time and budget because Armenian AI infrastructure matters even when the immediate business case does not. ArmBench-ASR turns that commitment into something other model builders can measure against.
Five datasets expose five different problems
ArmBench-ASR uses the same fixed material and processing rules for every evaluated system. Its two public components are Common Voice 26, a crowdsourced read-speech set, and Google's FLEURS Armenian test split. Metric published a small set of corrections after manually reviewing FLEURS transcripts.
Metric added three private collections. Poems contains expressive literary readings with background music. Movies contains conversational dialogue and produced-media noise. Infocom consists of Armenian news text recited by one speaker. Metric says it manually reviewed and corrected the reference transcripts for all three.
The interactive leaderboard breaks results out by dataset and lets users compare strict and normalized word and character error rates. Strict scoring preserves sensitivity to punctuation and capitalization. Normalized scoring lowercases text and removes punctuation after applying consistent Unicode, spacing and character-processing rules to both references and model output.
That distinction materially changes the numbers. Gemini 2.5 Pro's normalized combined WER was 6.42%, less than half its strict score. The gap indicates that punctuation, capitalization, spacing and orthographic choices account for a substantial portion of the measured errors. It also warns buyers against treating a single leaderboard percentage as a complete description of transcription quality.
Metric reports that every evaluated model recorded its highest strict WER on the private Movies collection. The median reached 61.87%, compared with 17.29% on Common Voice 26 and 17.02% on FLEURS. Background noise is one possible factor, according to Metric, although conversational delivery and produced audio create additional differences from read-speech datasets.
The domain results also soften the aggregate victory for closed APIs. NVIDIA's FastConformer led Common Voice at 7.70% WER, ahead of Gemini 2.5 Pro's 9.69%. HiSpeech's two Armenian models beat Gemini on strict WER for poetry, while Gemini led after normalization. Model choice therefore depends heavily on the audio and the desired output conventions.
The useful data is also the difficult part to verify
ArmBench-ASR's three private datasets make the benchmark broader than a public read-speech test. Their absence from the release also prevents outside researchers from fully reproducing the most distinctive portion of the evaluation.
Metric states the limitation directly. The current suite primarily covers Eastern Armenian and excludes Western Armenian and regional dialects. It tests transcription accuracy without measuring speaker diarization, word-level timestamps, streaming, long-form behavior or downstream performance. The evaluations occurred in late July and early August 2026, so hosted systems may later change through unversioned model updates or routing decisions.
That matters most for closed APIs. A future rerun under the same product name may reach a different underlying model. ArmBench-ASR used default parameters, with Gemini temperature set to zero and thinking disabled or kept to a minimal setting. Those choices make this release a dated comparison of configured services rather than a permanent ranking.
The benchmark still fills a practical gap beside broad projects such as Hugging Face's Open ASR Leaderboard and multilingual baselines such as OpenAI's Whisper. General-purpose evaluations can show whether a speech model travels across languages. Davtyan and Metric are asking the narrower question Armenian developers actually face: what happens when the speaker stops reading a clean test sentence and the movie starts.
Metric has been building a wider Armenian evaluation stack, including ArmBench projects for language models and text embeddings. SmartGateVC invested in Metric's pre-seed round in 2021, when Metric was described as an applied AI research lab formerly known as Pinsight. The amount and valuation were not published.
ArmBench-ASR extends that research program into speech while exposing where the next work lies. Metric plans to add code-switching, spontaneous conversations, interviews, calls, meetings, overlapping speakers, specialist vocabulary, diarization and timestamp quality. Each addition raises the cost of collecting and validating references, especially for a language with limited standardized data.
Davtyan's bet is that the missing infrastructure is worth building anyway. The first release gives Armenian model developers a baseline and a set of difficult audio domains. It also gives the best-funded speech providers a test beyond another clean reading set.