{"slug": "open-research-catalog-168-tts-and-106-speech-to-text-systems", "title": "Open research catalog: 168 TTS and 106 speech-to-text systems", "summary": "D3velop LLC's Open Gauntlet leaderboard lists 168 text-to-speech and 106 speech-to-text systems, but the ASR page's hero count of 111 is outdated, and the Fireworks and Yandex entries need updates. The catalog distinguishes between benchmarked LLMs and surveyed speech systems, with the ASR JSON internally consistent at 106, suggesting a static copy error. Fireworks' current model library lists Whisper V3 rates at $0.0015, $0.0009, $0.0032, and $0.0035 per audio minute, contradicting the row that only V3 Turbo was confirmed. Yandex SpeechKit STT documentation now exposes 16 locale codes, three version tags, and limited speaker labeling, enabling a more complete entry.", "body_md": "For now, I took a quick look at a few things:\n\nThis seems useful precisely because it does **not** present the 274 speech systems as though they had all been measured under one protocol. The distinction between the directly measured LLM leaderboard and the “surveyed, not benchmarked” [TTS/STT research catalogs](https://github.com/D3velop-llc/open-gauntlet-leaderboard) is clear, and the latter includes many of the things that tend to matter only after someone tries to use a system: deployment mode, licensing splits, lifecycle, pricing units, diarization, language coverage, and operational catches.\n\nI am not a speech specialist, so I stayed with things I could check fairly narrowly against the public data and first-party documentation. I found a few concrete entries that may be worth refreshing, plus two small maintenance changes that might prevent the same class of issue from recurring.\n\nThe [ASR page](https://opengauntlet.com/asr.html) currently says:\n\n`surveyed, not benchmarked — 111 systems`\n\nin the hero`Listening surveyed · 106`\n\nin the pipeline/navigation`asr.json`\n\nThe JSON contains 106 system records, declares `counts.total: 106`\n\n, and its category totals also sum to 106. TTS appears internally consistent at 168.\n\nSo this looks less like a substantive catalog problem and more like one static copy of the count surviving after the underlying data changed.\n\nA small build-time invariant might be enough to prevent this:\n\n```\nsystems.length\n== counts.total\n== sum(counts.by_category)\n== hero count\n== navigation / footer count\n```\n\nAlternatively, deriving every visible count from the same JSON value would remove the duplicated state entirely.\n\nThe current Fireworks row says that only the Whisper V3 Turbo rate could be confirmed on a Fireworks-owned page, while the commonly cited `$0.0015/min`\n\nrate for Whisper V3 Large and the streaming prices could not.\n\nThe current [Fireworks model library](https://fireworks.ai/models?modelTypes=Audio) lists:\n\n`$0.0015 / Audio Minute`\n\n`$0.0009 / Audio Minute`\n\n`$0.0032 / Audio Minute`\n\n`$0.0035 / Audio Minute`\n\nThe older first-party [audio transcription launch post](https://fireworks.ai/blog/audio-transcription-launch) also lists the `$0.0015`\n\nand `$0.0009`\n\nWhisper rates together, with some of the inference conditions around them.\n\nI would not necessarily merge all four prices into one undifferentiated number: the two streaming systems appear to be separate products, and dedicated endpoints remain a different pricing path. But the narrower statement that the base Whisper price cannot be found on a Fireworks-owned page now seems refreshable.\n\nThe current row is understandably conservative: it records that Yandex’s documentation returned bot verification during the August 1 research pass, and therefore leaves pricing, languages, streaming, deployment options, and speaker labeling largely unverified.\n\nThe documentation is still somewhat awkward to retrieve automatically, but the current first-party pages expose enough information for a more complete row.\n\nThe [SpeechKit STT overview](https://aistudio.yandex.ru/docs/en/speechkit/stt/) describes an active service with:\n\nThe current [supported languages and recognition models](https://aistudio.yandex.ru/docs/en/speechkit/stt/models) page explicitly lists 16 locale codes plus automatic language detection. It also documents three moving version tags:\n\n`general`\n\n`general:rc`\n\n`general:deprecated`\n\nand says the previous `general`\n\nversion is supported for two weeks after an update. That version-tag behavior may be particularly useful to record because it makes the hosted service a moving target even when the product name does not change.\n\nThere is also first-party documentation for [speaker labeling](https://aistudio.yandex.ru/docs/en/speechkit/stt/speaker-labeling). It is limited rather than general-purpose diarization:\n\n`FULL_DATA`\n\nmodeSo I would not simply replace `no-diarization`\n\nwith an unrestricted `diarization`\n\nlabel. Something like “limited two-speaker labeling under specific API v3 conditions” would be more precise.\n\nFinally, [SpeechKit Hybrid system requirements](https://aistudio.yandex.ru/docs/en/speechkit/hybrid-speechkit/system-requirements) describe a customer-infrastructure deployment using Linux, Docker, and supported NVIDIA GPUs, including capacity figures for streaming recognition, file recognition, and file recognition with speaker labeling.\n\nThat creates a classification choice rather than one obviously correct edit:\n\n```\nIf SpeechKit Hybrid is treated as the same product family:\n    update deployment / self-hosting notes in the existing row\n\nIf hosted SpeechKit and Hybrid are materially different purchasing products:\n    keep the hosted row and add Hybrid as a separate commercial on-prem entry\n```\n\nEither route seems more accurate than `Cloud only`\n\n, but keeping hosted per-use pricing separate from individually licensed Hybrid deployment would avoid mixing unlike products.\n\nThe current Mistral Voxtral Transcribe 2 row already captures the important split reasonably well:\n\nThe row currently has no `source_url`\n\n, but there are straightforward first-party sources.\n\nThe [Voxtral Mini Transcribe 2 model card](https://docs.mistral.ai/models/model-cards/voxtral-mini-transcribe-26-02) gives:\n\n`voxtral-mini-2602`\n\n`$0.003/min`\n\nThe [Mistral audio overview](https://docs.mistral.ai/studio-api/audio/overview) distinguishes:\n\nSo this seems mostly like an opportunity to attach first-party links to the scopes already described, rather than a reason to rewrite the row.\n\nBeyond those entry-level updates, two low-cost changes seem likely to return more value than a large schema redesign.\n\nThe 111/106 mismatch is exactly the kind of small error that becomes hard to notice in a generated static site because each individual artifact still looks reasonable.\n\nA build check over the published artifacts could fail when:\n\n```\nJSON record count != declared JSON count\ncategory total != declared total\nHTML count != JSON count\nREADME count != JSON count\n```\n\nThe README check could perhaps remain advisory if it is intentionally hand-written, while JSON/HTML inconsistencies could fail the build.\n\nThe site and README link “Request a model” and “Report a correction” to GitHub’s new-issue page. At the moment, the repository’s [Issues page](https://github.com/D3velop-llc/open-gauntlet-leaderboard/issues) says:\n\nIssue creation is restricted in this repository\n\nThat may be completely intentional—for example, as a spam-control choice. GitHub only recently added an official [collaborator-only issue creation setting](https://github.blog/changelog/2026-06-29-restrict-issue-creation-to-collaborators-only/), so I would not assume it is a configuration mistake.\n\nHowever, the forum post already says corrections can also be collected in this thread. If the GitHub restriction is intentional, the smallest fix may simply be to make that existing fallback visible beside the correction button:\n\n```\nReport a correction:\n- GitHub, if you have access\n- otherwise use the Hugging Face Forum thread\n```\n\nThat would preserve the repository setting without leaving ordinary readers at a dead end.\n\nI would not say that provenance is generally missing. In a small static pass over the public data, the numerical ratings I found were already accompanied by a metric, named source, source URL, and as-of date. That is a good distinction to preserve.\n\nThe harder-to-maintain information appears to be the operational prose around those ratings:\n\nFor those fields, even a lightweight `checked_at`\n\nand an optional claim-specific source may make later corrections much cheaper:\n\n```\nsource_url: https://example.com/product\nchecked_at: 2026-08-04\n\nclaim_sources:\n  price:\n    url: https://example.com/pricing\n    checked_at: 2026-08-04\n    scope: serverless batch API\n\n  speaker_labeling:\n    url: https://example.com/docs/speaker-labeling\n    checked_at: 2026-08-04\n    scope: API v3, mono, maximum two speakers\n```\n\nThis does not need to become a full provenance ontology. Applying it only to information expected to change would probably capture most of the benefit.\n\nHugging Face’s developing [Evaluation Results](https://huggingface.co/docs/hub/en/eval-results) format is one nearby example of keeping source, date, dataset/revision, notes, and contribution status distinct, although the catalog has a broader purpose than an evaluation-results file.\n\nOverall, these look like normal maintenance pressure in a large, fast-moving catalog rather than a problem with the project’s basic approach.\n\nMy default order would be:\n\n`checked_at`\n\nonly where the underlying fact is likely to change.That seems likely to improve the catalog’s maintainability without turning it into a much heavier system.", "url": "https://wpnews.pro/news/open-research-catalog-168-tts-and-106-speech-to-text-systems", "canonical_source": "https://discuss.huggingface.co/t/open-research-catalog-168-tts-and-106-speech-to-text-systems/178416#post_2", "published_at": "2026-08-04 03:34:58+00:00", "updated_at": "2026-08-04 03:44:56.715775+00:00", "lang": "en", "topics": ["ai-research", "ai-products", "ai-tools"], "entities": ["D3velop LLC", "Open Gauntlet", "Fireworks", "Yandex SpeechKit", "Whisper V3 Turbo", "Whisper V3 Large"], "alternates": {"html": "https://wpnews.pro/news/open-research-catalog-168-tts-and-106-speech-to-text-systems", "markdown": "https://wpnews.pro/news/open-research-catalog-168-tts-and-106-speech-to-text-systems.md", "text": "https://wpnews.pro/news/open-research-catalog-168-tts-and-106-speech-to-text-systems.txt", "jsonld": "https://wpnews.pro/news/open-research-catalog-168-tts-and-106-speech-to-text-systems.jsonld"}}