{"slug": "a-10kb-page-that-sings-text-can-you-decode-it-back-pitch-only-baseline-already", "title": "A ~10KB page that sings text — can you decode it back? (pitch-only baseline already hits 43%)", "summary": "A developer proposes a tiny deterministic singing-code sandbox for synthetic Singing Voice Transcription, breaking the problem into audio-to-notes, audio-to-text, and word-motif alignment layers. The project's generator can produce perfect labels for audio, text, notes, and timing, enabling a reproducible benchmark before scaling to neural ASR models.", "body_md": "This looked interesting, so I checked whether there are reusable pieces around it:\n\nYes — I think this is very plausibly decodable and developable, but I would not start by framing it as only “ASR” or only “a language question”.\n\nA useful nearby frame is:\n\na tiny deterministic singing-code / synthetic Singing Voice Transcription sandbox\n\nIn other words, this can be separated into several layers:\n\n```\naudio → notes / motifs\naudio or motifs → text\nword ↔ motif alignment\n```\n\nThat separation seems useful because nearby fields already have reusable tools, metrics, datasets, and failure-mode maps.\n\nThe interesting part is that this project has something real singing datasets usually do not have: **a generator that can produce perfect labels** for audio, text, notes/motifs, timing, vowels, and word↔motif alignment.\n\nSo before jumping to a large neural ASR model, I would probably grow it in this order:\n\n```\ntoy demo\n→ reproducible benchmark\n→ structured metadata\n→ note/motif scoring\n→ text scoring\n→ word↔motif alignment scoring\n→ stronger non-ML baselines\n→ toy / injective / noisy / style variants\n→ CTC / Wav2Vec2 / SVT-style joint models\n→ human-learnability / code-language boundary questions\n```\n\nA particularly useful neighboring area is **Singing Voice Transcription**, especially newer work that tries to unify lyric transcription, note transcription, and lyric-note alignment.\n\nIn ordinary speech ASR, the target is mostly:\n\n```\naudio → text\n```\n\nBut in singing transcription, the target is often closer to:\n\n```\naudio → lyrics\naudio → notes / melody\nlyrics ↔ notes alignment\n```\n\nThat maps surprisingly well onto this project:\n\n| SVT / singing transcription layer | This project’s analogue |\n|---|---|\n| lyrics transcription | audio → original text |\n| note transcription | audio → note/motif sequence |\n| lyric-note alignment | word ↔ motif alignment |\n| phoneme/vowel alignment | vowel/formant cues inside motifs |\n| style / singer variation | later style/noisy/timbre variants |\n| out-of-distribution singing | later robustness tests |\n\nSo I would treat this less as “general ASR” at first, and more as a controlled synthetic version of an SVT problem.\n\nSome nearby references:\n\n| Reference | Why it seems relevant |\n|---|---|\n|\n\nI would split the benchmark into at least three measurable tasks.\n\n| Layer | Task | Possible metric family |\n|---|---|---|\nNote/motif recovery |\nrecover pitch events, notes, or motif sequence from audio | MIR-style note transcription metrics, motif accuracy |\nText recovery |\nrecover original word sequence | word accuracy, WER, CER |\nAlignment recovery |\nrecover which word corresponds to which motif and when | alignment accuracy, timing error |\nRobustness |\nrecover the same labels under perturbations | score degradation under noise/compression/reverb/etc. |\n\nUseful existing tooling:\n\n| Tool | Use |\n|---|---|\n|\n\nInstead of only storing `audio,text`\n\n, it may be useful to store the generated structure.\n\nSomething like:\n\n```\n{\n  \"audio\": \"sample_0001.wav\",\n  \"text\": \"the red bird\",\n  \"words\": [\n    {\n      \"word\": \"red\",\n      \"motif_id\": \"m_014\",\n      \"notes\": [\"A4\", \"C5\"],\n      \"vowels\": [\"e\"],\n      \"start\": 0.42,\n      \"end\": 0.81\n    }\n  ],\n  \"codec_version\": \"loom-v0\",\n  \"split\": \"test\",\n  \"difficulty\": \"toy\",\n  \"collision_group\": null\n}\n```\n\nThis would make the dataset usable from several angles:\n\n| User wants to test… | They can use… |\n|---|---|\n| text decoding | `audio` , `text` |\n| pitch/motif recovery | `audio` , `notes` , `motif_id` |\n| vowel-aware decoding | `audio` , `vowels` , `word` |\n| alignment | `start` , `end` , `word` , `motif_id` |\n| codec audit | `motif_id` , `collision_group` , `codec_version` |\n| robustness | same labels under noisy variants |\n\nThe DALI dataset is a useful mental model here: it stores lyrics and vocal notes with time alignment and multiple lyric granularities. For this project, the analogous hierarchy could be:\n\n```\nnote → motif → word → line/sample\n```\n\nI would probably avoid jumping straight from the current pitch-only baseline to a large ASR model.\n\nA more informative ladder might be:\n\n| Baseline | What it tells you |\n|---|---|\n| current FFT pitch-only greedy decoder | floor baseline |\n| pitch-only + dynamic programming | removes some greedy parsing weakness |\n| pYIN/aubio pitch + DP decoder | better note tracking without full ASR |\n| pitch + vowel-aware decoder | tests whether vowel/formant information helps |\n| oracle note sequence → text | isolates codec/parser ambiguity |\n| oracle note+vowel sequence → text | estimates upper bound from symbolic information |\n| small CTC model | first neural baseline |\n| Wav2Vec2/Whisper-like ASR fine-tuning | general ASR transfer baseline |\n| SVT-style joint decoder | future structured model: audio → motifs + text |\n\nThis gives more diagnostic value than a single score, because each step answers a different question.\n\nTo keep the idea clean, I would separate variants rather than mixing all goals into one task.\n\n| Variant | Goal |\n|---|---|\ntoy |\ncurrent fun/demo version; ambiguity allowed |\ninjective |\ncollision-free / prefix-safe version for stricter codec testing |\nnoisy |\ncompression, additive noise, reverb, resampling, time stretch |\nstyle |\ntimbre, vowel, vibrato, voice/singer variation |\nphysical |\nspeaker-to-microphone playback |\nopen-vocab |\nheld-out words or motifs |\nhuman |\nlearnability and human decoding experiments |\n\nThe **injective** variant would be useful if the question is “can this be a lossless audio code?”\n\nThe **toy** variant is still useful if the question is “how far can decoders get under ambiguity?”\n\nThe **noisy/style** variants are useful if the question becomes closer to ASR/SVT robustness.\n\nSVT is useful for the singing/transcription side, but the codec side also has nearby prior art.\n\n| Reference | Relevance |\n|---|---|\n|\n\nThis suggests another layer of knobs:\n\n```\nsymbol duration\nnote gap\nmotif gap\nword gap\nstart/end markers\nchecksum\nerror correction\npayload rate\ncollision report\n```\n\nThat does not mean this should become a modem. It just means the modem world has useful engineering vocabulary for controlled acoustic codes.\n\nI would keep the language question, but separate it into layers:\n\n| Layer | Question |\n|---|---|\n| recoverability | can a decoder recover the source text? |\n| unique decodability | is the audio code unambiguous in principle? |\n| robustness | does it survive noise, timing changes, compression, etc.? |\n| learnability | can humans learn it without memorizing arbitrary pairs? |\n| convention | could multiple users share it consistently? |\n| productivity | can it support new utterances systematically? |\n| language-like behavior | does it go beyond codebook lookup? |\n\nNearby conceptual references:\n\n| Area | Why it helps |\n|---|---|\n|\n\nSo I would phrase the language angle carefully:\n\nHigh decoding accuracy would be interesting, but it would mostly show recoverability under the chosen code and evaluation setup. Human learnability, convention, and productivity are separate next questions.\n\nThat keeps the door open without overclaiming.\n\nIf I were trying to grow this with existing pieces, I would probably do:\n\n**Keep the current toy benchmark reproducible**\n\n**Add structured metadata**\n\n**Add two scoring tracks**\n\n`mir_eval`\n\n-style note/motif scoring`jiwer`\n\n-style text scoring**Add an alignment track**\n\n**Add a baseline ladder**\n\n**Split variants**\n\n**Only then try heavier models**\n\nThe strongest part, to me, is that this creates a small deterministic audio world where several normally tangled problems can be tested separately:\n\n```\ncodec design\nacoustic decoding\nmotif recovery\ntext recovery\nvowel cues\nalignment\nnoise robustness\nhuman learnability\ncode/language boundary\n```\n\nReal SVT datasets have annotation cost, singer variability, accompaniment, noisy alignment, and style differences. This toy system can start with none of that, then add complexity deliberately.\n\nSo the most useful framing may be:\n\na controllable synthetic SVT/audio-code benchmark, with perfect labels and gradually adjustable ambiguity.\n\nThat seems like a nice place to test simple decoders first, then stronger ASR/SVT models later.", "url": "https://wpnews.pro/news/a-10kb-page-that-sings-text-can-you-decode-it-back-pitch-only-baseline-already", "canonical_source": "https://discuss.huggingface.co/t/a-10kb-page-that-sings-text-can-you-decode-it-back-pitch-only-baseline-already-hits-43/176840#post_6", "published_at": "2026-06-20 01:03:19+00:00", "updated_at": "2026-06-20 01:13:52.824733+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "ai-research", "developer-tools"], "entities": ["DALI dataset", "Wav2Vec2", "CTC", "SVT"], "alternates": {"html": "https://wpnews.pro/news/a-10kb-page-that-sings-text-can-you-decode-it-back-pitch-only-baseline-already", "markdown": "https://wpnews.pro/news/a-10kb-page-that-sings-text-can-you-decode-it-back-pitch-only-baseline-already.md", "text": "https://wpnews.pro/news/a-10kb-page-that-sings-text-can-you-decode-it-back-pitch-only-baseline-already.txt", "jsonld": "https://wpnews.pro/news/a-10kb-page-that-sings-text-can-you-decode-it-back-pitch-only-baseline-already.jsonld"}}