{"slug": "creating-word-recordings-with-ai-learning-where-to-cut", "title": "Creating word recordings with AI: learning where to cut", "summary": "A developer building a spelling game used audio.cpp with the VoxCPM2 and Chatterbox voice models to generate AI word recordings in English, French, and Norwegian, then slowed the reference voice to 85% tempo with ffmpeg to improve clarity. The generated audio was uneven, with some words losing their beginnings, stopping early, or carrying fragments of preceding words, and adjusting generation steps, guidance, speed, seeds, and word-control tokens produced inconsistent results. The developer moved from OmniVoice to VoxCPM2 because of licensing requirements.", "body_md": "Writing\n\n# Creating word recordings with AI: learning where to cut\n\nA playful spelling game, some troublesome AI audio, and the small tools that helped me turn generated sentences into usable word recordings.\n\n## On this page\n\nI dipped my toe into AI, voice cloning, and generated audio by making a spelling game.\n\n## The game\n\nListen to a word and try to spell it.\n\nIt evolved to include hearts, clues, streaks, single player, multiplayer, replay mode, and all sorts. It’s wild what you can do when you have AI at your side.\n\nIt was fun to make and fun to play.\n\nThe audio was more uneven. Some words sounded great. Others had lost their beginning, stopped too early, or brought a fragment of the preceding word along with them. For a game where the player has to work out what they heard, that is a fairly central problem.\n\nI wanted a WAV file of each word, spoken clearly from beginning to end.\n\n## The voices\n\nUsing [audio.cpp](https://github.com/0xShug0/audio.cpp), I created AI voices for English, French, and Norwegian.\n\nInitially I used [OmniVoice](https://huggingface.co/k2-fsa/OmniVoice), before moving to [VoxCPM2](https://github.com/OpenBMB/VoxCPM) for voice creation. That move was driven by the [licensing requirements](#licence).\n\nI used VoxCPM2 to create a reference recording, then used that recording with [Chatterbox](https://huggingface.co/ResembleAI/chatterbox) (model running thru audio.cpp) to generate the sentences. The reference gave Chatterbox an example of the voice and delivery I wanted.\n\n### Creating a voice\n\nAfter my first attempts, a [Reddit comment about helping children learn to spell](https://www.reddit.com/r/AskTeachers/comments/1ocvs89/comment/nkpmz1c/) made me think more carefully about the voice. I wanted clear articulation and a patient delivery.\n\nA back-and-forth with AI led to a teacher prompt. Alongside it, I supplied a short reference sentence: enough speech to establish the voice and accent without making the model follow a long passage’s rhythm.\n\nHere is a complete command example, using a reference sentence from my experiments. Run it from the audio.cpp directory after building the CLI, and adjust the model path for your setup.\n\n```\nTEACHER_PROMPT=$(cat <<'PROMPT'\n(\nA professional female Norwegian literacy teacher in her thirties.\nHer voice is warm, calm, clear, and exceptionally articulate.\nShe has precise consonants, clean vowels, excellent diction, and natural Norwegian pronunciation.\nHer voice is friendly and reassuring without sounding childish.\nShe speaks with a composed, confident teaching presence.\nHer pitch is natural and comfortable.\nHer voice is clean, with no raspiness, breathiness, vocal fry, or exaggerated emotion.\n)\nPROMPT\n)\n\nREFERENCE_TEXT='Når vi snakker tydelig og rolig, blir det lettere å høre hver enkelt lyd. Vi bruker stemmen naturlig, med klare vokaler og presise konsonanter.'\n\n./build/macos-metal-release/bin/audiocpp_cli \\\n  --task tts \\\n  --family voxcpm2 \\\n  --model \"$HOME/models/VoxCPM2-GGUF/voxcpm2-q8_0.gguf\" \\\n  --backend metal \\\n  --text \"${TEACHER_PROMPT} ${REFERENCE_TEXT}\" \\\n  --out teacher.wav\n```\n\nI then slowed the recording to 85% of its original tempo:\n\n```\nffmpeg -i teacher.wav \\\n  -filter:a \"atempo=0.85\" \\\n  teacher-slow.wav\n```\n\nThat gave me a more deliberate reference recording to use for sentence generation. The next job was making the individual words reliable and clear.\n\n## The audio\n\nWe went on a journey, AI and I. Some experiments moved us forward; others sent us back to listening.\n\n### Add knobs\n\nI started by adjusting the controls available to me: generation steps, guidance, speed, seeds, and word-control tokens.\n\nGenerate a word, listen, change something, try again.\n\nThere were encouraging results. There were also results that made it hard to know what I had learnt. A different seed could rescue a recording, but it did not explain why the previous attempt had failed. A setting that helped one word did not necessarily help the next.\n\nIt wasn’t going well.\n\n### Give the word a sentence\n\nThe models I tried struggled with isolated words, so we added a sentence template to give them more context and encourage a natural delivery.\n\nThis gave the model some context and gave me a pattern to look for. The last word should follow a pause. Find that pause, cut there, and save the result.\n\n```\nThis is the word ‘acquaintance’. Acquaintance.\n```\n\n### The problem\n\nWhere should I cut the sentence to capture the final word?\n\nIt sounded like something FFmpeg could do in one line. That turned out to be wishful thinking.\n\nThe first useful distinction was between detecting silence and deciding what that silence meant. FFmpeg could report quiet intervals. It could not tell me that a particular interval separated the introductory sentence from the final word.\n\nEven reading the settings involved some learning. A minimum silence of `0.12` meant a quiet interval had to last 120 milliseconds to qualify. It was not how long to record. If the useful pause was shorter, my script could miss it completely.\n\nReducing that threshold found more pauses. It also gave the script more opportunities to pick the wrong one.\n\n### A plausible cut can still be wrong\n\nI tried rules based on the last pause, the longest pause, and how much sound remained afterwards.\n\nEach had a reason behind it. Each met a recording that exposed the assumption.\n\nWith `kjole`, an early cut kept almost the whole sentence. Rejecting an output that retained 80% or more of the recording caught that sort of mistake. But a file could be much shorter than the original and still contain the wrong audio.\n\nAt the other extreme, one candidate left only about 26 milliseconds of sound above the threshold. That was a useful clue: perhaps I had found a gap inside the word and kept only its ending.\n\nI added a rough minimum based on the number of letters. That helped reject implausibly small fragments, but letters are an imperfect measure of spoken duration. Quiet sounds also complicate any rule based on time above a volume threshold.\n\n`Acquaintance` kept coming back. Its ending sounded troublesome even when I could hear the sentence pronounce it properly. But a check of the trial clip showed it contained the complete, unchanged ending. My first description of the problem was not enough to diagnose it. `Jente` exposed a different difficulty: the gap before the repeated Norwegian word could be tight enough that a seemingly reasonable cut still sounded wrong.\n\nI needed to see what the rules were choosing.\n\n### Visualise\n\nI asked AI to build a small browser tool for opening a WAV, selecting a region, and playing exactly that region. It became audio-lab. As it evolved, it became much easier to show the AI what I meant and check the changes we made.\n\nI could drag the boundaries, zoom in, slow playback, and listen again. Then we added silence detection, with all the detected gaps highlighted together over the waveform.\n\nThe decibel threshold made more sense when I could change it and see which areas qualified as quiet. These bands represented sound below a chosen threshold for a chosen duration. They were not proof of empty space between words. A quiet part of speech could end up inside a band too.\n\nThe next useful addition was selection JSON: a small record of the file and the start and end times of a region. The extractor could write its proposed selection, and audio-lab could load it over the original recording.\n\nThat made the script’s decision inspectable. I could load its JSON, hear exactly what it proposed, and compare the boundary with the detected gaps. A file fingerprint helped make sure the selection belonged to the recording I had opened.\n\nI could also move the selection by hand and copy the new coordinates back into the discussion. Instead of describing a vague problem at the beginning, I could point to an interval and explain what I heard there.\n\n### Hear the difference\n\nHere’s what those decisions looked and sounded like. This recreated comparison uses one saved `acquaintance` sentence, made with a later template. Both cuts use that same recording, so the comparison demonstrates the cutting failure, not a change in generated speech.\n\nThe simple rule chooses a short gap after the final word has begun. The later rule chooses the longer gap before it. Both keep the original ending.\n\nThe controls below each screenshot play its extracted WAV; click the image to inspect it at full size. To reproduce the selections in audio-lab, open the complete sentence WAV and then load either selection JSON. The [reproduction commands](reproduce-examples.txt) include the extraction settings.\n\nThis was a particularly useful role for AI. It could make the interface I needed while I was still discovering what I needed to inspect.\n\n### Estimate the sentence, then look nearby\n\nWe knew more than the waveform alone. We knew the word and the sentence template.\n\nThat suggested estimating where the introductory sentence should end, then looking for a nearby pause. I tried counting letters and assigning a rough number of milliseconds per letter.\n\nThere was a mistake hidden in that too. The small number I had used as a minimum-duration check for a word was unsuitable as an estimate of normal speaking time. Those calculations served different purposes, even though both used milliseconds per letter.\n\nA larger estimate got us into a more useful part of the recording. Then leading silence, speaking pace, and trailing silence became important. A fixed rate was still a guess, and different voices did not all speak at that rate.\n\nThe next step used the recording itself to estimate the rate. Find the sentence’s start and the last sound, account for the known text, and evaluate candidate gaps with their duration taken out of the timing calculation. Use that estimate to help choose a plausible boundary.\n\nThis was still a collection of heuristics. I was learning from examples and improving rules; I was not training a model to recognise word boundaries.\n\nAudio-lab made that distinction useful rather than frustrating. I could inspect a guess and work out why it had failed.\n\n### Change the sentence, too\n\nAfter all that attention to the cut, I learnt something I wish I had tried earlier: changing the sentence template could improve the generated audio itself.\n\nI experimented with the wording, quotation marks, and punctuation:\n\n```\nThis is the word ‘{word}’. {spoken}.\nThis is the word \"{word}\". {spoken}.\nThis is the word '{word}'. {spoken}.\nThis is the word '{word}.' {spoken}.\n```\n\nThis version worked best in my English tests:\n\n```\nPlease say the word '{word}.' {spoken}.\nPlease say the word 'acquaintance.' Acquaintance.\n```\n\nNotice the full stop inside the quotation marks and the capitalised first letter of the repeated word. The combination helped in my tests. I hadn’t isolated each change, so I couldn’t say how much the wording, punctuation, or capitalisation contributed individually.\n\n`Acquaintance` worked cleanly with this template. The source sentence in the audio comparison above uses it. I had been asking where to cut the recording; changing what I asked the model to say was another way to improve the result.\n\n### Give cleanup its own job\n\nA later `acquaintance` recording had a little hiss before the word. The main extraction could get me close while still leaving an unwanted opening sound.\n\nI could have kept adding conditions to the extractor. Instead, I added a separate cleanup step operating on the extracted word.\n\nIt examined the beginning for an opening sound followed by a qualifying quiet gap, then proposed a later start with a little padding. It wrote a preview WAV and JSON so I could inspect the result in audio-lab before accepting it.\n\nThis was trimming an unwanted lead-in, rather than removing noise throughout the recording. Finding a quiet gap did not by itself establish that everything before it was disposable.\n\nSeparating the steps helped. I could keep the generated sentence, repeat extraction, and try cleanup settings independently. Some recordings that still sounded rough after extraction became useful after cleanup.\n\n### A second opinion, and a better listening desk\n\nI used [Whisper](https://github.com/openai/whisper) and [nb-whisper](https://huggingface.co/NbAiLab/nb-whisper-small) to compare recognised text with the expected word. A mismatch gave me somewhere to look.\n\nThere was a configuration mistake here too: sending `language=no` did not mean I was using nb-whisper. It selected Norwegian transcription on the server receiving the request. The regular Whisper and nb-whisper models were running separately; I had to use the correct endpoint.\n\nRecognition results also needed listening judgment. In one Norwegian batch, seven recordings were flagged, but after listening I considered only one wrong. In another English check, I agreed with the seven flagged results. A mismatch was useful evidence, not a verdict.\n\n`review.py` made this manageable. It turned the batch JSON into a listening page, with the source sentence, extracted audio, available cleanup previews, word IDs, transcripts, and expandable metadata. Filters let me concentrate on mismatches or recordings with cleanup versions.\n\nThat little page saved a lot of opening files and remembering which version I had just heard.\n\n## Licence\n\nI wanted to keep the possibility of commercial use open, and the specific OmniVoice Word-Control weights I was trying were marked CC-BY-NC-4.0. Checking the licence on the code alone would have missed that. [OmniVoice Word-Control model card](https://huggingface.co/multimodalart/omnivoice-word-control#license--lineage).\n\nThat pushed me towards Chatterbox for the sentence-generation workflow and VoxCPM2 for voice experiments. Their published model cards list MIT and Apache-2.0 respectively. The particular model and weights mattered when making that decision. [Chatterbox](https://huggingface.co/ResembleAI/chatterbox), [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2).\n\n## Tooling\n\nThe two small tools made a big difference: audio-lab let me inspect a cut, and the review page let me move quickly through a batch, listening and comparing versions.\n\nBefore AI, I would have weighed up the time needed to build a tool against the time it might save. That was harder when I didn’t yet know what the tool needed to do. With AI at my side, I could get a first version in minutes, then improve it in small steps as I learnt.\n\nI still had to listen, notice the mistakes, and decide what was worth trying next. But I could turn a question into a waveform selection, a comparison page, or a repeatable command while the problem was fresh in my mind. Without those tools, I don’t think I would have reached my virtual finish line.\n\nThe goal stayed simple: press play and hear one complete word, clearly enough to have a fair go at spelling it.", "url": "https://wpnews.pro/news/creating-word-recordings-with-ai-learning-where-to-cut", "canonical_source": "https://freshteapot.net/writing/creating-word-audio-with-ai/", "published_at": "2026-09-15 09:17:29+00:00", "updated_at": "2026-09-15 09:40:03.509879+00:00", "lang": "en", "topics": ["ai-tools", "generative-ai", "ai-products"], "entities": ["audio.cpp", "VoxCPM2", "Chatterbox", "OmniVoice", "ffmpeg", "OpenBMB", "ResembleAI"], "alternates": {"html": "https://wpnews.pro/news/creating-word-recordings-with-ai-learning-where-to-cut", "markdown": "https://wpnews.pro/news/creating-word-recordings-with-ai-learning-where-to-cut.md", "text": "https://wpnews.pro/news/creating-word-recordings-with-ai-learning-where-to-cut.txt", "jsonld": "https://wpnews.pro/news/creating-word-recordings-with-ai-learning-where-to-cut.jsonld"}}