{"slug": "i-built-a-local-ai-music-studio", "title": "I Built a Local AI Music Studio", "summary": "A developer built Miso, a self-hosted local music generation and remix studio layered on top of audio.cpp that unifies multiple audio model families behind a single interface. The tool supports full song generation, section repainting, stem separation, voice conversion, and audio-to-MIDI transcription, with a guided prompt builder that translates one set of UI controls into each model's expected syntax. Testing showed ACE-Step's repaint route is influenced far more by surrounding audio and lyrics than by text prompts, unlike the standard text-to-music path.", "body_md": "I have a habit of starting projects because I want one very specific thing.\n\nThen somewhere along the way I apparently decide, \"Well, if I'm already doing this...\"\n\nAnd suddenly the tiny experiment has a database, Docker Compose stack, job queue, model manager, audio editor, waveform player, and enough GPU models to eat 40 GB of disk space.\n\nSo, anyway.\n\nI made **Miso**.\n\nMiso is a self-hosted local music generation and remix studio built on top of [audio.cpp](https://github.com/0xShugo/audio.cpp).\n\nThe original idea was pretty simple: I wanted a nice interface for experimenting with local music models without constantly dropping back to command-line tools, juggling generated files, or trying to remember which ridiculous combination of parameters made the song I actually liked.\n\nIt got slightly out of hand.\n\nThere are already plenty of ways to generate a song from a text box.\n\nThat part is fun, but the thing I kept wanting was everything that comes **after** generation.\n\nWhat if I like the song except for eight seconds in the middle?\n\nWhat if I want the vocals by themselves?\n\nWhat if I want to change the singer?\n\nWhat if I want the same melody arranged in a completely different style?\n\nWhat if I want to turn something into MIDI and keep screwing with it somewhere else?\n\nThat became the real idea behind Miso.\n\nA project doesn't just hold the final song. It keeps the takes, prompts, lyrics, stems, scores, transcriptions, and the lineage between them, so I can experiment without losing track of how I got there.\n\nMiso currently exposes several different music and audio model families through audio.cpp because they are good at very different things.\n\nACE-Step 1.5 is the main workhorse. It handles full song generation, covers, and repainting sections of existing tracks.\n\nYuE2 can generate songs too, but it does something I find especially interesting: it plans an ABC music score first and can give that score back to you.\n\nMiniMax Music 3 and HeartMuLa are alternate song generators with their own prompting styles.\n\nStable Audio 3 handles instrumental generation and sound effects.\n\nThen there are separate models for stem separation, voice conversion, and audio-to-MIDI transcription.\n\nThe funny part is that they absolutely do **not** accept prompts the same way.\n\nOne model wants a production-style caption.\n\nAnother wants a short description plus comma-separated tags.\n\nACE-Step wants style information in one place, BPM and key as separate parameters, and lyrics somewhere else.\n\nSo Miso has a guided builder that translates the same set of UI controls into whatever syntax the selected model actually expects.\n\nI wanted to be able to think about \"upbeat synthpop, female vocal, 120 BPM\" instead of remembering which API field each model decided should contain \"120 BPM.\"\n\nOne of the features Miso is most built around is repainting.\n\nYou load a take, drag across part of its waveform, and replace only that section.\n\nSo instead of regenerating a whole song because one chorus went weird, you can select that section and try again.\n\nACE-Step's repaint route turned out to have some quirks that were interesting to figure out, too.\n\nThe surrounding music influences the replacement much more strongly than the text prompt does. Lyrics have more control over a vocal repaint than telling the prompt to suddenly turn the section into something wildly different.\n\nThat wasn't something I wanted to guess about, so I ended up testing it.\n\nAt one point I ran deliberately opposite prompts through the route and compared the results. The normal text-to-music route reacted dramatically. Repaint barely cared.\n\nWhich is one of the reasons I enjoy projects like this. Half of building the interface becomes figuring out what these models *actually* do rather than what I assume they do from an API description.\n\nMiso can also split a track into stems.\n\nDepending on the model, that can mean:\n\nOnce the stems exist, they get their own mixer.\n\nYou can solo them, mute them, change their levels, swap a vocal using a voice conversion model, mix everything back together, and save the result as another take.\n\nThat led to another rabbit hole: voice conversion.\n\nRVC sounds better but only has a small set of packaged voices.\n\nVevo2 has lower output bandwidth, but it can use basically any other vocal reference in the project.\n\nSo both stayed.\n\nI would rather expose two tools with different tradeoffs than pretend one is objectively the \"correct\" choice.\n\nThe current release, 1.4.0, connects MIDI transcription to YuE2's score support.\n\nPreviously, Miso could already:\n\nThe pieces just weren't connected.\n\nNow they are.\n\nYou can separate a song, transcribe the vocal stem, turn those note events into an ABC melody, inspect the detected tempo and key, correct them if necessary, and feed the score into YuE2 with a completely different style.\n\nSo you can take the melody from a recording and use it as the basis for a new arrangement.\n\nI tested that path with a known melody instead of just listening and declaring, \"yeah, sounds about right.\"\n\nI generated a simple version of *Twinkle Twinkle Little Star*, ran it back through separation, transcription, and score conversion, and recovered the expected pitches across the recording.\n\nRhythm is less perfect than pitch, and real vocals can produce odd little tied or split notes, but the result is usable by the model.\n\nThat was one of those features where suddenly several earlier pieces of the project made much more sense together.\n\nA local music generation and remix studio. You bring a prompt or a song, and Miso gives you a real workspace for generating, remixing, splitting, and finishing music with models that run on your own machine. Nothing is sent anywhere.\n\nMiso runs on [audio.cpp](https://github.com/0xShug0/audio.cpp), a C++ inference runtime for\naudio models. Miso is the studio around it: projects that persist, a history of every take\nand a record of exactly how each clip was made so you can change one thing and try again.\n\nMiso installs models, keeps your projects and audio, generates with ACE-Step from a guided prompt builder with lyrics written for you if you want them, repaints a section of a track, covers a take, holds any two takes against each other, splits a song into stems you can mix and export, writes full 48 kHz stereo songs with YuE2 and hands back the score…\n\nThe whole thing runs as two Docker containers.\n\nOne runs Miso itself.\n\nThe other runs audio.cpp and owns the GPU models.\n\nThe browser never talks directly to audio.cpp. It talks to Miso, and Miso talks to the inference backend.\n\nThat separation became useful almost immediately.\n\nProjects, imports, playback, exports, and editing still work if the model backend is stopped.\n\nIt also means the audio.cpp server can live on another machine with the GPU while Miso runs somewhere else.\n\nThe app itself uses React, TypeScript, Vite, Node, Hono, and SQLite.\n\nI deliberately kept most ordinary audio editing in the browser rather than turning everything into another server job.\n\nThings like trimming, fading, gain changes, normalization, splitting, WAV conversion, and MP3 export do not need a giant AI model involved.\n\nSometimes software can just... do the normal thing.\n\nWild concept, I know.\n\nI did sneak one language model feature in.\n\nMiso can optionally connect to an OpenAI-compatible API or a local llama.cpp server to help write lyrics or expand a music prompt.\n\nBut it is completely separate from the music models.\n\nNothing gets silently rewritten.\n\nIt gives you a suggestion, shows it to you, and you decide whether to use it.\n\nThat was important to me because I don't want an assistant \"improving\" a prompt behind my back and then making it impossible to figure out why two generations were different.\n\nIf you accept an expanded prompt, Miso keeps both the original and the version that actually ran.\n\nNot really about calling them.\n\nThat part is usually the easy bit.\n\nThe interesting problems were things like:\n\nAnd, of course, learning that \"the documentation says this option does X\" and \"the model meaningfully responds to X\" are not always the same statement.\n\nThat last one is probably applicable to a lot more than music models.\n\nMiso is open source, and right now it has mostly been tested on my own Linux machine with an RTX 4090 laptop GPU.\n\nSo I am absolutely not going to pretend I have tested every GPU, driver, operating system, or model combination in existence.\n\nBut it has grown from \"I want a good way to play around with local music models\" into something I actually think is pretty awesome.\n\nAnd that is my favorite kind of project.\n\nThe kind where I started building a tool because I wanted to experiment with something...\n\n...and then ended up building the experiment itself.", "url": "https://wpnews.pro/news/i-built-a-local-ai-music-studio", "canonical_source": "https://dev.to/sizzlebop/i-built-a-local-ai-music-studio-3fb9", "published_at": "2026-09-20 21:30:07+00:00", "updated_at": "2026-09-20 21:54:16.021146+00:00", "lang": "en", "topics": ["ai-tools", "generative-ai", "ai-products", "developer-tools"], "entities": ["Miso", "audio.cpp", "ACE-Step 1.5", "YuE2", "MiniMax Music 3", "HeartMuLa", "Stable Audio 3"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-local-ai-music-studio", "markdown": "https://wpnews.pro/news/i-built-a-local-ai-music-studio.md", "text": "https://wpnews.pro/news/i-built-a-local-ai-music-studio.txt", "jsonld": "https://wpnews.pro/news/i-built-a-local-ai-music-studio.jsonld"}}