Stem Separation Explained: How Music Source Separation Models Split a Song, With a Python Example

A technical explainer by Tania Alwan details how music source separation models split a finished stereo mix into vocals, drums, bass and other stems, framing it as an underdetermined inverse problem that deep learning has made practical. The piece walks through time-frequency masking with librosa, notes phase-reuse artifacts, and surveys waveform and hybrid architectures including Meta's Hybrid Transformer Demucs, measured by SDR on the MUSDB18 benchmark. It includes a pip-installable Demucs example, a residual-RMS check for comparing models, and a comparison of hosted services such as LALAL.AI, Moises and Voicestars.

By Tania Alwan Stem separation, also called music source separation, takes a finished stereo mix and estimates the individual parts that went into it: vocals, drums, bass and everything else. It powers karaoke apps, remix tools and AI cover workflows. It's also a nice example of an "impossible" inverse problem that deep learning made practical. This post covers how separation works, how quality is measured, and how to run an open-source model yourself in a few lines of Python. A stereo mix is, roughly, a sum of sources: mix t = vocals t + drums t + bass t + other t Given only mix , we want to estimate each term. Mathematically that's underdetermined: infinitely many combinations sum to the same signal. Models solve it by learning what each kind of source usually sounds like from large sets of multitrack recordings. The classic deep-learning approach works in the time-frequency domain. python import numpy as np import librosa y, sr = librosa.load "mix.wav", sr=44100, mono=True S = librosa.stft y, n fft=4096, hop length=1024 mag, phase = np.abs S , np.angle S a trained model would predict this; here it's a placeholder vocal mask = model.predict mag shape == mag.shape, values in 0, 1 vocals = librosa.istft vocal mask mag np.exp 1j phase , hop length=1024 Weakness: reusing the mixture phase limits quality. When two sources overlap in the same bin, a single mask can't fully separate them, which is where "watery" artefacts come from. Waveform models like the original Demucs operate directly on raw audio with a convolutional encoder-decoder, sidestepping the phase problem. Hybrid models combine a spectrogram branch and a waveform branch. Hybrid Transformer Demucs HT Demucs , released by Meta's research team, adds transformer layers between the two branches to model longer-range context. Other strong families include band-split architectures, which process different frequency bands with dedicated sub-networks. The open research community iterates on these quickly, often through public challenges. The standard metric is SDR signal-to-distortion ratio , in decibels; higher is better. Benchmarks typically use the MUSDB18 dataset, which contains multitrack songs split into vocals, drums, bass and other. Keep two caveats in mind: Always listen, not just read numbers. Demucs is open source and installable with pip. python -m pip install -U demucs demucs song.mp3 default model, 4 stems demucs --two-stems=vocals song.mp3 vocals + no vocals only Outputs are written to a separated/ folder, organised by model name and track. Check demucs --help for the model options available in your installed version. Practical notes: python import subprocess, pathlib for f in pathlib.Path "tracks" .glob " .wav" : subprocess.run "demucs", "--two-stems=vocals", str f , check=True If you're comparing models for a project, a simple, repeatable test beats impressions. python import soundfile as sf, numpy as np mix, = sf.read "song.wav" stems = sf.read f"separated/htdemucs/song/{s}.wav" 0 for s in "vocals","drums","bass","other" residual = mix - np.sum stems, axis=0 print "residual RMS:", np.sqrt np.mean residual 2 and why they happen Run it yourself when: Use a hosted service when: On the hosted side there are options like LALAL.AI, Moises and the Voicestars stems splitter https://www.voicestars.co/ai-stems-splitter , which outputs six stems vocals, drums, bass, guitars, piano, other . Pricing models differ credits, subscriptions, per-minute rates , so compare against your expected volume. Take a song you own, run demucs --two-stems=vocals on it, and listen to the outputs on headphones. Then compare against a hosted tool to see which artefacts each one leaves. If you want a browser comparison point, the Voicestars stems splitter https://www.voicestars.co/ai-stems-splitter is one option. Disclosure: this article was written in partnership with Voicestars.