Show HN: Claude optimizes mel-band roformer to run 4x faster on Mac

A developer released fast-melroformer, an open-source project that runs the 228M-parameter Mel-Band RoFormer vocal-separation checkpoint at about 4x real time on an 8 GB M3 MacBook Air, up from 1.08x for the stock PyTorch fp32 model on the GPU (MPS). The speedups came from moving RoPE and RMSNorm off fp32, folding output gates into the qkv matmul, custom Metal kernels, and running the 12 transformer blocks on the Neural Engine via Core ML, with tiled attention lifting throughput from 3.61x to 4.19x. Output matched the fp32 run at 56.5 dB SDR over a 3-minute clip, though one quiet passage matched at 40 dB, and a 45-minute TV episode now takes about 11 minutes instead of 42.

Fast voice separation on Apple Silicon. This runs a Mel-Band RoFormer source separation model on the Neural Engine and the GPU, at about 4x real time on an 8 GB M3 MacBook Air. It uses an unmodified public checkpoint vocals mel band roformer.ckpt , 228M parameters . Over a 3 minute clip the output matches the fp32 run at 56.5 dB SDR, which you can't hear. A 45 minute TV episode takes about 11 minutes instead of 42. Measured end to end on the same 3 minute clip: | step | speed vs real time | |---|---| | stock model, PyTorch fp32 on the GPU MPS | 1.08x | | fp16 | 1.35x | | RoPE and RMSNorm moved off fp32, output gates folded into the qkv matmul | 2.19x | | custom Metal kernels for the attention glue | 2.62x | | the 12 transformer blocks on the Neural Engine Core ML | 3.61x | | time attention split into small tiles so it fits the Neural Engine | 4.19x | The biggest surprise was the rotary embeddings. They ran in fp32 and rebuilt their angle tables on every call, which ate 30 to 46% of attention time on MPS. Fixing that is a small patch fastmbr/patch.py that helps any Mac user of this model. Quantization would not have helped speed. 88% of the weights sit in the mask estimators, and those take only 4% of the time. NOTES.md has every measurement, including the ideas that failed. You need an Apple Silicon Mac, macOS 15 and uv https://github.com/astral-sh/uv . git clone https://github.com/aipdv/fast-melroformer && cd fast-melroformer uv sync .venv/bin/audio-separator --download model only -m vocals mel band roformer.ckpt --model file dir ~/models/melroformer bench/guard.sh .venv/bin/python bench/ane stack.py convert builds the Neural Engine model, once .venv/bin/python -m fastmbr.separate episode.mp4 -o out/ You get episode vocals.wav and episode other.wav . The first run compiles the model for the Neural Engine, which takes about 5 minutes. After that it loads in 2 seconds. --engine gpu uses only the GPU path 2.6x . --engine ref16 and --engine ref32 run the unmodified model for comparison. - I only tested on one machine: an 8 GB M3 MacBook Air with no fan. It slows down when it gets hot, so long files run a bit below 4x. - The Neural Engine runs in fp16. The full clip matches at 56.5 dB, but one quiet passage matched at 40 dB, which is right around where a difference could start to be heard. Listen to quiet scenes if quality matters to you. - bench/guard.sh runs one job at a time and stops it if RAM gets low. Running two of these jobs at once froze my 8 GB Mac. - coremltools warns that it hasn't been tested with this torch version. It worked anyway. - fastmbr/patch.py : GPU fixes on top of audio-separator's model. - fastmbr/metal.py : custom Metal kernels. - fastmbr/ane.py : the transformer rewritten for the Neural Engine. - fastmbr/hybrid.py : Neural Engine for the transformer, GPU for the rest. - fastmbr/separate.py : the command line tool. - bench/ : the scripts behind every number. - Checkpoint: KimberleyJSN/melbandroformer https://huggingface.co/KimberleyJSN/melbandroformer MIT - Reference implementation: python-audio-separator https://github.com/nomadkaraoke/python-audio-separator MIT - Architecture: Mel-Band RoFormer for Music Source Separation https://arxiv.org/abs/2310.01809 , Wang, Lu and Won