Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI

Kyutai and the Mirelo team released MuScriptor, an open-weight decoder-only Transformer for multi-instrument music transcription to MIDI. The model, trained on synthetic and real recordings across three stages, achieves state-of-the-art results on the D_Test benchmark, with a Multi F1 score of 48.2% for the large variant after reinforcement learning post-training. The release includes three model sizes (103M, 307M, 1.4B) on Hugging Face, with inference code under MIT license and weights under CC BY-NC 4.0.

Automatic Music Transcription AMT converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release MuScriptor to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres. This article explains how MuScriptor works, what the benchmarks show, and how to run it. What is MuScriptor? At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme. The release ships three weight variants on Hugging Face. Their sizes are small 103M , medium 307M, default , and large 1.4B . The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted. How the Three-Stage Pipeline Works MuScriptor’s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last. - Pre-training uses D<sub Synth</sub , roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes them during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations. - Fine-tuning uses D<sub Real</sub , an internal set of 170,000 recordings. Together they total more than 11,000 hours with aligned note annotations. Most alignments come from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a maximum time-dilation factor. - Reinforcement learning post-training uses D<sub RL</sub , 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward sums three F-scores: onset, frame, and offset. As a result, the model learns to favor cleaner transcriptions. Performance For evaluation, the research team use D<sub Test</sub , 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the mir eval library. Among them, Multi F1 is strictest, since it also requires the correct instrument. The table below traces each training stage against the YourMT3+ baseline, using the large ~1.3B model. | Model D<sub Test</sub | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 | |---|---|---|---|---|---| | YourMT3+ baseline | 32.5 | 45.5 | 17.8 | 41.4 | 21.9 | | MuScriptor · D<sub Synth</sub | 34.5 | 48.9 | 16.1 | 21.0 | 16.2 | | MuScriptor · D<sub Synth</sub + D<sub Real</sub | 54.4 | 69.3 | 42.3 | 43.3 | 41.6 | | MuScriptor · D<sub Synth</sub + D<sub Real</sub + D<sub RL</sub | 60.4 | 73.3 | 49.0 | 50.2 | 48.2 | Clearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding D<sub Real</sub then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing. Cross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals. Getting Started Installation takes one command, and inference streams note events directly. python pip install muscriptor or: uv add muscriptor from pathlib import Path from muscriptor import TranscriptionModel Downloads the default "medium" variant also accepts "small" / "large" model = TranscriptionModel.load model Stream note events; optionally condition on known instruments for event in model.transcribe "audio.wav", instruments= "acoustic piano", "drums" : print event NoteStartEvent / NoteEndEvent / ProgressEvent Or write a MIDI file directly Path "out.mid" .write bytes model.transcribe to midi "audio.wav" For the released models, keep cfg coef at 1, since they are already RL post-trained. Additionally, uvx muscriptor serve launches a browser web UI with a live piano roll. Use Cases with Examples Because the output is standard MIDI, many workflows open up: Producers can extract a MIDI bassline from a mix, then re-voice it in a DAW. Musicologists can convert historical recordings into editable scores for analysis. MIR researchers can feed transcriptions into chord or key recognition systems. Educators can build practice tools showing a live piano roll during playback. Developers can transcribe only drums by passing instrument conditioning. Strengths and Weaknesses Strengths: - Trained on 170k real recordings spanning classical to heavy metal. - Open weights plus MIT-licensed inference code, in three size variants. - Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D<sub Test</sub . - Instrument conditioning customizes output and stabilizes cross-segment predictions. - A streaming API emits note events and MIDI, alongside a browser web UI. Weaknesses: - Weights are CC BY-NC 4.0, so commercial deployment is restricted. - The tokenizer drops velocity and cannot represent overlapping same-pitch, same-instrument notes. - Onset and offset accuracy stay lower on chorals and similar styles. - The large model wants a GPU for practical speed. - The 5-second segment size limits long-range context and inference speed. Check out the Paper, GitHub Repo and . Model Weights https://huggingface.co/MuScriptor Also, feel free to follow us on and don’t forget to join our Twitter https://x.com/intent/follow?screen name=marktechpost and Subscribe to 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ . Wait are you on telegram? our Newsletter https://www.aidevsignals.com/ now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/wbash1wF6efRj8G58