{"slug": "kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-to", "title": "Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI", "summary": "Kyutai and the Mirelo team released MuScriptor, an open-weight decoder-only Transformer for multi-instrument music transcription to MIDI. The model, trained on synthetic and real recordings across three stages, achieves state-of-the-art results on the D_Test benchmark, with a Multi F1 score of 48.2% for the large variant after reinforcement learning post-training. The release includes three model sizes (103M, 307M, 1.4B) on Hugging Face, with inference code under MIT license and weights under CC BY-NC 4.0.", "body_md": "Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release ** MuScriptor** to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres.\n\n**This article explains how MuScriptor works, what the benchmarks show, and how to run it.**\n\n**What is MuScriptor?**\n\nAt its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme.\n\nThe release ships three weight variants on Hugging Face. Their sizes are `small`\n\n(103M), `medium`\n\n(307M, default), and `large`\n\n(1.4B). The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted.\n\n**How the Three-Stage Pipeline Works**\n\nMuScriptor’s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last.\n\n- Pre-training uses\n**D<sub>Synth</sub>**, roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes them during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations. - Fine-tuning uses\n**D<sub>Real</sub>**, an internal set of 170,000 recordings. Together they total more than 11,000 hours with aligned note annotations. Most alignments come from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a maximum time-dilation factor. - Reinforcement learning post-training uses\n**D<sub>RL</sub>**, 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward sums three F-scores: onset, frame, and offset. As a result, the model learns to favor cleaner transcriptions.\n\n**Performance**\n\nFor evaluation, the research team use **D<sub>Test</sub>**, 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the `mir_eval`\n\nlibrary. Among them, Multi F1 is strictest, since it also requires the correct instrument.\n\nThe table below traces each training stage against the YourMT3+ baseline, using the large (~1.3B) model.\n\n| Model (D<sub>Test</sub>) | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 |\n|---|---|---|---|---|---|\n| YourMT3+ (baseline) | 32.5 | 45.5 | 17.8 | 41.4 | 21.9 |\n| MuScriptor · D<sub>Synth</sub> | 34.5 | 48.9 | 16.1 | 21.0 | 16.2 |\n| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> | 54.4 | 69.3 | 42.3 | 43.3 | 41.6 |\n| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> + D<sub>RL</sub> | 60.4 | 73.3 | 49.0 | 50.2 | 48.2 |\n\nClearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding D<sub>Real</sub> then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing.\n\nCross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals.\n\n**Getting Started**\n\nInstallation takes one command, and inference streams note events directly.\n\n``` python\n# pip install muscriptor   (or: uv add muscriptor)\nfrom pathlib import Path\nfrom muscriptor import TranscriptionModel\n\n# Downloads the default \"medium\" variant (also accepts \"small\" / \"large\")\nmodel = TranscriptionModel.load_model()\n\n# Stream note events; optionally condition on known instruments\nfor event in model.transcribe(\"audio.wav\", instruments=[\"acoustic_piano\", \"drums\"]):\n    print(event)   # NoteStartEvent / NoteEndEvent / ProgressEvent\n\n# Or write a MIDI file directly\nPath(\"out.mid\").write_bytes(model.transcribe_to_midi(\"audio.wav\"))\n```\n\nFor the released models, keep `cfg_coef`\n\nat 1, since they are already RL post-trained. Additionally, `uvx muscriptor serve`\n\nlaunches a browser web UI with a live piano roll.\n\n**Use Cases with Examples**\n\nBecause the output is standard MIDI, many workflows open up:\n\n**Producers** can extract a MIDI bassline from a mix, then re-voice it in a DAW.**Musicologists** can convert historical recordings into editable scores for analysis.**MIR researchers** can feed transcriptions into chord or key recognition systems.**Educators** can build practice tools showing a live piano roll during playback.**Developers** can transcribe only drums by passing instrument conditioning.\n\n**Strengths and Weaknesses**\n\n**Strengths:**\n\n- Trained on 170k real recordings spanning classical to heavy metal.\n- Open weights plus MIT-licensed inference code, in three size variants.\n- Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D<sub>Test</sub>.\n- Instrument conditioning customizes output and stabilizes cross-segment predictions.\n- A streaming API emits note events and MIDI, alongside a browser web UI.\n\n**Weaknesses:**\n\n- Weights are CC BY-NC 4.0, so commercial deployment is restricted.\n- The tokenizer drops velocity and cannot represent overlapping same-pitch, same-instrument notes.\n- Onset and offset accuracy stay lower on chorals and similar styles.\n- The large model wants a GPU for practical speed.\n- The 5-second segment size limits long-range context and inference speed.\n\nCheck out the ** Paper, GitHub Repo **and\n\n**.**\n\n[Model Weights](https://huggingface.co/MuScriptor)**Also, feel free to follow us on**\n\n**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://www.aidevsignals.com/)\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)", "url": "https://wpnews.pro/news/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-to", "canonical_source": "https://www.marktechpost.com/2026/07/10/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-instrument-music-transcription-to-midi/", "published_at": "2026-07-10 20:21:08+00:00", "updated_at": "2026-07-10 20:49:14.089610+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["Kyutai", "Mirelo", "MuScriptor", "Hugging Face", "YourMT3+"], "alternates": {"html": "https://wpnews.pro/news/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-to", "markdown": "https://wpnews.pro/news/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-to.md", "text": "https://wpnews.pro/news/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-to.txt", "jsonld": "https://wpnews.pro/news/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-to.jsonld"}}