# Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI

> Source: <https://www.marktechpost.com/2026/07/10/kyutai-releases-muscriptor-an-open-weight-decoder-only-transformer-for-multi-instrument-music-transcription-to-midi/>
> Published: 2026-07-10 20:21:08+00:00

Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release ** MuScriptor** to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres.

**This article explains how MuScriptor works, what the benchmarks show, and how to run it.**

**What is MuScriptor?**

At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme.

The release ships three weight variants on Hugging Face. Their sizes are `small`

(103M), `medium`

(307M, default), and `large`

(1.4B). The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted.

**How the Three-Stage Pipeline Works**

MuScriptor’s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last.

- Pre-training uses
**D<sub>Synth</sub>**, roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes them during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations. - Fine-tuning uses
**D<sub>Real</sub>**, an internal set of 170,000 recordings. Together they total more than 11,000 hours with aligned note annotations. Most alignments come from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a maximum time-dilation factor. - Reinforcement learning post-training uses
**D<sub>RL</sub>**, 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward sums three F-scores: onset, frame, and offset. As a result, the model learns to favor cleaner transcriptions.

**Performance**

For evaluation, the research team use **D<sub>Test</sub>**, 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the `mir_eval`

library. Among them, Multi F1 is strictest, since it also requires the correct instrument.

The table below traces each training stage against the YourMT3+ baseline, using the large (~1.3B) model.

| Model (D<sub>Test</sub>) | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 |
|---|---|---|---|---|---|
| YourMT3+ (baseline) | 32.5 | 45.5 | 17.8 | 41.4 | 21.9 |
| MuScriptor · D<sub>Synth</sub> | 34.5 | 48.9 | 16.1 | 21.0 | 16.2 |
| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> | 54.4 | 69.3 | 42.3 | 43.3 | 41.6 |
| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> + D<sub>RL</sub> | 60.4 | 73.3 | 49.0 | 50.2 | 48.2 |

Clearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding D<sub>Real</sub> then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing.

Cross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals.

**Getting Started**

Installation takes one command, and inference streams note events directly.

``` python
# pip install muscriptor   (or: uv add muscriptor)
from pathlib import Path
from muscriptor import TranscriptionModel

# Downloads the default "medium" variant (also accepts "small" / "large")
model = TranscriptionModel.load_model()

# Stream note events; optionally condition on known instruments
for event in model.transcribe("audio.wav", instruments=["acoustic_piano", "drums"]):
    print(event)   # NoteStartEvent / NoteEndEvent / ProgressEvent

# Or write a MIDI file directly
Path("out.mid").write_bytes(model.transcribe_to_midi("audio.wav"))
```

For the released models, keep `cfg_coef`

at 1, since they are already RL post-trained. Additionally, `uvx muscriptor serve`

launches a browser web UI with a live piano roll.

**Use Cases with Examples**

Because the output is standard MIDI, many workflows open up:

**Producers** can extract a MIDI bassline from a mix, then re-voice it in a DAW.**Musicologists** can convert historical recordings into editable scores for analysis.**MIR researchers** can feed transcriptions into chord or key recognition systems.**Educators** can build practice tools showing a live piano roll during playback.**Developers** can transcribe only drums by passing instrument conditioning.

**Strengths and Weaknesses**

**Strengths:**

- Trained on 170k real recordings spanning classical to heavy metal.
- Open weights plus MIT-licensed inference code, in three size variants.
- Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D<sub>Test</sub>.
- Instrument conditioning customizes output and stabilizes cross-segment predictions.
- A streaming API emits note events and MIDI, alongside a browser web UI.

**Weaknesses:**

- Weights are CC BY-NC 4.0, so commercial deployment is restricted.
- The tokenizer drops velocity and cannot represent overlapping same-pitch, same-instrument notes.
- Onset and offset accuracy stay lower on chorals and similar styles.
- The large model wants a GPU for practical speed.
- The 5-second segment size limits long-range context and inference speed.

Check out the ** Paper, GitHub Repo **and

**.**

[Model Weights](https://huggingface.co/MuScriptor)**Also, feel free to follow us on**

**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)

**and Subscribe to**

[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**

[our Newsletter](https://www.aidevsignals.com/)

[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)