Show HN: Duetto – pretrained music transformer that plays a piano with you

duo-f-blues.mp4 #

Play a MIDI keyboard and a transformer plays a second piano with you, in real time. It listens like a partner: your key, the chord you play, your pulse, your phrases. It comps while you play and takes the melody when you rest.

The model is the pretrained Anticipatory Music Transformer (Lakh MIDI), running on a Jetson; the laptop handles MIDI, sound and the piano roll.

duo-you-comp-it-solos.mp4 #

Above: you play low chords, the model improvises over them (the it solos preset). Files in samples/duo.

ssh -t jetson sudo jetson_clocks   # after every Jetson reboot
./run_duet.sh                      # on the laptop

run_duet.sh restarts everything: the model server on the Jetson, the SSH tunnel, FluidSynth, main.py on the keyboard and the piano roll. Closing the piano roll stops the duo and saves it in generated/.

Without a Jetson: python -m music_transformer.server in one terminal, python main.py in another (Ctrl+C stops and saves).

Setup #

• Jetson (Orin Nano, JetPack 6): the repo in~/Ai-Duet with a venv made with--system-site-packages (NVIDIA's torch), the model as safetensors in~/models/music-medium-800k , power mode MAXN_SUPER, andssh jetson working with a key.
• Laptop : FluidSynth with a General MIDI soundfont (SOUNDFONT=path/to.sf2 ./run_duet.sh for a sampled grand), and a venv withpip install mido python-rtmidi pygame imageio-ffmpeg .
• Code changes : the Jetson does not pull from git; copy the files over (scp music_transformer/*.py jetson:~/Ai-Duet/music_transformer/ ) and run./run_duet.sh again. The session tests need torch: runpython -m unittest tests.test_duo_session there.
• Logs :~/amt_server.log on the Jetson;duet.log ,fluidsynth.log ,pianoroll.log in/tmp on the laptop.

The top of the piano roll says what the model hears (key, chord, tempo) and what it is doing. Keys: F fullscreen, K show or hide the controls, Esc close.

Controls are sliders (drag them; hold Shift to drag finely) and steppers (click the arrows). The wheel or the arrow keys over a control turn it, and the wheel elsewhere scrolls the panel. A double click resets a control, and the bottom of the panel says what the control under the mouse does. A control is dimmed while what it shapes is not happening.

• AI mind - Presets: duet ,call & response ,it comps (it plays the harmony, you improvise),it solos (you play the harmony, it improvises),free
  • Musical controls: density, harmony, role, dialogue, groove, risk, focus, key follow
  • Range: lowest ,highest octave
  • Scale: root andscale force it into a scale (C major, D harmonic minor, F minor pentatonic...) instead of your key
  • Tempo: bpm fixes its tempo instead of following yours,metronome clicks it
• Presets:
• Your sound ,AI sound : instrument, volume, pan, envelope, reverb and delay, in FluidSynth
• Advanced : every knob of the model, described inKNOBS inmusic_transformer/duo.py , where the defaults are too

Knobs and sound are kept in ~/.config/ai-duet/.

PolyForm Noncommercial 1.0.0: use it, change it and share it freely for anything noncommercial (personal, study, research, education, hobby). Commercial use is not allowed. The model itself is the Anticipatory Music Transformer, under its own license.