{"slug": "heartmula-open-source-music-generation-model", "title": "HeartMuLa – Open-Source Music Generation Model", "summary": "The HeartMuLa project released a family of open-source music foundation models, including a music language model that generates songs from lyrics and tags with multilingual support, a high-fidelity music codec, and a lyrics transcription model. The project's latest HeartMuLa-oss-3B-happy-new-year version is currently the best open-source model for lyrics controllability and music quality, while the internal 7B version achieves performance comparable to Suno. All model weights are now available under the Apache 2.0 license, with online demo spaces launched on Hugging Face and ModelScope.", "body_md": "[Demo 🎶](https://heartmula.github.io/) | 📑 [Paper](https://arxiv.org/pdf/2601.10547)\n\n[HeartMuLa-oss-3B-happy-new-year 🤗](https://huggingface.co/HeartMuLa/HeartMuLa-oss-3B-happy-new-year) | [HeartMuLa-oss-3B-happy-new-year ](https://modelscope.cn/models/HeartMuLa/HeartMuLa-oss-3B-happy-new-year)\n\nHeartMuLa is a family of open sourced music foundation models including:\n\n- HeartMuLa: a music language model that generates music conditioned on lyrics and tags with multilingual support covering almost all languages.\n- HeartCodec: a 12.5 hz music codec with high reconstruction fidelity;\n- HeartTranscriptor: a whisper-based model specifically tuned for lyrics transcription; Check\n[this page](/HeartMuLa/heartlib/blob/main/examples/README.md)for its usage. - HeartCLAP: an audio–text alignment model that establishes a unified embedding space for music descriptions and cross-modal retrieval.\n\nBelow shows the experiment result of our oss-3B version compared with other baselines.\n\nOur latest internal version of HeartMuLa-7B achieves **comparable performance with Suno** in terms of musicality, fidelity and controllability.\n\n-\n🚀\n\n**10 Apr. 2026** We launched online demo spaces on\n\n[Hugging Face](https://huggingface.co/spaces/HeartMuLa/heartmula)and[ModelScope](https://www.modelscope.cn/studios/HeartMuLa/heartmula/). -\n🚀\n\n**13 Feb. 2026** We released our\n\n**HeartMuLa-oss-3B-happy-new-year** version. This version is currently the best open-sourced model in terms of lyrics controllability and music quality. We recommend using**HeartMuLa-oss-3B-happy-new-year** and**HeartCodec-oss-20260123** for music generation. -\n⚖️\n\n**03 Feb. 2026** We have released our\n\n[HeartMuLa-Benchmark](https://modelscope.cn/datasets/HeartMuLa/HeartMuLa-Benchmark)(referred to as**HeartBeats Benchmark** in our paper) as introduced in our[paper](https://arxiv.org/pdf/2601.10547). This benchmark comprises heterogeneous AI-generated lyrics and tags across diverse languages and genres, providing a rigorous and fair evaluation framework. -\n🚀\n\n**23 Jan. 2026** By leveraging Reinforcement Learning, we have continuously refined our model and are proud to officially release\n\n**HeartMuLa-RL-oss-3B-20260123**. This version is designed to achieve more precise control over styles and tags. Simultaneously, we are launching** HeartCodec-oss-20260123**, which optimizes audio decoding quality. -\n🫶\n\n**20 Jan. 2026**[Benji](https://github.com/benjiyaya)has created a wonderful[ComfyUI custom node](https://github.com/benjiyaya/HeartMuLa_ComfyUI)for HeartMuLa. Thanks Benji! -\n⚖️\n\n**20 Jan. 2026** License update: We update the license of this repo and all related model weights to\n\n**Apache 2.0**. -\n🚀\n\n**14 Jan. 2026**\n\nThe official release of**HeartTranscriptor-oss** and the first**HeartMuLa-oss-3B** version along with our**HeartCodec-oss**.\n\n- ⏳ Release scripts for inference acceleration and streaming inference. The current inference speed is around RTF\n$\\approx 1.0$ . - ⏳ Support\n**reference audio conditioning**,** fine-grained controllable music generation**,** hot song generation**. - ⏳ Release the\n**HeartMuLa-oss-7B** version. - ✅ Release inference code and pretrained checkpoints of\n\n**HeartCodec-oss, HeartMuLa-oss-3B, and HeartTranscriptor-oss**.\n\nWe recommend using `python=3.10`\n\nfor local deployment.\n\nClone this repo and install locally.\n\n```\ngit clone https://github.com/HeartMuLa/heartlib.git\ncd heartlib\npip install -e .\n```\n\nDownload our pretrained checkpoints from huggingface or modelscope using the following command:\n\n```\n# if you are using huggingface\nhf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'\nhf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'\nhf download --local-dir './ckpt/HeartCodec-oss' HeartMuLa/HeartCodec-oss-20260123\n\n# if you are using modelscope\nmodelscope download --model 'HeartMuLa/HeartMuLaGen' --local_dir './ckpt'\nmodelscope download --model 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year' --local_dir './ckpt/HeartMuLa-oss-3B'\nmodelscope download --model 'HeartMuLa/HeartCodec-oss-20260123' --local_dir './ckpt/HeartCodec-oss'\n```\n\nAfter downloading, the `./ckpt`\n\nsubfolder should structure like this:\n\n```\n./ckpt/\n├── HeartCodec-oss/\n├── HeartMuLa-oss-3B/\n├── gen_config.json\n└── tokenizer.json\n```\n\nTo generate music, run:\n\n```\npython ./examples/run_music_generation.py --model_path=./ckpt --version=\"3B\"\n```\n\nBy default this command will generate a piece of music conditioned on lyrics and tags provided in `./assets`\n\nfolder. The output music will be saved at `./assets/output.mp3`\n\n.\n\n-\nHow to specify lyrics and tags?\n\nThe model will load lyrics from the txt file\n\n`--lyrics`\n\nlink to (by default`./assets/lyrics.txt`\n\n). If you would like to use your own lyrics, just modify the content in`./assets/lyrics.txt`\n\n. If you would like to save your lyrics to another path, e.g.`my_awesome_lyrics.txt`\n\n, remember to input arguments`--lyrics my_awesome_lyrics.txt`\n\n.For tags it's basically the same.\n\n-\nCUDA out of memory?\n\nIf you have multi-GPUs (e.g. 2 4090s), we recommend placing the params of HeartMuLa and HeartCodec separately on different devices. You can do it by typing\n\n`--mula_device cuda:0 --codec_device cuda:1`\n\nIf you are running on a single GPU, use\n\n`--lazy_load true`\n\nso that modules will be loaded on demand and deleted once inference completed to save GPU memory.\n\nAll parameters:\n\n`--model_path`\n\n(required): Path to the pretrained model checkpoint`--lyrics`\n\n: Path to lyrics file (default:`./assets/lyrics.txt`\n\n)`--tags`\n\n: Path to tags file (default:`./assets/tags.txt`\n\n)`--save_path`\n\n: Output audio file path (default:`./assets/output.mp3`\n\n)`--max_audio_length_ms`\n\n: Maximum audio length in milliseconds (default: 240000)`--topk`\n\n: Top-k sampling parameter for generation (default: 50)`--temperature`\n\n: Sampling temperature for generation (default: 1.0)`--cfg_scale`\n\n: Classifier-free guidance scale (default: 1.5)`--version`\n\n: The version of HeartMuLa, choose between [`3B`\n\n,`7B`\n\n]. (default:`3B`\n\n) #`7B`\n\nversion not released yet.`--mula_device/--codec_device`\n\n: The device where params will be placed. Both are set to`cuda`\n\nby default. You can use`--mula_device cuda:0 --codec_device cuda:1`\n\nto explicitly place different modules to different devices.`--mula_dtype/--codec_dtype`\n\n: Inference dtype. By default is`bf16`\n\nfor HeartMuLa and`fp32`\n\nfor HeartCodec. Setting`bf16`\n\nfor HeartCodec may result in the degradation of audio quality.`--lazy_load`\n\n: Whether or not to use lazy loading (default: false). If turned on, modules will be loaded on demand to save GPU usage. Recommended format of lyrics and tags:\n\n```\n[Intro]\n\n[Verse]\nThe sun creeps in across the floor\nI hear the traffic outside the door\nThe coffee pot begins to hiss\nIt is another morning just like this\n\n[Prechorus]\nThe world keeps spinning round and round\nFeet are planted on the ground\nI find my rhythm in the sound\n\n[Chorus]\nEvery day the light returns\nEvery day the fire burns\nWe keep on walking down this street\nMoving to the same steady beat\nIt is the ordinary magic that we meet\n\n[Verse]\nThe hours tick deeply into noon\nChasing shadows,chasing the moon\nWork is done and the lights go low\nWatching the city start to glow\n\n[Bridge]\nIt is not always easy,not always bright\nSometimes we wrestle with the night\nBut we make it to the morning light\n\n[Chorus]\nEvery day the light returns\nEvery day the fire burns\nWe keep on walking down this street\nMoving to the same steady beat\n\n[Outro]\nJust another day\nEvery single day\n```\n\nRegarding tags, check this [issue](https://github.com/HeartMuLa/heartlib/issues/17) for reference.\nOur different tags are comma-separated without spaces as illustrated below:\n\n```\npiano,happy,wedding,synthesizer,romantic\n```\n\nThis repository is licensed under the Apache 2.0 License.\n\n```\n@misc{yang2026heartmulafamilyopensourced,\n      title={HeartMuLa: A Family of Open Sourced Music Foundation Models}, \n      author={Dongchao Yang and Yuxin Xie and Yuguo Yin and Zheyu Wang and Xiaoyu Yi and Gongxi Zhu and Xiaolong Weng and Zihan Xiong and Yingzhe Ma and Dading Cong and Jingliang Liu and Zihang Huang and Jinghan Ru and Rongjie Huang and Haoran Wan and Peixu Wang and Kuoxi Yu and Helin Wang and Liming Liang and Xianwei Zhuang and Yuanyuan Wang and Haohan Guo and Junjie Cao and Zeqian Ju and Songxiang Liu and Yuewen Cao and Heming Weng and Yuexian Zou},\n      year={2026},\n      eprint={2601.10547},\n      archivePrefix={arXiv},\n      primaryClass={cs.SD},\n      url={https://arxiv.org/abs/2601.10547}, \n}\n```\n\nIf you are interested in HeartMuLa, feel free to reach us at [heartmula.ai@gmail.com](mailto:heartmula.ai@gmail.com)\n\nWelcome to join us through [discord](https://discord.gg/BKXF5FgH) or Wechat group.\n\nScan the QR code on the left to join our Wechat group. If it expires, feel free to raise an issue to remind us of updating.\n\nIf the number of group members exceeds 200, joining the group via directly scanning the QR code is restricted by WeChat. In this case, scan our team member's QR code on the right and send a request writing **HeartMuLa Group Invite**. We will invite you into the group manually.", "url": "https://wpnews.pro/news/heartmula-open-source-music-generation-model", "canonical_source": "https://github.com/HeartMuLa/heartlib", "published_at": "2026-05-31 11:27:54+00:00", "updated_at": "2026-05-31 11:47:05.942628+00:00", "lang": "en", "topics": ["generative-ai", "artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["HeartMuLa", "HeartCodec", "HeartTranscriptor", "HeartCLAP", "Suno", "Hugging Face", "ModelScope", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/heartmula-open-source-music-generation-model", "markdown": "https://wpnews.pro/news/heartmula-open-source-music-generation-model.md", "text": "https://wpnews.pro/news/heartmula-open-source-music-generation-model.txt", "jsonld": "https://wpnews.pro/news/heartmula-open-source-music-generation-model.jsonld"}}