{"slug": "custom-voices-with-qwen3-tts", "title": "Custom Voices With Qwen3-TTS", "summary": "Armin Ronacher documented using Qwen3-TTS, Alibaba's Qwen text-to-speech model family, to create custom narrator voices for a dungeon-crawler game he is building with his children, after initially using Kokoro-82M and switching because Qwen3 supports voice cloning. Ronacher cloned a voice from 28 seconds of his own PyAI Conf talk recording, transcribed with Whisper, and blended it with a VoiceDesign-generated voice using the qwen-tts 0.1.1 Python package; he noted Qwen3's 1.7B-Base model has no documented way to clone a voice and direct it with instructions, unlike the 1.7B-CustomVoice model with nine built-in speakers and the 1.7B-VoiceDesign model. Ronacher warned that a short, imperfect conference recording was enough to produce a halfway convincing imitation of a person, adding that \"when then presented to you on a low quality phone call, it won't be very hard to trick your grandmother.", "body_md": "With my kids I’m currently building a little [dungeron\ncrawler](https://github.com/mitsuhiko/dungeon-crawler) and we’re trying to\n[learn a bit about game\ndevelopment](https://mitsuhiko.github.io/dungeon-crawler/learn/) along the way.\nHowever given that we live in the modern times of increasing capabilities we\nwanted to get a narrator and voice over.\n\nBut we also wanted it to not sound like everyone’s standard AI voices so I\nlooked into what options there are on the table.  After some investigation I\nnarrowed down to using [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) which\nuses voice embeddings for custom voices.\n\nWith a bit of creativity you can also do more with those custom voices than appears initially. In case this is of interest to you, this post just covers what we learned and with some examples. You can probably point your agent to it and it can do something with it.\n\nQwen3 is not the only TTS model in town, and it wans’t even my first choice.  We\ninitially used [Kokoro](https://huggingface.co/hexgrad/Kokoro-82M).  However\nQwen3 supports voice cloning and we wanted to have it feel a bit more unique.\n\nIn practice it’s a bit tricker though. Qwen3 ships three kinds of models:\n\n`1.7B-Base` (and 0.6B): clones a voice from a few seconds of reference audio`1.7B-CustomVoice`: comes with nine built-in speakers and instructions` 1.7B-VoiceDesign`: invents a voice from a text description\nThe base model clones either in context or from the speaker embedding alone but there is no documented way to clone a voice and direct it with instructions. That means unlike the custom voice and voice design models you cannot say “speak sad” or “whisper” with a cloned base model.\n\nAt present the official options for a custom voice are fine-tuning Base on a single speaker or to make a clip with VoiceDesign and clone it with Base. That is not all that bad, but it means instructions are gone.\n\nFor this blog post I took 28 seconds of me from my talk at [PyAI\nConf](https://www.youtube.com/watch?v=8RHYyRUxVrA), cut between two pauses so\nthat it starts and ends on a full sentence, and transcribed it with Whisper.\nThe transcript is really only needed for baking but to compare against an\nalready existing voice it’s useful to have it say the same thing as the original\nrecording.\n\nNote that a conference talk recording is not ideal but it works anyways. Which is also a good reminder that you really don’t need a particularly clean recording (or a very long one) to get a halfway convincing version of a person. Particularly when then presented to you on a low quality phone call, it won’t be very hard to trick your grandmother.\n\nThe first thing you can do is to blend that recording with another voice.  For\ninstance you can use VoiceDesign to make up a voice from a description.  For this blog\npost the designed voice was created like this, with Qwen’s own Python package\n[`qwen-tts`](https://pypi.org/project/qwen-tts/) (` pip install qwen-tts`, I used\n0.1.1), which the rest of the code in this post uses too:\n\n``` python\nimport torch\nfrom qwen_tts import Qwen3TTSModel\n\nkw = dict(device_map=\"mps\", dtype=torch.bfloat16, attn_implementation=\"sdpa\")\ndesign = Qwen3TTSModel.from_pretrained(\"Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign\", **kw)\nwavs, sr = design.generate_voice_design(\n    text=open(\"voices/transcript.txt\").read(), language=\"English\",\n    instruct=\"A man in his forties with a warm, natural, medium-low voice, speaking \"\n             \"clearly and calmly at an easy conversational pace, like someone reading \"\n             \"aloud to a friend.\",\n)\n```\n\nYou can see how the two different voices behave, particularly when blended together:\n\n[Interactive demo: see the post on the website.](https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices/)\n\nWhat Base takes from a recording when cloning from the embedding alone is a speaker embedding which is basically 2048 numbers out of a speaker encoder that was trained. The same voice gets similar numbers and different voices get different ones. Which btw, is also a convenient way to do speaker detection from recordings if you want to! It’s not perfect, but it’s quite helpful for a single audio track recording to tell two speakers apart if you do some basic slicing and dicing (more on that below).\n\nBecause it is just a vector, you can do arithmetic on it. For instance a weighted average of two embeddings is a spaker that is somewhere between two voices:\n\n```\nbase = Qwen3TTSModel.from_pretrained(\"Qwen/Qwen3-TTS-12Hz-1.7B-Base\", **kw)\n\ndef fingerprint(path):\n    prompt = base.create_voice_clone_prompt(ref_audio=path, x_vector_only_mode=True)\n    return prompt[0].ref_spk_embedding\n\ndef blend(a, b, t):\n    a, b = a.float(), b.float()\n    mixed = (1 - t) * a + t * b\n    return (mixed / mixed.norm() * ((1 - t) * a.norm() + t * b.norm())).to(torch.bfloat16)\n\narmin, designed = fingerprint(\"voices/armin.flac\"), fingerprint(\"voices/designed.flac\")\n```\n\nHere is the same sentence in eleven blends:\n\nNote that it stops sounding like me quite quickly, but it retains some of my features. And it for sure ends up with creating a unique voice regardless. I found some better successes when ensuing that the two voices have similar spectograms (eg: similar “recording setups”). In this recording you can hear that one voice picked up by the lapel mic was a bit hollow compared to the designed voice.\n\nIf you like a a voice that comes out, you can bake it back into a new recording that you can then use as reference:\n\n``` python\nimport soundfile as sf\n\nmixed = blend(armin, designed, 0.5)\naudio, rate = say(open(\"voices/transcript.txt\").read(), mixed)  # say() is shown further down\nsf.write(\"voices/final.flac\", audio, rate)\nfinal = fingerprint(\"voices/final.flac\")\n```\n\nThe baked voice does not come back exactly as the blend but it makes handling it easier if you want to. It’s just one voice reference file.\n\nSo on the question of using those embeddings for speaker recognition: how good\ndoes that work?  I took an episode of [State of Agentic\nCoding](https://www.youtube.com/watch?v=qCpY4K9jtOg), the podcast Ben and I\nrecord, cut it into 2-second windows one second apart and compared each with an\nembedding of me form the conference.\n\nSomewhat surprising, but maybe not too surprising using my conference recording clip as the reference this does not work at all with Qwen3. If instead I take the 20 seconds where I introduce myself at the start of the episode as the reference, it gets 98% of the rest right.\n\nOn the other hand using a model that was trained for speaker verification\ninstead does manage to pick me up from the conference recording clip.  For\ninstance [ECAPA](https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb),\nfinds me from the conference clip in 97% of the windows.\n\nHere are 20 minutes of that podcast, classified by speaker with the different approaches:\n\nBut the embeddings a very useful for the voice cloning!  But how to give them\ninstructions?  Base and CustomVoice manipulate the speaker the same way: one\nvector inserted into the talker’s input sequence.  In CustomVoice that vector is\na row of the embedding table.  In Base it is the x-vector from the speaker\nencoder, which only Base has.  The generation code takes either\n(`voice_clone_prompt` is a regular argument of `generate`), and the transformer\nweights of the two models are nearly identical.  So take the embedding with\nBase, then hand it to CustomVoice instead of one of its speakers:\n\n```\ncv = Qwen3TTSModel.from_pretrained(\"Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice\", **kw)\n\ndef say(text, xvec, instruct=None, language=\"English\"):\n    ids = cv._tokenize_texts([cv._build_assistant_text(text)])\n    told = [cv._tokenize_texts([cv._build_instruct_text(instruct)])[0]\n            if instruct else None]\n    voice = {\"ref_code\": [None], \"ref_spk_embedding\": [xvec],\n             \"x_vector_only_mode\": [True], \"icl_mode\": [False]}\n    codes, _ = cv.model.generate(\n        input_ids=ids, instruct_ids=told, voice_clone_prompt=voice,\n        languages=[language], non_streaming_mode=True,\n        **cv._merge_generate_kwargs(temperature=0.7, subtalker_temperature=0.7),\n    )\n    wavs, rate = cv.model.speech_tokenizer.decode([{\"audio_codes\": codes[0]}])\n    return wavs[0], rate\n\nwav, rate = say(\"I really thought this would work.\", armin,\n                \"Speak in a sad, sorrowful tone, voice low and heavy, on the verge of tears.\")\n```\n\nPick a speaker for the slot, a model and an instruction:\n\nInstructions work on cloned voices, but less reliably than on the built-in speakers, and it depends on the emotion, but honestly, it came out better than I anticipated. Definitely good enough to have some true custom voices for a voice over that don’t necessarily sound like something, someone else has in their game.\n\nIf a voice is a point in a 2048-dimensional space and blending works we can maybe do some other funny math on it. The obvious one is to try to see if some math can be used to manipulate gender.\n\nTo find it I had VoiceDesign create 14 different voices for men and women each. When you then take the average woman minus the average man you can then move any voice along that direction:\n\n```\nwomen = torch.stack([fingerprint(f\"design/f{i}.wav\") for i in range(14)]).float()\nmen = torch.stack([fingerprint(f\"design/m{i}.wav\") for i in range(14)]).float()\nmiddle = (women.mean(0) + men.mean(0)) / 2\naxis = women.mean(0) - men.mean(0)\naxis = axis / axis.norm()\n\ndef move(xvec, t):\n    \"\"\"Put the voice at position t along the gender axis, keep the rest.\"\"\"\n    x = xvec.float()\n    return (x + (t - (x - middle) @ axis) * axis).to(torch.bfloat16)\n```\n\nHere is the result for my clone:\n\nAnyways, that’s that. If you want to play around with Qwen3 TTS I hope this was of some help, and maybe you got some ideas of what to do with the voice embeddings. Use it for good, not evil.", "url": "https://wpnews.pro/news/custom-voices-with-qwen3-tts", "canonical_source": "https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices/", "published_at": "2026-10-29 00:00:00+00:00", "updated_at": "2026-10-29 19:16:40.705018+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-tools", "ai-safety"], "entities": ["Qwen3-TTS", "Qwen", "Armin Ronacher", "Kokoro-82M", "Whisper", "qwen-tts", "PyAI Conf"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/custom-voices-with-qwen3-tts", "markdown": "https://wpnews.pro/news/custom-voices-with-qwen3-tts.md", "text": "https://wpnews.pro/news/custom-voices-with-qwen3-tts.txt", "jsonld": "https://wpnews.pro/news/custom-voices-with-qwen3-tts.jsonld"}}