By Peter Marelas. A native macOS app that runs Fish Audio's Drama 3 text-to-speech model locally on Apple silicon. The 35B mixture-of-experts backbone and the codebook tower run on hand-written Metal compute kernels that read the released safetensors weights in place. The DAC audio codec runs through Core ML. No MLX, no PyTorch at runtime.
Built with Fish Audio. The model weights are under the Fish Audio Community License; cloning a real person's voice without their consent is not permitted.
Drama-Studio-Promo-10MB.mp4 #
- Apple silicon Mac. Checked generation requires macOS 26 or later and an installed on-device speech model for the script language. Unchecked preview targets macOS 15. Intel Macs are not supported by the engine.
- About 49 GB of free disk for the model, codec and setup caches. Allow more space for generated audio, saved projects and app updates.
- 64 GB or more of unified memory is recommended. This is not a verified minimum. Generation has been tested on an M3 Max with 128 GB. An M1 Mac Mini with 16 GB failed twice with a GPU out-of-memory error before producing audio, despite paging. See the hardware test . 32 GB and 64 GB Macs remain untested. The engine maps about 41 GB of model weights.
- A display that can fit a 900 by 640 point studio window.
- An internet connection for the initial downloads and update checks. Speech generation runs locally after setup.
The setup memory override tests the warning only. It does not simulate a lower-memory Mac or establish that generation works on one.
Download DramaStudio-<version>.dmg from thelatest release .
2.
Open the disk image and drag Drama Studio to Applications.
3.
Open Drama Studio from Applications. The release is not notarized, so macOS blocks the first open. Open System Settings, go to Privacy & Security, scroll to Security, and click Open Anyway next to the message about Drama Studio. Confirm with your
password. If you use Terminal, you can runxattr -dr com.apple.quarantine /Applications/DramaStudio.app instead.
4.
Read the license terms in the setup window and click I Agree . Fish
Audio licenses the model under the Fish Audio Community License, and
Drama Studio generates nothing until you agree.
5.
Wait for the downloads. The window shows the progress of each file, the total, and the time left:
- the Drama 3 weights from Hugging Face, 41.5 GB, pinned to revision
e0a9218; - the Core ML voice codec from the release, 1.5 GB. You can , or quit and open the app again. The download continues where it stopped. Each file is checked against a SHA-256 value that ships with the app before it is kept.
- the Drama 3 weights from Hugging Face, 41.5 GB, pinned to revision
Wait 30 seconds to a minute while the first load tunes the GPU kernels for your Mac. 8. Click Generate to speak your first line.
If you already downloaded the model with hf download fishaudio/drama-3,
Drama Studio uses that copy and skips the 41.5 GB download. If you are below
64 GB of memory, the setup window says so before anything downloads. You can
continue at your own risk.
Use Clear Script in the toolbar or Script menu to empty the editor. For an imported document, Clear Section empties only the selected section. Existing takes stay available and may need regeneration after the text changes. Use Edit > Undo to restore cleared text. Clearing is disabled during generation.
Screenplay Mode requires a speaker name, a colon and a space before dialogue:
Alex: Hello there.
Sam: Good morning.
A line without a name continues the previous speaker. Text before the first named speaker belongs to Narrator. Up to 16 speakers are supported. If the script has no named dialogue, turn Screenplay Mode off for plain narration, or use the help button beside the error for the format and examples.
Drama Studio checks GitHub for a newer stable release at launch and downloads it in the background. Automatic checks and downloads are on by default. Turn them off in Settings > Updates, or use Drama Studio > Check for Updates to check manually at any time.
Once the download is verified, Settings > Updates offers Install and Restart. Save your project and finish generation or API requests before confirming. Later leaves the running app unchanged; click Review Update in Settings > Updates when ready. Check for Updates opens the same tab. There is no separate Software Update window. Quitting normally does not install an update. A deferred update is downloaded again on the next automatic or manual check.
Updates require the app to be installed in a folder you can write to, such as
Applications or your user's Applications folder. Apps running from a mounted
DMG, App Translocation, or a Swift build directory cannot update themselves.
The updater changes only the app bundle; models, voices, and projects stay in
place. It uses the release's DramaStudio-<version>.dmg and SHA256SUMS, checks
the bundle identity, version, and minimum macOS requirement, verifies its code
signature, and requires the
same signing identity for Developer ID builds. Ad-hoc builds rely on GitHub
HTTPS and the published checksum for authenticity.
After restart, setup compares the installed codec's recorded archive SHA-256 with the digest shipped in the updated app. Matching codecs are reused even when the release URL changes. A changed digest or missing codec models triggers a verified download and installation. Older installs without a recorded digest need one verified download to establish it.
- Script editor with import of
.txt,.md,.rtf,.rtfd,.html,.docx,.srtand.vtt, drag and drop, and a screenplay mode that turnsName: linedialogue into Drama 3 speaker tags (up to 16 speakers). - Takes : Generate or Generate All produces a fresh take each time. A
scrolling grid shows take durations, notes, export choices and outdated audio.
Other controls include an Expressiveness slider under Advanced (one control
over temperature, top-p and top-k, with the vendor's defaults in the middle
and the Hugging Face Space's sampling at the top), and an
insert menu for performance cues such as
[laughs],[sighs]and<whisper>…</whisper>. Each chunk's frame cap follows the length of its text. Normal generation checks each decoded sentence with local, audio-only speech recognition before playback or reuse as history. It also checks each complete adjacent pair before releasing the next sentence, so a bad join retries only the unplayed sentence within the same attempt limit. Each raw sentence must pass both recognizers with the existing native confidence checks. For a pair of those exact verified sentences, the native checker may mark only the first nonterminal word of the second sentence uncertain while both full transcripts still match. That narrow boundary case uses the raw sentence's confidence; ambiguous word positions, missing confidence, or other uncertain words fail. Arbitrary-file checks have no such generated-sentence evidence. These bounded windows cover every generated join within one request; they do not establish a global transcript for a long recording or certify joins between separate requests. Missing or uncertain words trigger a bounded retry; exhaustion fails instead of creating a completed take. Checked generation requires macOS 26, an installed Apple on-device speech model for the script's language, and the bundled independent recognizer. Both audio-only recognizers must find every scripted word before a sentence is accepted. A disagreement retries generation; an unavailable checker stops generation. These checks reduce word errors but cannot guarantee recognition accuracy or acting quality. Older takes without word-check history are kept for playback and export, but do not automatically become voice references or retake history. Voice auditions always use checked generation. Recognition checks word coverage, not acting quality or perfect cue execution. - Document narration : a PowerPoint deck, a PDF or an EPUB opens as
sections, one per slide, page or chapter, and narrates section by section
with the same voice throughout. Exports are the whole narration, one file
per section with a
manifest.csv, SRT or WebVTT captions, and for a deck a slide narration package. Projects save and reopen as.dramaprojbundles (seeDocuments ). - Pronunciation rules in Settings rewrite words before the model hears them, in the app and through the API.
- Voice cloning from one or more reference clips (any format AVFoundation reads) plus their transcripts, kept in a local voice library. To reuse a generated voice, select a take and clickSave Voice to Library besideUse for Export . Name the voice and save it. The app selects it for future generations, including after you edit the script. This works with Model default takes and individual document sections. Complete, checked takes containing one speaker can be saved; outdated script text does not prevent it.Save Session Voice in Cast saves the retained session voice instead.
- Streaming playback as chunks finish, scrubbing, a speed control from 0.75x to 1.5x, the sentence being spoken highlighted in the editor, and export to WAV or M4A.
- A local HTTP API and an MCP server , so scripts and agents can speak through the same engine and voice library (seeAgents and programs ).
- Menu bar residency : closing the window leaves Drama Studio in the menu bar (the waveform-and-microphone icon) with the API still serving. The menu shows the model state, the API address, and the current job. ⌘Q quits.
The system requirements also apply to source builds. Install the following tools and model files:
- macOS 15 or newer, Xcode 26 command line tools.
- The model in the Hugging Face cache:
uvx --from huggingface_hub hf download fishaudio/drama-3
- The Core ML codec models, built once from
codec.pth:
cd tools && uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python "torch==2.7.1" coremltools "numpy<2.4" einops soundfile
.venv/bin/python -I ../Sources/DramaEngine/CodecConversion/convert_codec.py
They land in ~/Library/Application Support/DramaStudio/Codec/ (encoder
809 MB in fp32, decoder 353 MB in fp16).
python3 tools/bootstrap_word_recognition.py # downloads and checks pinned framework and local speech models
swift build -c release
python3 tools/stage_word_recognition.py --bin .build/release
tools/make_app.sh # assembles dist/DramaStudio.app, with drama-mcp and drama-say in Contents/MacOS
open dist/DramaStudio.app
Command line:
The app build stages the recognition runtime automatically. CLI builds and tests
need the bootstrap and staging steps above. The English and multilingual models
use about 975 MB on disk; recognition runs locally with no script hints. English
uses small.en; other supported languages use small. Only the selected model
is loaded. Recognition requires both an installed Apple model and a supported
independent model language. Older native-only checks remain recorded in project
files, but their audio is excluded from automatic checked-reference reuse.
Run the retained wrong-word and complete-ending controls with
swift test -c release --filter LocalWordVerifierTests. On macOS 26 these controls
fail if required local assets are missing; they do not skip that failure.
tools/make_cli.sh assembles a signed, relocatable CLI runtime in
dist/drama-cli, including the helper, framework, resource bundle, models and
licenses. Set DS_VERIFY_NATIVE_WORD_CHECKS=1 when running either packaging
script to require the installed native model and exercise the full dual check
on the retained complete-ending and wrong-word recordings. Packaging always
tests the independent runtime. The full offline check can also be rerun with
drama check-words <WAV> <script> <accepted|rejected>, or with the app executable's
--check-words flag. These commands do not load the speech generation model.
.build/release/drama tune # once per Mac, about 30 s; the app does it on first load
.build/release/drama generate --text "Hello there." --out frames.npy
.build/release/drama bench # add --untuned to compare against the hand-written schedule
.build/release/drama serve # the HTTP API without the app; --port 8780, --no-token
.build/release/drama-say "Hello there." # speaks through the running app, like say; see From the terminal
DRAMA_PREVIEW_ENGINE=1 runs the app or drama serve on a mock engine that
produces placeholder tones, for working on everything but the model.
The app looks for the weights in this order: the folder chosen in Settings,
its own download in ~/Library/Application Support/DramaStudio/Models/drama-3-e0a9218/,
then the newest Hugging Face cache snapshot. A build from tools/make_app.sh
has no codec download, so its setup window shows the Python conversion command
above when the codec models are missing; tools/make_release.sh builds the
release DMG and the codec zip and writes the zip's URL and SHA-256 into the
app. DRAMA_STUDIO_HOME=<folder> moves the downloads, the codec, the tuning
and the settings into one folder and ignores the Hugging Face cache, to try a
first run without touching your own. DRAMA_STUDIO_MEMORY_GB=16 shows the
low-memory warning on a Mac that has enough. tools/pin_model_plan.py --check
compares the pinned download plan with Hugging Face.
Importing a .pptx, .pdf or .epub (File > Import Script or Document,
drag and drop, or opening the file with the app) opens it as sections: a
deck's slides, with the speaker notes as the text and the slide text when a
slide has no notes; a PDF's pages; an EPUB's chapters. The list shows each
section's label, a preview, its duration once narrated, a skip box and a
speaker picker. The editor edits the selected section.
- Generate narrates every section that is not skipped, in order, one request per section, cast with that section's speaker's voice. The speaker's previous section goes along as history, and with no library voice cast the speaker's first narrated section is also the voice, the way takes and the live route already carry a voice. Narrate This Section generates one section again. Click a take to audition it, then click Use for Export to choose it for the narration. The Export badge identifies the chosen take; auditioning another take does not change it.
- Editing text or changing a speaker or voice marks the affected audio as Needs regeneration .Generate Changed Sections updates only missing or outdated sections. Export asks before writing outdated or incomplete audio. Earlier takes remain available, including after a project is reopened.
- Export writes the whole narration, Per Section (numbered files and a
manifest.csv), Captions as SRT or WebVTT, timed from the frames each sentence took, and for a deck a Slide Narration Package:slide-NN.wavper narrated slide and aREADME.txton adding them with Insert > Audio. The app does not change the deck. - Save Project writes a
.dramaprojbundle with the document or script, settings, voice assignments, session voices, and every take with its original text, voice assignments, codes, and audio. Library assignments still refer to voices in the local library. Save updates the current project;Save Project As creates a copy. Format 2 projects require this version of Drama Studio. Older projects still open, with a verification warning for audio whose generation history was not stored. - Import, Open Project, and Quit offer Save, Discard Changes, or Cancel when work is unsaved. A failed or cancelled save leaves the current session open. After editing s for two seconds, the app writes a recovery draft in the background; drafts wait until generation stops. The next launch offers to restore a draft left by an interrupted session. Recovery does not replace the last manually saved project.
- Flatten to Script turns the sections into one plain script and removes their takes from the session. Its confirmation offers to save a project copy first.
On the five-slide sample deck (python-pptx, slide 3 without notes), the
real engine at the default settings narrated 24.3 s of audio in 34.5 s
including the model load. Whisper small.en heard slide 4, whose notes read
"Revenue grew twelve percent. Costs stayed flat, and margins improved.", as
"Revenue grew 12%. Cost stayed flat, and margins improved." Two slides lost
words: slide 1 spoke only "Welcome, everyone." of its two sentences, and
slide 3 only its title. Each was one engine chunk in that build (the splitter joined a
sentence under 24 bytes to the next), and with "Stream audio while
generating" on, a chunk that comes out short is kept rather than generated
again. With the setting off, Narrate This Section gave slide 1 both sentences
and slide 3 "How it works Import". That is one run per slide, not a
measurement. Because narration is exported, document sections now always
generate in the retrying sentence mode whatever the streaming setting says;
playback of a section starts when the section is done rather than three
seconds in.
eval/reliability/REPORT.md is a 330-run study of whether every sentence
comes out as speech: 33 conditions (short and long lines, numbers,
abbreviations, all caps, performance cues, multi-speaker tags, Spanish,
French and German, a cloned voice, both expressiveness extremes, and two
control conditions), five seeds each, in full and balanced.
| rows | full | balanced |
|---|---|---|
| plain prose under the shipped defaults | 130 of 130 | 126 of 130 |
| performance-cue lines | 16 of 20 | 14 of 20 |
| retries off (control) | 1 of 5 | 3 of 5 |
| reference 300-byte chunking (control) | 7 of 10 | 9 of 10 |
Paired by seed, balanced is never right where full is wrong, and wrong where
full is right in 4 of 130 prose pairs. Performance cues are the one area
where both modes drop content a few times in twenty; [laughs] can also turn
into a laughter loop that the app's text-derived frame cap ends near 8 s.
Reproduce with eval/reliability.py (slices under eval/reliability/slices/).
The balanced mode was removed after this study, so the engine has one
precision, full; the column stays as the record of why.
The cue dropouts were then investigated on eight cue lines, five seeds each,
eleven variants (eval/cues/, section "Performance cues: investigation" of
the report). The model stops at a cue that follows a sentence end, whatever
the chunking; longer chunks lose more. Three guards now ship: a laughter loop
is detected in the code stream (the main tokens cycle with a period of 17 or
more, which the 16-frame repetition redraw forces) and the chunk is
regenerated; a chunk that stops without eos is regenerated when it spoke
under its expected duration; and the frame cap grows by 1.5 s per [cue] so
a laugh or a is not cut off. Every sentence present went from 44 of 55
runs to 38 of 40 on the cue lines; on six prose lines at five seeds the
old and new rules score 30 of 30 and 29 of 30, the one difference being a
line that both rules retry and where the new rule's random path lost. The
Space's sampling (temperature 1.0, top-p 0.9, top-k off) was then run over
the whole study (section "The Space's sampling on prose"): no prose row
lost in 130, every sentence of the 20 cue lines, 5 retries instead of 19,
the two-sentence line reaching the end of speech on its first attempt with
retries off in 5 of 5, and 0.007 more word error rate on prose (standard
error 0.004), mostly abbreviations and a few slurred words. It is the top of
the Expressiveness slider and the API's expressiveness: 1; the vendor's
sampling stays the default. The Insert Cue menu says what remains: about one
take in twenty still drops the clause after a cue, so listen before
exporting.
While Drama Studio is open, including when only its menu bar icon is left, it
serves an HTTP API on http://127.0.0.1:8780. drama serve serves the same
API without the app. The API listens on the loopback interface only. Settings
holds the port, a switch for starting the server with the app, and the token.
The menu bar has Start and Stop. When the port is taken, the menu shows
"API: Port 8780 is already in use."
Every route except /v1/health needs the bearer token from
~/Library/Application Support/DramaStudio/api-token, created the first time
the server starts. The token keeps web pages, which can reach localhost, from
driving the engine. drama serve --no-token turns it off.
TOKEN=$(cat ~/Library/Application\ Support/DramaStudio/api-token)
AUTH="Authorization: Bearer $TOKEN"
The speech request follows the SGLang Drama 3 cookbook:
| Route | Does |
|---|---|
GET /v1/health |
{status, model_loaded, engine: "metal" or "mock", busy, tuned, version} .tuned isnull until the model loads. While a generation runs it addsestimate: {elapsed_seconds, remaining_seconds, frames_done, frames_expected} ;remaining_seconds isnull once the last sentence is in its silence tail or a retry runs. |
GET /v1/models |
The one model, fishaudio/drama-3 , with its snapshot id. |
POST /v1/audio/speech |
Speaks input . Returns the audio file, or a PCM stream. |
POST /v1/tts |
The same, with the text in text . |
GET /v1/audio/speech/stream |
Speaks the input query parameter as Server-Sent Events. SeeStreaming . |
POST /v1/audio/speech/live |
Speaks text while the client is still sending it, and streams the audio back on the same connection. See Streaming . |
GET /v1/voices |
The voice library: id, name, transcript, seconds, source file. |
POST /v1/voices |
Clones a voice: JSON {name, transcript, audio_base64 or audio_path} , or multipart with afile part plusname andtranscript fields. |
DELETE /v1/voices/{id} |
Deletes a voice. |
GET /v1/takes/{id} |
One of the last 20 takes: {id, input, text, seconds, frames, codes_base64, previous_take_ids} . |
The speech body takes these fields:
| Field | Meaning |
|---|---|
input |
The text. Performance cues such as [laughs] and<|speaker:1|> tags work as in the app. |
voice |
A library voice, by id or name. |
references |
Clips to clone on the fly: [{audio_path or audio_base64, text}] . |
speaker_voices |
Library voices per speaker for a multi-speaker script: {"0": "Narrator", "1": "<id>"} , with no gaps from speaker 0. |
response_format |
wav (default, 16-bit),pcm (raw 16-bit little-endian), orm4a . |
stream |
true streams completed, checked chunks. |
expressiveness |
0 to 1, default 0.5: the app's Expressiveness slider. 0.5 is the vendor's sampling, 1 the Hugging Face Space's. |
seed |
A fixed sampler seed, for repeatable takes. |
character |
A voice designed from a description instead of a clip: {name, description, scene?} . |
note |
A director's note, such as "softer and warmer" . The line is generated again as one turn, with the note as a direction and as a cue before every sentence. |
previous_takes |
Earlier takes of the line, oldest first: [{take_id}] or[{text, codes_base64}] . The last two are context. With novoice orreferences , the first one is the voice. |
apply_pronunciation |
true (default) rewritesinput with the pronunciation rules in~/Library/Application Support/DramaStudio/pronunciation.json , the same rules the app uses, before it is spoken;false speaks it as sent. |
A request takes at most one of voice, references, speaker_voices, and
character. Every generation is a take: the response carries its id in
X-Drama-Take-Id, and the server keeps the last 20. A take_id in
previous_takes also brings the takes that take was made after, so naming a
line's latest take carries its first take, the voice. Leave out input to
retake the line of the last named take, with its voice and character.
codes_base64 is the take's codes as 16-bit little-endian integers, codebook
by codebook, as GET /v1/takes/{id} returns them.
A file response carries X-Drama-Frames, X-Drama-Seconds, and
X-Drama-Generation-Seconds. The engine runs one generation at a time. A
request that arrives while another runs, through the API or in the window, gets
429 with Retry-After. A bad request gets 400 with a plain-text message
that names the field.
curl -sS localhost:8780/v1/health
curl -sS -H "$AUTH" localhost:8780/v1/audio/speech \
-d '{"input": "Hello there. [laughs] Good to see you.", "voice": "Narrator"}' -o hello.wav
curl -sS -H "$AUTH" localhost:8780/v1/tts -d '{"text": "The alias route."}' -o alias.wav
curl -sS -H "$AUTH" localhost:8780/v1/audio/speech \
-d '{"input": "Cloned on the fly.", "references": [{"audio_path": "/path/to/clip.wav", "text": "What the clip says."}]}' -o clone.wav
curl -sS -H "$AUTH" -D headers.txt localhost:8780/v1/audio/speech \
-d '{"input": "[low and menacing] Pray ye never find out.", "character": {"name": "Captain", "description": "gravelly old male voice, weathered and slow"}}' -o captain.wav
TAKE=$(awk 'tolower($1) == "x-drama-take-id:" {print $2}' headers.txt | tr -d '\r')
curl -sS -H "$AUTH" localhost:8780/v1/audio/speech \
-d "{\"note\": \"softer and warmer\", \"previous_takes\": [{\"take_id\": \"$TAKE\"}]}" -o captain-retake.wav
All three streaming routes send completed chunks when retries are enabled. The engine validates each attempt before releasing audio, so a rejected attempt can be replaced for any voice.
The audio is always 16-bit little-endian mono at 44.1 kHz, as the
X-Drama-Format: pcm_s16le_mono and X-Drama-Sample-Rate: 44100 headers say.
stream: true and the event stream also send X-Drama-Take-Id and
X-Drama-Estimated-Seconds before the first audio.
PCM. stream: true on POST /v1/audio/speech answers with
Transfer-Encoding: chunked and raw PCM in the HTTP chunks. A chunk boundary
means nothing; concatenate the chunks. Every chunk holds a whole number of
2048-sample frames. Pipe it into a player:
curl -sSN -H "$AUTH" localhost:8780/v1/audio/speech \
-d '{"input": "A long script, read as it is generated.", "stream": true, "response_format": "pcm"}' |
ffplay -f s16le -ar 44100 -ch_layout mono -nodisp -autoexit -
Server-Sent Events. GET /v1/audio/speech/stream takes its fields as query
parameters: input (required), voice, expressiveness,
seed, and character_name, character_description and character_scene
for a designed voice. apply_pronunciation=false turns off the pronunciation
rules, as in the speech body. It answers with Content-Type: text/event-stream
and these events:
| Event | data |
|---|---|
audio |
Base64 of the next PCM bytes, a whole number of frames. |
sentence |
{"index", "text", "seconds"} after the last audio of each sentence. |
done |
{"take_id", "frames", "seconds"} once the take is complete. |
error |
{"message"} , after which the stream ends. |
The route still needs the bearer token, and the server sends no CORS headers,
so a browser's EventSource, which cannot send an Authorization header, does
not work with the token on. Read the events with fetch and a stream reader
from a page the server already trusts, or with any HTTP client.
curl -sSN -H "$AUTH" "localhost:8780/v1/audio/speech/stream?input=Hello%20there.%20Good%20to%20see%20you."
Live text input. POST /v1/audio/speech/live is for an agent that writes
text a piece at a time. It takes voice, expressiveness,
seed and the character_* fields as query parameters, and the text as the
request body:
- Send the body with
Transfer-Encoding: chunked, or with aContent-Lengththat covers everything you will send. The server reads it as it arrives. - The body is UTF-8 text in lines ended by
\n(or\r\n). The server acts on a line once its newline arrives. - The server joins the lines with spaces and speaks each sentence once a
sentence end (
.,!,?,;,…, or a CJK one, optionally followed by a closing quote or bracket) arrives. Sentences are cut with the same splitter asinput, so<|speaker:N|>tags and cues work. - The pronunciation rules rewrite each sentence before it is spoken, unless
the query has
apply_pronunciation=false. - An empty line speaks the text held back so far, finished or not. The end of the body does the same, and a client may half-close its side of the connection to end the body.
- Each sentence is a take, and the session's takes carry the voice and the
history: the first take is the voice of every later sentence when no
voiceis given, and the last two are context. - The response is chunked PCM as for
stream: true. SendAccept: text/event-streamto get the events above instead, withdonecarrying{"sentences", "seconds", "take_ids"}. The response ends once the body has ended and its last sentence is spoken. - The session holds the engine from the request to the end of the response,
so other requests get
429meanwhile.
tools/live_speech.py sends each line of its standard input as one chunk as
soon as it reads it, and writes the PCM to standard output:
some-agent | tools/live_speech.py --voice Narrator |
ffplay -f s16le -ar 44100 -ch_layout mono -nodisp -
tools/live_speech.py --events < script.txt # prints each event with its arrival time
curl -T - also speaks a piped body, but it holds the piped lines back before
sending them: on the mock engine its first audio came 1.7 s later than the
same feed through tools/live_speech.py.
Manage voices:
curl -sS -H "$AUTH" localhost:8780/v1/voices
curl -sS -H "$AUTH" localhost:8780/v1/voices -F file=@clip.wav -F name=Narrator -F "transcript=What the clip says."
curl -sS -H "$AUTH" -X DELETE localhost:8780/v1/voices/<id>
tools/api_smoke.sh [base-url] runs all of the above against a running server
and checks each answer.
drama-mcp is an MCP server over stdio (protocol 2025-06-18) that calls the
HTTP API. tools/make_app.sh puts it at
DramaStudio.app/Contents/MacOS/drama-mcp; swift build puts it at
.build/debug/drama-mcp. When nothing answers, it opens Drama Studio and waits
up to 90 seconds for /v1/health. It looks for the app in DRAMA_APP_PATH,
then the app that contains it, then /Applications/DramaStudio.app, then the
repository's dist/DramaStudio.app. DRAMA_API_URL overrides
http://127.0.0.1:8780. It reads the token from the same file.
| Tool | Does |
|---|---|
synthesize_speech |
text , with optionalvoice orcharacter ({name, description, scene?} ),output_path ,expressiveness , andplay . Writes a WAV, by default to~/Library/Application Support/DramaStudio/Output/<timestamp>.wav . Returns the path, seconds, frames, generation seconds, andtake_id .play plays it withafplay . |
retake_with_note |
take_id andnote , with optionaloutput_path andplay . Generates that take's line again in the same voice with the note. Returns whatsynthesize_speech returns, with the newtake_id . |
list_voices |
The voice library. |
add_voice |
Clones audio_path with itstranscript undername . |
delete_voice |
Deletes a voice by id . |
studio_status |
Health, the model snapshot, and the API address. |
There is no MCP tool for streaming or live text input. An MCP tool call is one
request and one result, so it cannot carry audio while text is still being
written; an agent that wants that calls POST /v1/audio/speech/live itself.
Register it with Claude Code:
claude mcp add drama-studio -- /Applications/DramaStudio.app/Contents/MacOS/drama-mcp
Or with Claude Desktop, in ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"drama-studio": {
"command": "/Applications/DramaStudio.app/Contents/MacOS/drama-mcp"
}
}
}
drama-say speaks through the running app the way /usr/bin/say speaks
through macOS. It sends POST /v1/audio/speech, finds and opens the app the
way drama-mcp does, and loads no model of its own. tools/make_app.sh puts
it at DramaStudio.app/Contents/MacOS/drama-say.
drama-say "Hello there. [laughs] Good to see you."
drama-say -v Narrator -o hello.wav -f chapter.txt
drama-say --character "Captain: gravelly old male voice" --verbose "Pray ye never find out."
drama-say --retake "$TAKE" --note "softer and warmer"
With no -o, it plays the line as it generates: it sends stream: true and
plays each completed chunk when it arrives. Startup time depends on model
, the script, the voice and retries; it is not a fixed delay. -o FILE writes a WAV, M4A or PCM file, chosen by
the extension, and -o - writes raw PCM to standard output. With no text and
no -f, it reads the text from standard input. -v '?' lists the voice
library. --verbose prints the take id that --retake takes. drama-say --help lists every option.
It prints nothing on success. It exits with 64 on a bad command line, 75 when
the studio is busy with another generation, after printing the Retry-After
wait, and 1 on any other error, after printing the server's message.
DramaEngine is a Swift library, version 2.1.2 (DramaEngineVersion.string,
CHANGELOG.md). Add this repository as a package dependency and the
DramaEngine product to a target; the app, the server and the CLI are not
needed. The public API includes ModelLocation and DramaSetup for the
install, DramaSynthesizer (an actor) behind the SpeechSynthesizer
protocol, SpeechRequest with ReferenceVoice, VoiceDesign, Take and
GenerationConfig, the streamed SpeechEvent s with StreamedAudio and
GenerationEstimate, PronunciationDictionary, DramaCodec for standalone
encode and decode, and MockSynthesizer for previews and tests. Everything
else in Sources/DramaEngine is package: the repository's targets use it,
other packages cannot.
import DramaEngine
let report = DramaSetup.check(location: .huggingFaceCache())
guard let location = report.location, report.isReady else { fatalError(report.fixes.joined(separator: "\n")) }
let engine = DramaSynthesizer(location: location, codecDirectory: DramaCodec.defaultDirectory)
let voice = try await engine.encodeReference(audioURL: clip, transcript: "What the clip says.", name: "Narrator")
var request = SpeechRequest(text: "Hello there.", references: [voice])
request.config.streaming = .defaultLowLatency
request.config.maxRetries = 0 // Allow partial playback before a chunk is validated.
var audio = StreamedAudio()
for try await event in engine.synthesize(request) {
play(audio.newSamples(from: event))
}
-
DramaSetup.checkreads files only and names what is missing: a weight shard, the tokenizer,codec.pth, the Core ML codec models, or a tuned kernel schedule for this Mac, each with a fix.DramaSetup.preparebuilds the codec models with theconvert_codec.pythat ships in the engine's resource bundle, through a Python you name (it throws the exact commands when you name none and never downloads Python), and loads the model once to tune the kernels. -
Examples/SpeakCLIis a separate package that depends on this one by path, clones a voice from a clip, streams progress, and writes a WAV in 67 lines.Examples/README.mdhas the Xcode steps and the entitlements: none beyond the app's own, plus the microphone only to record clips. -
The DocC catalog (
Sources/DramaEngine/DramaEngine.docc) has the landing page and the articles "Getting the model", "Voices" and "Streaming". Build it withxcodebuild docbuild -scheme DramaEngine -destination 'generic/platform=macOS' -derivedDataPath .build/docbuild; no DocC plugin is needed.swift test --filter SDKUsageTestsruns the landing page's example onMockSynthesizerand fails if the page's copy drifts. -
tools/api_audit.py(afterswift package dump-symbol-graph --minimum-access-level public) lists the public symbols and fails on one without a doc comment or a public value type that is notSendable. -
tools/make_xcframework.shbuildsdist/DramaEngine.xcframeworkfor macOS on Apple silicon, with the Metal kernel sources and the conversion script inside the framework, then builds and runsSpeakCLIagainst it. -
Sources/DramaEngine/Weights: the eight safetensors shards are memory-mapped (MAP_SHARED) and wrapped as Metal buffers without copying. Experts stay in FP8 E4M3 with their 128x128 block scales and are decoded inside the kernel through a 256-entry table; every other weight is bf16 widened to fp32 in the kernel. -
Sources/DramaEngine/Kernels: zero-centered RMSNorm, a bf16 GEMV for decode and a simdgroup-matrix GEMM for prefill, the Gated DeltaNet depthwise conv and recurrence, GQA attention with an online softmax over an fp32 KV cache, partial RoPE (NeoX for the backbone, interleaved for the tower), the MoE router and expert-major FP8 kernels that dispatch only over active experts (one-row kernels for decode, tiled simdgroup-matrix kernels for prefill), the GPU sampler, and the codebook tower's embedding mixing. Tile shapes, threadgroup sizes and the attention key split are fields of aKernelSchedulethatAutotunefills per machine (see below). -
Sources/DramaEngine/Model: the backbone and tower runners, the reference's sampler (top-k, top-p, exponential-noise argmax, repetition-aware retry of the main token) on the CPU and as Metal kernels that reproduce its picks and generator stream bit for bit, prefill and decode sessions, and the chunked frame generator. A decode frame is one command buffer, with the next frame's buffer committed before the current one is waited for. -
Sources/DramaEngine/Prompt: thereference-bracket-v1prompt builder, the reference's chunker and history policy, plus a sentence-per-turn mode. -
Sources/DramaEngine/Tokenizer: a byte-level BPE tokenizer that matches Hugging Facetokenizerson a 661-case corpus. -
Sources/DramaEngine/Codec: Core ML wrappers for the DAC encoder (reference clips to codes) and decoder (codes to 44.1 kHz audio), both exported at one static 512-frame shape and run in windows with 256 frames of left context, plus the same decoder at a 64-frame shape for the silence stop's probe. -
Sources/DramaServer: the HTTP API onNetwork.framework. Request parsing and routing are pure functions from anHTTPRequestto anAPIRequest;APIServiceanswers them against anySpeechSynthesizerandVoiceStore, so the app,drama serve, and the tests share it. -
Sources/DramaMCPandSources/drama-mcp: the MCP server, JSON-RPC 2.0 over stdio, proxying the HTTP API. -
App/DramaStudio: the SwiftUI app, with the API server and the menu bar extra inApp/DramaStudio/Server. -
eval/: the faithfulness suite (seedocs/EVAL.md).
Current UI checks and responsiveness measurements are in UI verification. Historical engine experiments below record their original configurations; they are not current latency guarantees.
Design notes and decisions are in docs/DESIGN.md; the decision trail of the
build is .audit/drama-studio.tsv.
The engine is checked against the fish-speech v3.0.0-beta PyTorch
implementation running on the CPU in float32, teacher-forced on the same
prompt and frames. Numbers from eval/compare.py:
| quantity | relative L2 vs float32 oracle | top-1 agreement | mean KL |
|---|---|---|---|
| backbone hidden state, 121 positions (worst position 1.3e-6) | 5.7e-7 | ||
| main-token candidate logits | 3.0e-7 | 1.000 | 1.3e-12 |
| nine tower head logits | 4.4e-7 | 1.000 | 4.2e-12 |
| residual stream after each of the 40 layers, and each of the 13 tower layers, at the first step | under 2e-6 each |
For scale, the same oracle run in bfloat16 on the CPU differs from its float32 run by a relative L2 of 5.2e-2 on the hidden state with a top-1 agreement of 0.846. The Metal engine sits at float32 precision; it keeps the FP8 expert weights exact and widens everything else, so it does not reproduce the CUDA release's W8A8 activation quantization, and no claim is made about that.
The sampler is checked separately: over 1174 sampled picks, none falls outside the reference's top-k / top-p set, and the count of rank-0 picks matches the expected value within noise.
End-to-end, the sentences the model does speak transcribe cleanly under
Whisper, but with the reference's chunking it often stops after the first
sentence of a two-sentence prompt or trails into silence, and the reference
PyTorch implementation does the same on the same text and seeds. The
sentence-per-turn mode and the retry rule below exist for that reason.
See eval/README.md for how to reproduce every number.
The codec uses Core ML to access Apple's inference backends. In the recorded experiments on macOS 26.5 and an M3 Max, the Neural Engine was slower than the CPU:
- The decoder's Neural Engine compile for the 512-frame shape ran for more than
35 minutes without finishing; the 64-frame shape compiled in about 7 minutes
and then ran three times slower on the ANE than on the CPU, because the
compute plan puts 72 percent of the decoder's cost (long-sequence
convolutions and the Snake activation's
sinafter upsampling) on the CPU anyway. - The ANE's native transposed convolution lost about 15 dB of accuracy, so the export rewrites transposed convolutions as polyphase convolutions.
- The GPU path returns wrong audio for the chained residual units of the last decoder block (about 5 dB SNR); each unit alone is correct, which points at op fusion, though the cause was not isolated.
- Core ML's enumerated shapes crash the CPU backend (SIGTRAP) on multi-channel convolutions here, so the models use one static shape.
So the shipped configuration runs the encoder in fp32 and the decoder in fp16
on the CPU through Core ML: 1.5 s per 512-frame window (23.8 s of audio) for
the decoder and 2.1 s for a 10 s reference clip in the encoder. The encoder
matches PyTorch's codes exactly; the decoder is 35 to 39 dB from the fp32
reference. DramaCodec.init(computeUnits:) still accepts .cpuAndNeuralEngine
for anyone who wants to retry on a different machine or OS.
Measured with eval/e2e.py on six texts, plain and cloned, three seeds each
(36 runs per row), scored by Whisper small.en word error rate, whether the model
reached eos, trailing silent frames, and resemblyzer speaker similarity to the
reference clip for the cloned runs.
| configuration | mean WER | runs at or under 0.1 WER | eos reached | mean quiet tail | speaker similarity |
|---|---|---|---|---|---|
| reference chunking (300 bytes), no probe | 0.141 | 69% | 89% | 103 frames | 0.867 |
| one sentence per turn | 0.063 | 86% | 100% | 30 frames | 0.867 |
| sentence per turn, silence stop, short-chunk retry (previous defaults) | 0.016 | 92% | 100% | 2 frames | 0.867 |
The app uses the shipped chunking and retry policies. The drama CLI exposes
--chunking for unchecked comparisons; checked generation always uses sentence
turns. --no-retry and --max-frames remain available.
Uncloned outputs score 0.54 to 0.56 against the reference clip, so the cloned
0.87 is the voice carrying over, not the metric's floor. The only misses in the
last row are Whisper rendering "the 21st of March" as "21 March" in three runs;
every other run transcribes exactly. The retry rule's threshold was chosen on
the same 72 runs of the first two rows and then applied to the third, so treat
it as tuned to this sentence set. The word error rates above use the PyTorch
codec to decode. Decoding the same 36 frame sequences with the shipped Core ML
fp16 decoder instead (eval/coreml_vs_pytorch_wer.py) gives the same mean
word error rate, 0.063 against 0.063, with no run scoring worse.
Medians of five interleaved runs against the previous engine on an idle
machine (eval/perf_ab.sh, raw runs under eval/out/perf/ab_final/). GPU
times come from drama bench; the end-to-end rows are wall time from
drama generate --warm on a 324-row prompt (a 10 s reference clip plus text)
and 600 frames.
| step | before | now |
|---|---|---|
| weights mapped and kernels compiled | 0.4 to 0.7 s | 0.4 to 0.7 s |
| warm-up (page walk, pipelines, first command buffer) | 4 s cold, then a 0.37 s first forward | 4 s cold; the first forward then costs 0.05 s |
| backbone prefill, 128 rows | 647 ms GPU | 213 ms GPU |
| tower prefill, 128 rows | 171 ms GPU | 38 ms GPU |
| backbone plus tower prefill, 300 rows in one chunk | three 128-row chunks | 382 plus 93 ms GPU |
| end-to-end prefill of a 324-row prompt | 2.44 s | 0.72 s |
| backbone decode, one frame | 17 to 18 ms GPU | 17 to 18 ms GPU |
| tower decode, one frame | 6.7 ms GPU | 6.7 ms GPU |
| decode, one frame | ten command buffers, sampling on the CPU between tower layers | one command buffer, 25.6 to 27.0 ms GPU, the next frame's buffer committed before the current one is waited for |
| end-to-end decode | 31.5 frames/s (range 28.7 to 31.6) | 38.5 frames/s (range 37.0 to 39.1), 1.8x real time |
Prefill runs dense projections through a simdgroup-matrix GEMM and the FP8
experts through tiled kernels that dequantize each weight tile once; at 128
rows nearly every expert is active, so a chunk reads all 30 GB of expert
weights once and the chunk is 512 rows so a 10 s reference clip is one pass.
Small chunks pay for that choice: an 8-row chunk takes 66 ms instead of 42,
because one tile shape serves every row count so that a row's result is the
same in any chunk (the prefix cache depends on it). Decode is bound by weight
bandwidth: the one-row backbone forward reads about 3.4 GB of weights in 17
to 18 ms, about 200 GB/s averaged over its 600 small dispatches, and the
no-experts bench delta puts the eight active experts at about 250 GB/s.
drama profile --rows N prints the GPU time of each kernel group on the real
weights (one layer repeated, so its numbers can be cache-warm), and
drama kbench times the GEMM tile shapes and the expert kernels on synthetic
data. The before column comes from the same session as the now column; the
same baseline binary measured 583 to 586 ms for the 128-row prefill in two
earlier series that day, so day-to-day drift of about 10 percent is normal.
The kernel changes keep every fp32 product and change only summation orders;
the faithfulness table above is from the new engine (hidden 5.7e-7, candidate
logits 3.0e-7, tower logits 4.4e-7 relative L2, top-1 1.0; it was 6.5e-7,
3.3e-7 and 5.0e-7 before). The GPU sampler reproduces the CPU sampler's picks
and generator stream: 160,000 chained picks identical in the long parity run
on the pre-merge code (DRAMA_SAMPLER_PARITY_SAMPLES=20000), 16,000 in the
default run on the merged code, and every generation compared between the
CPU and GPU samplers on the previous kernels produced the same frames.
Reordered sums do change the result on some prompts: in the end-to-end suite
28 of 36 runs produced the same frames as before, one differed only in where
the codec silence probe stopped it, and 7 differed deterministically, with
the summary metrics unchanged (mean word error rate 0.016 in both, eos
reached in all runs, speaker similarity 0.870 against 0.867). On the one such
prompt examined (eval/specs/qshort.json), the two engines agree to 2e-7
through layer 29 and part at layer 35, the shape of a discrete routing change
on a near-tie (the router ids themselves were not dumped). A fresh float32
oracle dump of that prompt takes the new engine's side: 9.5e-7 on the hidden
state, where the previous engine was off by 3.9e-3 and failed the suite's
thresholds. That holds for this one prompt only; the faithfulness gate above
runs on a single prompt and would not catch the same kind of divergence on
another.
A second round on the decode frame, run as a hillclimb: seven experiments in
small synthetic settings first, then one change at a time into the real
frame, each kept only when the output stayed bit-identical and the frame got
measurably shorter. The decision log is .audit/hillclimb.tsv.
What the experiments established before anything changed. A decode frame was 1312 dispatches streaming 5.94 GB of weights in 21.7 ms, and its large GEMVs already ran at 346 to 378 GB/s against a measured read ceiling of 383 GB/s. At that rate the bytes alone take 15.5 ms, so the 30 percent target of this round (14.6 ms) sits below what any kernel change can reach; fewer bytes per frame were the only route, and the only route that keeps the output exact is lossless compression. The bf16 mantissas carry 6.97 of 7 bits and are incompressible; the exponents carry about 2.8 bits of 8, so a 3-bit exponent code with a per-tile window and byte escapes packs a weight into 11.3 bits with exact round trip. The kernel that unpacks it is compute-bound at about 285 GB/s of physical reads, so it wins only on the K=2048 shapes with wide N.
What shipped, every step with a byte-identical dump and the faithfulness table unchanged (hidden 1.2e-6 relative L2 at most, top-1 1.0):
| step | change | frame (bench rows 1, backbone + tower), under load | quiet GPU, 2026-10-23 |
|---|---|---|---|
| baseline | 21.1 ms | 21.0 ms (15.0 + 6.0) | |
| MoE layer | one decode row's mixture of experts as three dispatches (bf16 router and shared pass, routing plus routed gate-up, routed down plus shared down plus combine), FP8 decoded with integer ops instead of the lookup table | 20.2 | 20.1 |
| encoder | a concurrent compute encoder with a barrier before every dispatch except the later members of a sibling group (the four GDN input projections, q k v, w1 and w3, and so on); q/k norm, rope and the KV writes in one dispatch | 19.7 | 19.5 |
| samplers | 256-thread sampling kernels in place of one 1024-thread threadgroup; no change on a quiet GPU, and 47 to 22 ms per frame when another GPU client is active, which is what a user with an animating window sees; on a quiet GPU the kernels cost 0.67 ms per frame, about 0.2 ms more than the old shape, and the old shape stalls under the window server alone | 19.7 | not re-benched |
| tower | embedding plus both concat norms in one dispatch, attention split over 8 threadgroups with an in-order combine and the gate folded in, the residual adds folded into wo and w2, w1 and w3 as one dispatch with the SiLU | 19.6 | not re-benched |
| packed weights | eight weight families (GDN qkv and z, attention q, the semantic lm-head slice, tower wqkv, heads, w1 and w3) read from an exponent-coded copy at one row, 0.94 GB fewer bytes per frame | 18.2 | 18.1 (12.9 + 5.2) |
Reverted: a multi-matrix GEMV that grouped sibling projections into one dispatch (within noise once the concurrent encoder had taken the same gaps), more threadgroups for the MoE down kernel (each doubling cost 3 to 9 us per layer in its prologue), and the load folds of norms, gates and SiLU into the GEMVs (no gain on the tuned schedule). The frame went from 1312 dispatches to 759.
The packed copy lives under ~/Library/Application Support/DramaStudio/Packed/,
2.1 GB per checkpoint, built on the first load in about two seconds across
all cores and verified by a full unpack before it is used; a cache that fails
to build or load falls back to bf16 with the reason logged, and
DRAMA_PACKED=0 turns it off without a rebuild. Prefill, chunk invariance
and the prefix cache never touch it: every path with more than one row reads
the bf16 shards as before.
Caveats, and what a quiet session settled (2026-10-23). Every number in the
hillclimb was taken while another process held the GPU (the window server
and a virtual machine), so the ranges carried stalls and no end-to-end figure
was quoted. A second session rechecked the round with the virtual machine
stopped and the app quit, one job at a time. The ioreg utilization counter
reads 100 percent under the window server alone and is no gate; the gate was
the sampler stages of a traced 120-frame generation (eval/quiet_label.sh),
0.7 ms per frame when the GPU is free, taken before and inside every batch.
The tune was redone there: the decode GEMV moved to 8 simdgroups, the narrow
GEMV kept its tile this time, the MoE prefill tile went to 32x32x32, and the
128-row forward went from 233 to 189 ms. The quiet column above is a
10-round interleaved bench per pair except the baseline, 5 rounds. The
concurrent encoder stays: against a serial-encoder build of the same code it
is 0.5 ms shorter on the bench line and 0.0 to 1.2 ms shorter per frame over
three interleaved 300-frame traces, with a fatter tail (p99 39 to 41 ms
against 33 to 35), which playback's 64-frame buffer absorbs. The parked
multi-matrix GEMV was rebased (branch hill/step3-gemv-decode-quiet) and
measured again: 0.1 ms, inside the A/A gap of the bench, so it stays parked.
End to end, eval/perf_ab.sh over five 425-frame generations: 35.1 frames
per second [31.7..36.4] against 20.7 for the baseline, whose 1024-thread
samplers stall under the window server alone. Short runs after idle reach
48 to 50 frames per second (19.2 to 19.7 ms per frame); the sustained figure
is lower because this Mac slows bandwidth-bound work by 20 to 40 percent
after minutes of GPU load and recovers over minutes of idle, so absolute
decode numbers depend on the GPU's recent history while interleaved A/B
deltas hold. The remaining ideas are small: the MoE down kernel's prologue,
and the w2 and 4096x2048 families in the packed path.
Default sentence chunking keeps short complete sentences separate. It no longer folds a short opening sentence into the next one, which could let the model end after the opening and omit the rest of that chunk.
GenerationConfig.streaming requests when decoded audio leaves the engine.
When maxRetries and minSpokenRatio are positive, every request uses
.sentence mode. Each chunk is decoded and checked before its selected attempt
leaves the engine. Checked generation also verifies both complete neighboring
chunks together before exposing their join. A mismatch retries only the current
unplayed chunk within the existing attempt limit. Already emitted audio stays
fixed. This applies to every voice, plain text, and performance cues.
With explicitly unchecked preview and retries disabled,
.lowLatency(stepFrames: 64) sends partial audio every
64 generated frames, about 3 seconds. The CPU decodes those frames while the GPU
generates the next ones. Such a chunk cannot be replaced after playback begins.
Normal generation waits for the model's end-of-speech signal, subject to its frame cap and loop guard. It does not treat silence as completion, because a voice can before the remaining words. Explicit silence probes run at fixed frame checkpoints and finish before sampling resumes, so decode timing cannot change the stop frame. Normal generation checks the decoded words with on-device speech recognition. Missing, changed, or uncertain words trigger a retry before audio enters playback or generation history. Cue retries alternate the requested sampling with broader replacements and ask for precise enunciation. Exhausted retries report a failure. Reference pace affects the time allowance, not the words that must be spoken.
Explicit recognition labels such as *gasp* or (gasps) count as nonverbal
annotations only when the script requests that cue family. Ordinary spoken
words remain required. The native checker selects an installed regional locale
from your macOS language preferences, with a deterministic fallback.
Acceptance policy revision 5 records the verified-sentence boundary rule. Archived takes retain their original check revision; playback does not upgrade their verification history. Opt-in local audits record the exact PCM identities, unit indices, token and confidence-span positions used by the boundary check.
The decode reuses the windows a full decode of the chunk would run
(DecodeWindow.covering), so a chunk's partials join into exactly its audio.
On the Core ML codec, 600 frames decoded in 64-frame steps differ from one
full decode by a maximum absolute error of 0
(DRAMA_CODEC_SEAM=1 swift test --filter testSeamOnTheRealCodec); decoding
each step with a bare 256-frame left context instead differs by 0.027. All
full-window codec work of a run goes through one serial queue: a low-latency
chunk's last frames decode while the next chunk prefills and generates. The
optional silence stop reads fixed frame checkpoints through the 64-frame probe
shape. Each reading completes before generation samples another frame. Readings
start after the voice-aware expected duration plus the configured silent tail.
drama generate --probe-every N and --probe-after-expected no explicitly enable
the probe for measurement. In sentence mode, each attempt's full audio is decoded
once at its end.
The measurements below describe the earlier asynchronous streaming implementation.
They predate universal retry buffering and the removal of silence stopping from
normal generation. eval/reliability/REPORT.md records the earlier probe timing
experiments.
Measured on the M3 Max (16 cores, 12 performance), with the same two-sentence
line, 9.4 s of audio in 202 frames, seed 7, in every run. The machine was not
idle: other users' jobs kept the load average at 5 to 7 (two Python processes
at a full core each), so every comparison interleaves its sides, and the raw
rows are in .audit/streaming.tsv.
Time to the first audio byte of stream: true through curl -N
(tools/streaming_ttfb.py, 5 runs per side, alternating). Before is the
engine at 9e645ef, which decoded every chunk after the whole request was
generated. Both sides returned the same 827,392 bytes, with HTTP 200 and curl
exit 0 every time. Server CPU comes from ps -o time= before and after each
request:
| before | after (low latency) | |
|---|---|---|
| first audio byte | 12.95 s (12.61 to 13.20) | 4.49 s (4.48 to 4.60) |
| last byte | 14.48 s (14.14 to 14.72) | 8.57 s (8.49 to 8.68) |
| server CPU per request | 39.0 s | 22.6 s |
| average cores in use | 2.69 | 2.65 |
| peak cores over any 0.5 s | 4.3 (3.8 to 4.7) | 4.0 (3.8 to 4.5) |
The first byte now comes after the prefill (0.7 s), 64 frames (1.8 s), and one decode of a 512-frame window (about 2 s). The request also uses less CPU: before, the silence probe decoded the whole chunk so far every 16 frames whenever no probe was running, and every chunk was decoded once more at the end; now each window is decoded once.
Generation speed with the decode on and off, in one process
(DRAMA_STREAMING_BENCH=1 swift test -c release -Xswiftc -enable-testing --filter StreamingBenchTests, 5 runs per mode, modes interleaved, medians
with ranges). "Decode after" turns the silence stop off, so the codec is idle
while frames land and runs once per chunk at its end:
| mode | frames/s while frames land | end-to-end frames/s | first audio | wall | CPU cores |
|---|---|---|---|---|---|
| decode after | 34.9 (34.7 to 37.5) | 19.1 | 5.07 s | 10.58 s | 1.20 |
| sentence | 33.0 (32.2 to 33.7) | 15.2 | 6.51 s | 13.30 s | 2.03 |
| low latency, 64 frames | 32.7 (32.4 to 33.6) | 18.9 | 6.77 s | 10.68 s | 2.11 |
| low latency, 32 frames | 32.9 (32.8 to 33.5) | 19.2 | 6.03 s | 10.50 s | 2.17 |
- The CPU decode slows generation by about 6 percent while it overlaps the GPU (34.9 to 32.7 frames/s). Low latency still finishes with "decode after" and 2.6 s ahead of sentence mode, because the GPU never waits for the codec.
- What limits the first audio is the decode, not the GPU. One window takes
1.6 s alone ("decode after" spends 3.35 s on two) and 4.6 s while the GPU
generates: in the event trace of the first low-latency run, the window was
queued at 2.50 s and its audio arrived at 7.09 s. The cause of the slowdown
was not isolated; contention for memory bandwidth or for the chip's power
budget is a guess. The test runner also sees slower decodes than
drama serve(first audio 6.77 s here against 4.49 s through the server), and that cause was not isolated either. - For this short first sentence (4.7 s of audio), low latency is no faster than sentence mode to the first audio. It pays off on a long first sentence: on one 12.5 s sentence (3 runs), low latency started at 6.19 s (6.09 to 6.94) against 23.5 s for sentence mode. Part of that gap is the retry: sentence mode judged the first attempt too short and generated the sentence again, while low latency kept that attempt (336 to 369 frames).
- 32-frame steps start 0.7 s sooner but stutter: in every trace the first 1.5 s of audio ran out 1.1 to 1.5 s before the next partial arrived. With 64-frame steps every trace played without a gap, so 64 stays the default.
The app shows the time left beside the elapsed time while it generates
(GenerationEstimate): the expected frames of the unfinished chunks (text
length at 16 characters per second, or the reference voice's slower pace,
times 21.53 frames per second) less the
frames of the current chunk, over a 3 s moving average of frames per second
that starts at 35, plus one prefill per chunk not yet prefilled (0.7 s until
the first is measured). It reads "about 12 s left", rounded to 5 s above 20 s,
and "finishing…" or "retrying" once the frames stop predicting the end. The
same numbers are X-Drama-Estimated-Seconds and the estimate field of
/v1/health.
Kernel schedule parameters (the GEMV token tile and rows per threadgroup, the
GEMM tile, the MoE tile and K step, the one-row expert kernels' rows per
threadgroup, the attention key split, the GDN threads per state column) are
not constants any more. On the first load for a given (chip, macOS version,
kernel source digest, model snapshot) the engine times each kernel family's
real workload on the resident weights, rotated over ten layers, for every
value in its declared domain, keeps a candidate only when its output is within
a relative L2 of 1e-5 of the default's on the same inputs and it is at least 5
percent faster over five interleaved rounds, then times whole forwards of 128
and 300 rows and of one row on both schedules and drops a side's choices
unless the tuned one wins by 2 percent at every row count. The result is a
JSON report under ~/Library/Application Support/DramaStudio/Tuning/; every
later load reads it and measures nothing. drama tune runs it from the
command line and prints the table (about 30 s here); the app shows "Tuning
kernels for this Mac" once; drama schedule-ab times any two schedule files
against each other inside one process. The idea is Magnitude's (tune,
validate, cache per device); the kernels, the domains and the measurement scripts are
this project's. Nothing in the numerics changes beyond summation order: the
tuned engine scores hidden 5.8e-7, candidate logits 3.0e-7 and tower logits
4.4e-7 against the float32 oracle with top-1 1.0 (eval/out/perf/compare_tuned_final.txt),
and the A/B prompt below produced byte-identical frames with and without the
tuned schedule in all fourteen compared runs.
What it shipped on this M3 Max (eval/out/perf/tune_final2.txt): two decode
changes, a token tile of 1 with two simdgroups per threadgroup for the one-row
GEMVs (27 percent faster per kernel) and a 16-way key split for one-row
attention (27 percent). Everything on the prefill side stayed at the
hand-picked values: the GEMM tile, the MoE tile and the GDN split never
cleared the 5 percent bar, and the narrow-projection GEMV tile and the 4-way
prefill attention split, 80 and 5 percent faster per kernel, moved the whole
128- and 300-row forwards by 1.2 and 0.4 percent, under the margin, so the
whole-forward check reverted them. Two earlier versions of the measurement script would
have shipped a slower prefill: random activations route evenly over the 256
experts and favour a small MoE row tile that was 9 percent faster in the
measurement script and 5 to 12 percent slower in the real prefill, and a 2 percent
minimum gain measured at 128 and 32 rows let through a tile that lost 9
percent at 300 rows. Both are in .audit/autotune.tsv, and the isolation
that caught the second is eval/out/perf/schedule_ab_1.txt.
Measured, with the shipped schedule against the default:
| measurement | untuned | tuned |
|---|---|---|
backbone forward, one row, GPU, drama schedule-ab in one process, 7 interleaved rounds |
19.2 ms [18.4..19.5] | 15.3 ms [14.8..15.5], faster in 7 of 7 |
same, drama bench in separate processes, two five-run series |
18.0 [17.2..18.3], 17.9 [17.5..18.2] | 15.1 [14.7..15.8], 15.4 [14.8..16.7] |
tower decode, one row, GPU, drama bench |
6.9, 6.8 ms | 6.1, 6.2 ms |
end-to-end decode, 425 frames after a 324-row prompt, drama generate , five-run medians |
32.9 [15.0..37.1], 29.7 [21.8..33.1] frames/s | 33.5 [14.9..42.2], 35.1 [32.3..36.8] frames/s |
| same, GPU ms per frame, the three rounds of the first series where neither side was disturbed, then all five of the second | 26.9, 27.9, 30.4; 46.0, 34.6, 32.9, 33.7, 30.2 | 24.6, 23.7, 29.8; 31.0, 28.5, 27.2, 27.7, 29.7 |
| backbone prefill, 128 and 300 rows | unchanged: the shipped schedule runs the same kernels ( schedule-ab 196.9 against 201.0 and 382.9 against 382.0 ms, the instrument's own spread) |
The one-row forward is the solid number: about 20 percent less GPU time in
four series (the two bench series, the in-process A/B and the tuner's own
check at 17.1 against 14.6 ms), ranges not overlapping, from the decode GEMV
tile and the attention split, both of which put more and smaller threadgroups
on the same bytes. The end-to-end frames-per-second rows are softer: the two
drama generate series ran on a machine that was not idle (other sessions'
test jobs and the window server busy; load average 4.8 to 12), their ranges
are wide, and in the first series two of five rounds were disturbed on both
sides for a reason the load averages do not show, so the five-run medians
move only 2 and 18 percent while the per-frame GPU time is lower on the tuned
side in every round where neither side was disturbed. Those two series ran
with an earlier cache whose decode fields equal the shipped ones and whose
prefill fields have since been reverted. Prefill has no measured change and
none is expected: with the reverted fields the schedule is the default there.
To redo the end-to-end table on an idle machine: eval/tune_ab.sh .build/release/drama 5 eval/out/perf/ab_tune_idle then
eval/.venv/bin/python -I eval/perf_ab_summary.py eval/out/perf/ab_tune_idle.
Copyright 2026 Peter Marelas. Drama Studio's own code is under the Apache License 2.0; see LICENSE and NOTICE.
Built with Fish Audio. Drama Studio runs Fish Audio Drama 3, and the model weights and codec checkpoint are not included: the app downloads them from fishaudio/drama-3, where Fish Audio licenses them under the Fish Audio Community License, Version 1.0, copied here as LICENSE-FISH-AUDIO. That license governs the model and the audio you make with it. It allows research and non-commercial use, and commercial use while annual revenue is under USD 500,000; it forbids cloning a real person's voice without their consent and the other uses its Section IV(b) lists, and it needs a separate Fish Audio license to offer speech generation to others as a hosted service.
Section IV(b) of that license asks anyone who makes speech generation available
to others to bind them to its use restrictions, so the app's setup window shows
them and asks you to agree before it downloads or generates anything. Until you
agree, the window, the HTTP API (403) and through it the MCP server and
drama-say refuse to speak. The command-line drama tool is for developers
building from source and does not ask.
The fish-speech reference code copied under eval/oracle/ and
Sources/DramaEngine/CodecConversion/dac_reference/, the parts of DramaEngine
reimplemented from it, the converted Core ML codec, and the example cast from
Fish Audio's Drama 3 Space stay under the Fish Audio Community License as well;
THIRD_PARTY_NOTICES lists each one and what changed.
descript-audio-codec code keeps its MIT notice there. This project is not
affiliated with or endorsed by Fish Audio.
During setup, you can choose to share anonymous usage information to help improve Drama Studio. Sharing is off unless you enable it. Only counts of app launches, generation outcomes and setup failures are included. Scripts, audio, voices, file names and personal identifiers are excluded. Change your choice at any time in Settings > General > Privacy.
See usage monitoring configuration for release setup and the data contract.