{"slug": "drama-studio-local-realistic-text-to-speech-synthesis-engine-for-apple-silicon", "title": "Drama Studio: local realistic text-to-speech synthesis engine for Apple Silicon", "summary": "Developer Peter Marelas released Drama Studio, a native macOS app that runs Fish Audio's 35B Drama 3 text-to-speech model locally on Apple silicon using hand-written Metal compute kernels for the mixture-of-experts backbone and codebook tower, with the DAC audio codec running through Core ML and no MLX or PyTorch at runtime. The app requires about 49 GB of free disk, maps roughly 41 GB of model weights (a 41.5 GB download pinned to revision e0a9218 plus a 1.5 GB Core ML voice codec), and recommends 64 GB or more of unified memory; generation was tested on an M3 Max with 128 GB, while an M1 Mac Mini with 16 GB failed twice with a GPU out-of-memory error. The model weights fall under the Fish Audio Community License, which the app requires users to accept before generating anything and which prohibits cloning a real person's voice without consent.", "body_md": "By Peter Marelas. A native macOS app that runs Fish Audio's [Drama 3](https://huggingface.co/fishaudio/drama-3)\ntext-to-speech model locally on Apple silicon. The 35B mixture-of-experts\nbackbone and the codebook tower run on hand-written Metal compute kernels that\nread the released safetensors weights in place. The DAC audio codec runs\nthrough Core ML. No MLX, no PyTorch at runtime.\n\nBuilt with Fish Audio. The model weights are under the\n[Fish Audio Community License](https://huggingface.co/fishaudio/drama-3/blob/main/LICENSE);\ncloning a real person's voice without their consent is not permitted.\n\n## Drama-Studio-Promo-10MB.mp4\n\n- Apple silicon Mac. Checked generation requires macOS 26 or later and an installed on-device speech model for the script language. Unchecked preview targets macOS 15. Intel Macs are not supported by the engine.\n- About 49 GB of free disk for the model, codec and setup caches. Allow more space for generated audio, saved projects and app updates.\n- 64 GB or more of unified memory is recommended. This is not a verified\nminimum. Generation has been tested on an M3 Max with 128 GB. An M1 Mac Mini\nwith 16 GB failed twice with a GPU out-of-memory error before producing audio,\ndespite paging. See the [hardware test](https://github.com/maralski/drama-studio/blob/main/docs/ui-verification.md#16-gib-mac-mini-hardware-test) .\n32 GB and 64 GB Macs remain untested. The engine maps about 41 GB of model weights.\n- A display that can fit a 900 by 640 point studio window.\n- An internet connection for the initial downloads and update checks. Speech generation runs locally after setup.\n\nThe setup memory override tests the warning only. It does not simulate a lower-memory Mac or establish that generation works on one.\n\n1. \nDownload `DramaStudio-<version>.dmg` from the[latest release](https://github.com/maralski/drama-studio/releases/latest) .\n2. \nOpen the disk image and drag Drama Studio to Applications.\n3. \nOpen Drama Studio from Applications. The release is not notarized, so macOS blocks the first open. Open System Settings, go to Privacy & Security, scroll to Security, and click **Open Anyway** next to the message about Drama Studio. Confirm with your\npassword. If you use Terminal, you can run`xattr -dr com.apple.quarantine /Applications/DramaStudio.app` instead.\n4. \nRead the license terms in the setup window and click **I Agree** . Fish\nAudio licenses the model under the Fish Audio Community License, and\nDrama Studio generates nothing until you agree.\n5. \nWait for the downloads. The window shows the progress of each file, the total, and the time left: \n  - the Drama 3 weights from Hugging Face, 41.5 GB, pinned to revision\n`e0a9218` ;\n  - the Core ML voice codec from the release, 1.5 GB.\n You can pause, or quit and open the app again. The download continues where it stopped. Each file is checked against a SHA-256 value that ships with the app before it is kept.\n6. the Drama 3 weights from Hugging Face, 41.5 GB, pinned to revision\n7. \nWait 30 seconds to a minute while the first load tunes the GPU kernels for your Mac.\n8. \nClick **Generate** to speak your first line.\n\nIf you already downloaded the model with `hf download fishaudio/drama-3`,\nDrama Studio uses that copy and skips the 41.5 GB download. If you are below\n64 GB of memory, the setup window says so before anything downloads. You can\ncontinue at your own risk.\n\nUse **Clear Script** in the toolbar or Script menu to empty the editor.\nFor an imported document, **Clear Section** empties only the selected section.\nExisting takes stay available and may need regeneration after the text changes.\nUse Edit > Undo to restore cleared text. Clearing is disabled during generation.\n\nScreenplay Mode requires a speaker name, a colon and a space before dialogue:\n\n```\nAlex: Hello there.\nSam: Good morning.\n```\n\nA line without a name continues the previous speaker. Text before the first named speaker belongs to Narrator. Up to 16 speakers are supported. If the script has no named dialogue, turn Screenplay Mode off for plain narration, or use the help button beside the error for the format and examples.\n\nDrama Studio checks GitHub for a newer stable release at launch and downloads\nit in the background. Automatic checks and downloads are on by default. Turn\nthem off in **Settings > Updates**, or use **Drama Studio > Check for Updates**\nto check manually at any time.\n\nOnce the download is verified, **Settings > Updates** offers **Install and\nRestart**. Save your project and finish generation or API requests before\nconfirming. **Later** leaves the running app unchanged; click **Review Update**\nin Settings > Updates when ready. Check for Updates opens the same tab.\nThere is no separate Software Update window. Quitting normally does not install\nan update. A deferred update is downloaded again on the next automatic or manual check.\n\nUpdates require the app to be installed in a folder you can write to, such as\nApplications or your user's Applications folder. Apps running from a mounted\nDMG, App Translocation, or a Swift build directory cannot update themselves.\nThe updater changes only the app bundle; models, voices, and projects stay in\nplace. It uses the release's `DramaStudio-<version>.dmg` and `SHA256SUMS`, checks\nthe bundle identity, version, and minimum macOS requirement, verifies its code\nsignature, and requires the\nsame signing identity for Developer ID builds. Ad-hoc builds rely on GitHub\nHTTPS and the published checksum for authenticity.\n\nAfter restart, setup compares the installed codec's recorded archive SHA-256 with the digest shipped in the updated app. Matching codecs are reused even when the release URL changes. A changed digest or missing codec models triggers a verified download and installation. Older installs without a recorded digest need one verified download to establish it.\n\n- **Script editor** with import of`.txt` ,`.md` ,`.rtf` ,`.rtfd` ,`.html` ,`.docx` ,`.srt` and`.vtt` , drag and drop, and a screenplay mode that turns`Name: line` dialogue into Drama 3 speaker tags (up to 16 speakers).\n- **Takes** : Generate or Generate All produces a fresh take each time. A\nscrolling grid shows take durations, notes, export choices and outdated audio.\nOther controls include an Expressiveness slider under Advanced (one control\nover temperature, top-p and top-k, with the vendor's defaults in the middle\nand the Hugging Face Space's sampling at the top), and an\ninsert menu for performance cues such as`[laughs]` ,`[sighs]` and`<whisper>…</whisper>` . Each chunk's frame cap follows the length of its text.\nNormal generation checks each decoded sentence with local, audio-only speech\nrecognition before playback or reuse as history. It also checks each complete\nadjacent pair before releasing the next sentence, so a bad join retries only\nthe unplayed sentence within the same attempt limit. Each raw sentence must\npass both recognizers with the existing native confidence checks. For a pair\nof those exact verified sentences, the native checker may mark only the first\nnonterminal word of the second sentence uncertain while both full transcripts\nstill match. That narrow boundary case uses the raw sentence's confidence;\nambiguous word positions, missing confidence, or other uncertain words fail.\nArbitrary-file checks have no such generated-sentence evidence. These bounded windows\ncover every generated join within one request; they do not establish a global\ntranscript for a long recording or certify joins between separate requests. Missing or uncertain words\ntrigger a bounded retry; exhaustion fails instead of creating a completed take.\nChecked generation requires macOS 26, an installed Apple on-device speech\nmodel for the script's language, and the bundled independent recognizer.\nBoth audio-only recognizers must find every scripted word before a sentence\nis accepted. A disagreement retries generation; an unavailable checker stops\ngeneration. These checks reduce word errors but cannot guarantee recognition\naccuracy or acting quality.\nOlder takes without word-check history are kept for playback\nand export, but do not automatically become voice references or retake history.\nVoice auditions always use checked generation.\nRecognition checks word coverage, not acting quality or perfect cue execution.\n- **Document narration** : a PowerPoint deck, a PDF or an EPUB opens as\nsections, one per slide, page or chapter, and narrates section by section\nwith the same voice throughout. Exports are the whole narration, one file\nper section with a`manifest.csv` , SRT or WebVTT captions, and for a deck a\nslide narration package. Projects save and reopen as`.dramaproj` bundles\n(see[Documents](#documents) ).\n- **Pronunciation rules** in Settings rewrite words before the model hears\nthem, in the app and through the API.\n- **Voice cloning** from one or more reference clips (any format AVFoundation\nreads) plus their transcripts, kept in a local voice library.\nTo reuse a generated voice, select a take and click**Save Voice to Library** beside**Use for Export** . Name the voice and save it. The app selects it for\nfuture generations, including after you edit the script. This works with\nModel default takes and individual document sections. Complete, checked takes\ncontaining one speaker can be saved; outdated script text does not prevent it.**Save Session Voice** in Cast saves the retained session voice instead.\n- **Streaming playback** as chunks finish, scrubbing, a speed control from\n0.75x to 1.5x, the sentence being spoken highlighted in the editor, and\nexport to WAV or M4A.\n- **A local HTTP API and an MCP server** , so scripts and agents can speak\nthrough the same engine and voice library (see[Agents and programs](#agents-and-programs) ).\n- **Menu bar residency** : closing the window leaves Drama Studio in the menu\nbar (the waveform-and-microphone icon) with the API still serving. The menu\nshows the model state, the API address, and the current job. ⌘Q quits.\n\nThe [system requirements](#system-requirements) also apply to source builds.\nInstall the following tools and model files:\n\n- macOS 15 or newer, Xcode 26 command line tools.\n- The model in the Hugging Face cache:\n\n```\nuvx --from huggingface_hub hf download fishaudio/drama-3\n```\n\n- The Core ML codec models, built once from `codec.pth` :\n\n```\ncd tools && uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python \"torch==2.7.1\" coremltools \"numpy<2.4\" einops soundfile\n.venv/bin/python -I ../Sources/DramaEngine/CodecConversion/convert_codec.py\n```\n\nThey land in `~/Library/Application Support/DramaStudio/Codec/` (encoder\n809 MB in fp32, decoder 353 MB in fp16).\n\n```\npython3 tools/bootstrap_word_recognition.py  # downloads and checks pinned framework and local speech models\nswift build -c release\npython3 tools/stage_word_recognition.py --bin .build/release\ntools/make_app.sh            # assembles dist/DramaStudio.app, with drama-mcp and drama-say in Contents/MacOS\nopen dist/DramaStudio.app\n```\n\nCommand line:\n\nThe app build stages the recognition runtime automatically. CLI builds and tests\nneed the bootstrap and staging steps above. The English and multilingual models\nuse about 975 MB on disk; recognition runs locally with no script hints. English\nuses `small.en`; other supported languages use `small`. Only the selected model\nis loaded. Recognition requires both an installed Apple model and a supported\nindependent model language. Older native-only checks remain recorded in project\nfiles, but their audio is excluded from automatic checked-reference reuse.\n\nRun the retained wrong-word and complete-ending controls with\n`swift test -c release --filter LocalWordVerifierTests`. On macOS 26 these controls\nfail if required local assets are missing; they do not skip that failure.\n\n`tools/make_cli.sh` assembles a signed, relocatable CLI runtime in\n`dist/drama-cli`, including the helper, framework, resource bundle, models and\nlicenses. Set `DS_VERIFY_NATIVE_WORD_CHECKS=1` when running either packaging\nscript to require the installed native model and exercise the full dual check\non the retained complete-ending and wrong-word recordings. Packaging always\ntests the independent runtime. The full offline check can also be rerun with\n`drama check-words <WAV> <script> <accepted|rejected>`, or with the app executable's\n`--check-words` flag. These commands do not load the speech generation model.\n\n```\n.build/release/drama tune                      # once per Mac, about 30 s; the app does it on first load\n.build/release/drama generate --text \"Hello there.\" --out frames.npy\n.build/release/drama bench                     # add --untuned to compare against the hand-written schedule\n.build/release/drama serve                     # the HTTP API without the app; --port 8780, --no-token\n.build/release/drama-say \"Hello there.\"         # speaks through the running app, like say; see From the terminal\n```\n\n`DRAMA_PREVIEW_ENGINE=1` runs the app or `drama serve` on a mock engine that\nproduces placeholder tones, for working on everything but the model.\n\nThe app looks for the weights in this order: the folder chosen in Settings,\nits own download in `~/Library/Application Support/DramaStudio/Models/drama-3-e0a9218/`,\nthen the newest Hugging Face cache snapshot. A build from `tools/make_app.sh`\nhas no codec download, so its setup window shows the Python conversion command\nabove when the codec models are missing; `tools/make_release.sh` builds the\nrelease DMG and the codec zip and writes the zip's URL and SHA-256 into the\napp. `DRAMA_STUDIO_HOME=<folder>` moves the downloads, the codec, the tuning\nand the settings into one folder and ignores the Hugging Face cache, to try a\nfirst run without touching your own. `DRAMA_STUDIO_MEMORY_GB=16` shows the\nlow-memory warning on a Mac that has enough. `tools/pin_model_plan.py --check`\ncompares the pinned download plan with Hugging Face.\n\nImporting a `.pptx`, `.pdf` or `.epub` (File > Import Script or Document,\ndrag and drop, or opening the file with the app) opens it as sections: a\ndeck's slides, with the speaker notes as the text and the slide text when a\nslide has no notes; a PDF's pages; an EPUB's chapters. The list shows each\nsection's label, a preview, its duration once narrated, a skip box and a\nspeaker picker. The editor edits the selected section.\n\n- Generate narrates every section that is not skipped, in order, one request\nper section, cast with that section's speaker's voice. The speaker's\nprevious section goes along as history, and with no library voice cast\nthe speaker's first narrated section is also the voice, the way takes and\nthe live route already carry a voice. Narrate This Section\ngenerates one section again. Click a take to audition it, then click\n**Use for Export** to choose it for the narration. The Export badge identifies\nthe chosen take; auditioning another take does not change it.\n- Editing text or changing a speaker or voice marks the affected audio as\n**Needs regeneration** .**Generate Changed Sections** updates only missing\nor outdated sections. Export asks before writing outdated or incomplete audio.\nEarlier takes remain available, including after a project is reopened.\n- Export writes the whole narration, Per Section (numbered files and a\n`manifest.csv` ), Captions as SRT or WebVTT, timed from the frames each\nsentence took, and for a deck a Slide Narration Package:`slide-NN.wav` per\nnarrated slide and a`README.txt` on adding them with Insert > Audio. The\napp does not change the deck.\n- **Save Project** writes a`.dramaproj` bundle with the document or script,\nsettings, voice assignments, session voices, and every take with its original\ntext, voice assignments, codes, and audio. Library assignments still refer to\nvoices in the local library. Save updates the current project;**Save Project\nAs** creates a copy. Format 2 projects require this version of Drama Studio.\nOlder projects still open, with a verification warning for audio whose\ngeneration history was not stored.\n- Import, Open Project, and Quit offer Save, Discard Changes, or Cancel when work is unsaved. A failed or cancelled save leaves the current session open. After editing pauses for two seconds, the app writes a recovery draft in the background; drafts wait until generation stops. The next launch offers to restore a draft left by an interrupted session. Recovery does not replace the last manually saved project.\n- **Flatten to Script** turns the sections into one plain script and removes\ntheir takes from the session. Its confirmation offers to save a project copy\nfirst.\n\nOn the five-slide sample deck (`python-pptx`, slide 3 without notes), the\nreal engine at the default settings narrated 24.3 s of audio in 34.5 s\nincluding the model load. Whisper small.en heard slide 4, whose notes read\n\"Revenue grew twelve percent. Costs stayed flat, and margins improved.\", as\n\"Revenue grew 12%. Cost stayed flat, and margins improved.\" Two slides lost\nwords: slide 1 spoke only \"Welcome, everyone.\" of its two sentences, and\nslide 3 only its title. Each was one engine chunk in that build (the splitter joined a\nsentence under 24 bytes to the next), and with \"Stream audio while\ngenerating\" on, a chunk that comes out short is kept rather than generated\nagain. With the setting off, Narrate This Section gave slide 1 both sentences\nand slide 3 \"How it works Import\". That is one run per slide, not a\nmeasurement. Because narration is exported, document sections now always\ngenerate in the retrying sentence mode whatever the streaming setting says;\nplayback of a section starts when the section is done rather than three\nseconds in.\n\n`eval/reliability/REPORT.md` is a 330-run study of whether every sentence\ncomes out as speech: 33 conditions (short and long lines, numbers,\nabbreviations, all caps, performance cues, multi-speaker tags, Spanish,\nFrench and German, a cloned voice, both expressiveness extremes, and two\ncontrol conditions), five seeds each, in `full` and `balanced`.\n\n| rows | full | balanced | \n|---|---|---|\n| plain prose under the shipped defaults | 130 of 130 | 126 of 130 | \n| performance-cue lines | 16 of 20 | 14 of 20 | \n| retries off (control) | 1 of 5 | 3 of 5 | \n| reference 300-byte chunking (control) | 7 of 10 | 9 of 10 | \n\nPaired by seed, balanced is never right where full is wrong, and wrong where\nfull is right in 4 of 130 prose pairs. Performance cues are the one area\nwhere both modes drop content a few times in twenty; `[laughs]` can also turn\ninto a laughter loop that the app's text-derived frame cap ends near 8 s.\nReproduce with `eval/reliability.py` (slices under `eval/reliability/slices/`).\nThe `balanced` mode was removed after this study, so the engine has one\nprecision, `full`; the column stays as the record of why.\n\nThe cue dropouts were then investigated on eight cue lines, five seeds each,\neleven variants (`eval/cues/`, section \"Performance cues: investigation\" of\nthe report). The model stops at a cue that follows a sentence end, whatever\nthe chunking; longer chunks lose more. Three guards now ship: a laughter loop\nis detected in the code stream (the main tokens cycle with a period of 17 or\nmore, which the 16-frame repetition redraw forces) and the chunk is\nregenerated; a chunk that stops without eos is regenerated when it spoke\nunder its expected duration; and the frame cap grows by 1.5 s per `[cue]` so\na laugh or a pause is not cut off. Every sentence present went from 44 of 55\nruns to 38 of 40 on the cue lines; on six prose lines at five seeds the\nold and new rules score 30 of 30 and 29 of 30, the one difference being a\nline that both rules retry and where the new rule's random path lost. The\nSpace's sampling (temperature 1.0, top-p 0.9, top-k off) was then run over\nthe whole study (section \"The Space's sampling on prose\"): no prose row\nlost in 130, every sentence of the 20 cue lines, 5 retries instead of 19,\nthe two-sentence line reaching the end of speech on its first attempt with\nretries off in 5 of 5, and 0.007 more word error rate on prose (standard\nerror 0.004), mostly abbreviations and a few slurred words. It is the top of\nthe Expressiveness slider and the API's `expressiveness: 1`; the vendor's\nsampling stays the default. The Insert Cue menu says what remains: about one\ntake in twenty still drops the clause after a cue, so listen before\nexporting.\n\nWhile Drama Studio is open, including when only its menu bar icon is left, it\nserves an HTTP API on `http://127.0.0.1:8780`. `drama serve` serves the same\nAPI without the app. The API listens on the loopback interface only. Settings\nholds the port, a switch for starting the server with the app, and the token.\nThe menu bar has Start and Stop. When the port is taken, the menu shows\n\"API: Port 8780 is already in use.\"\n\nEvery route except `/v1/health` needs the bearer token from\n`~/Library/Application Support/DramaStudio/api-token`, created the first time\nthe server starts. The token keeps web pages, which can reach localhost, from\ndriving the engine. `drama serve --no-token` turns it off.\n\n```\nTOKEN=$(cat ~/Library/Application\\ Support/DramaStudio/api-token)\nAUTH=\"Authorization: Bearer $TOKEN\"\n```\n\nThe speech request follows the\n[SGLang Drama 3 cookbook](https://sgl-project.github.io/sglang-omni/cookbook/fishaudio_drama3.html):\n\n| Route | Does | \n|---|---|\n| `GET /v1/health` | `{status, model_loaded, engine: \"metal\" or \"mock\", busy, tuned, version}` .`tuned` is`null` until the model loads. While a generation runs it adds`estimate: {elapsed_seconds, remaining_seconds, frames_done, frames_expected}` ;`remaining_seconds` is`null` once the last sentence is in its silence tail or a retry runs. | \n| `GET /v1/models` | The one model, `fishaudio/drama-3` , with its snapshot id. | \n| `POST /v1/audio/speech` | Speaks `input` . Returns the audio file, or a PCM stream. | \n| `POST /v1/tts` | The same, with the text in `text` . | \n| `GET /v1/audio/speech/stream` | Speaks the `input` query parameter as Server-Sent Events. See[Streaming](#streaming-framing) . | \n| `POST /v1/audio/speech/live` | Speaks text while the client is still sending it, and streams the audio back on the same connection. See [Streaming](#streaming-framing) . | \n| `GET /v1/voices` | The voice library: id, name, transcript, seconds, source file. | \n| `POST /v1/voices` | Clones a voice: JSON `{name, transcript, audio_base64 or audio_path}` , or multipart with a`file` part plus`name` and`transcript` fields. | \n| `DELETE /v1/voices/{id}` | Deletes a voice. | \n| `GET /v1/takes/{id}` | One of the last 20 takes: `{id, input, text, seconds, frames, codes_base64, previous_take_ids}` . | \n\nThe speech body takes these fields:\n\n| Field | Meaning | \n|---|---|\n| `input` | The text. Performance cues such as `[laughs]` and`<\\|speaker:1\\|>` tags work as in the app. | \n| `voice` | A library voice, by id or name. | \n| `references` | Clips to clone on the fly: `[{audio_path or audio_base64, text}]` . | \n| `speaker_voices` | Library voices per speaker for a multi-speaker script: `{\"0\": \"Narrator\", \"1\": \"<id>\"}` , with no gaps from speaker 0. | \n| `response_format` | `wav` (default, 16-bit),`pcm` (raw 16-bit little-endian), or`m4a` . | \n| `stream` | `true` streams completed, checked chunks. | \n| `expressiveness` | 0 to 1, default 0.5: the app's Expressiveness slider. 0.5 is the vendor's sampling, 1 the Hugging Face Space's. | \n| `seed` | A fixed sampler seed, for repeatable takes. | \n| `character` | A voice designed from a description instead of a clip: `{name, description, scene?}` . | \n| `note` | A director's note, such as `\"softer and warmer\"` . The line is generated again as one turn, with the note as a direction and as a cue before every sentence. | \n| `previous_takes` | Earlier takes of the line, oldest first: `[{take_id}]` or`[{text, codes_base64}]` . The last two are context. With no`voice` or`references` , the first one is the voice. | \n| `apply_pronunciation` | `true` (default) rewrites`input` with the pronunciation rules in`~/Library/Application Support/DramaStudio/pronunciation.json` , the same rules the app uses, before it is spoken;`false` speaks it as sent. | \n\nA request takes at most one of `voice`, `references`, `speaker_voices`, and\n`character`. Every generation is a take: the response carries its id in\n`X-Drama-Take-Id`, and the server keeps the last 20. A `take_id` in\n`previous_takes` also brings the takes that take was made after, so naming a\nline's latest take carries its first take, the voice. Leave out `input` to\nretake the line of the last named take, with its voice and character.\n`codes_base64` is the take's codes as 16-bit little-endian integers, codebook\nby codebook, as `GET /v1/takes/{id}` returns them.\n\nA file response carries `X-Drama-Frames`, `X-Drama-Seconds`, and\n`X-Drama-Generation-Seconds`. The engine runs one generation at a time. A\nrequest that arrives while another runs, through the API or in the window, gets\n`429` with `Retry-After`. A bad request gets `400` with a plain-text message\nthat names the field.\n\n```\ncurl -sS localhost:8780/v1/health\ncurl -sS -H \"$AUTH\" localhost:8780/v1/audio/speech \\\n  -d '{\"input\": \"Hello there. [laughs] Good to see you.\", \"voice\": \"Narrator\"}' -o hello.wav\ncurl -sS -H \"$AUTH\" localhost:8780/v1/tts -d '{\"text\": \"The alias route.\"}' -o alias.wav\ncurl -sS -H \"$AUTH\" localhost:8780/v1/audio/speech \\\n  -d '{\"input\": \"Cloned on the fly.\", \"references\": [{\"audio_path\": \"/path/to/clip.wav\", \"text\": \"What the clip says.\"}]}' -o clone.wav\ncurl -sS -H \"$AUTH\" -D headers.txt localhost:8780/v1/audio/speech \\\n  -d '{\"input\": \"[low and menacing] Pray ye never find out.\", \"character\": {\"name\": \"Captain\", \"description\": \"gravelly old male voice, weathered and slow\"}}' -o captain.wav\nTAKE=$(awk 'tolower($1) == \"x-drama-take-id:\" {print $2}' headers.txt | tr -d '\\r')\ncurl -sS -H \"$AUTH\" localhost:8780/v1/audio/speech \\\n  -d \"{\\\"note\\\": \\\"softer and warmer\\\", \\\"previous_takes\\\": [{\\\"take_id\\\": \\\"$TAKE\\\"}]}\" -o captain-retake.wav\n```\n\nAll three streaming routes send completed chunks when retries are enabled. The engine validates each attempt before releasing audio, so a rejected attempt can be replaced for any voice.\n\nThe audio is always 16-bit little-endian mono at 44.1 kHz, as the\n`X-Drama-Format: pcm_s16le_mono` and `X-Drama-Sample-Rate: 44100` headers say.\n`stream: true` and the event stream also send `X-Drama-Take-Id` and\n`X-Drama-Estimated-Seconds` before the first audio.\n\n**PCM.** `stream: true` on `POST /v1/audio/speech` answers with\n`Transfer-Encoding: chunked` and raw PCM in the HTTP chunks. A chunk boundary\nmeans nothing; concatenate the chunks. Every chunk holds a whole number of\n2048-sample frames. Pipe it into a player:\n\n```\ncurl -sSN -H \"$AUTH\" localhost:8780/v1/audio/speech \\\n  -d '{\"input\": \"A long script, read as it is generated.\", \"stream\": true, \"response_format\": \"pcm\"}' |\n  ffplay -f s16le -ar 44100 -ch_layout mono -nodisp -autoexit -\n```\n\n**Server-Sent Events.** `GET /v1/audio/speech/stream` takes its fields as query\nparameters: `input` (required), `voice`, `expressiveness`,\n`seed`, and `character_name`, `character_description` and `character_scene`\nfor a designed voice. `apply_pronunciation=false` turns off the pronunciation\nrules, as in the speech body. It answers with `Content-Type: text/event-stream`\nand these events:\n\n| Event | `data` | \n|---|---|\n| `audio` | Base64 of the next PCM bytes, a whole number of frames. | \n| `sentence` | `{\"index\", \"text\", \"seconds\"}` after the last audio of each sentence. | \n| `done` | `{\"take_id\", \"frames\", \"seconds\"}` once the take is complete. | \n| `error` | `{\"message\"}` , after which the stream ends. | \n\nThe route still needs the bearer token, and the server sends no CORS headers,\nso a browser's `EventSource`, which cannot send an `Authorization` header, does\nnot work with the token on. Read the events with `fetch` and a stream reader\nfrom a page the server already trusts, or with any HTTP client.\n\n```\ncurl -sSN -H \"$AUTH\" \"localhost:8780/v1/audio/speech/stream?input=Hello%20there.%20Good%20to%20see%20you.\"\n```\n\n**Live text input.** `POST /v1/audio/speech/live` is for an agent that writes\ntext a piece at a time. It takes `voice`, `expressiveness`,\n`seed` and the `character_*` fields as query parameters, and the text as the\nrequest body:\n\n- Send the body with `Transfer-Encoding: chunked` , or with a`Content-Length` that covers everything you will send. The server reads it as it arrives.\n- The body is UTF-8 text in lines ended by `\\n` (or`\\r\\n` ). The server acts\non a line once its newline arrives.\n- The server joins the lines with spaces and speaks each sentence once a\nsentence end (`.` ,`!` ,`?` ,`;` ,`…` , or a CJK one, optionally followed by a\nclosing quote or bracket) arrives. Sentences are cut with the same splitter\nas`input` , so`<|speaker:N|>` tags and cues work.\n- The pronunciation rules rewrite each sentence before it is spoken, unless\nthe query has `apply_pronunciation=false` .\n- An empty line speaks the text held back so far, finished or not. The end of the body does the same, and a client may half-close its side of the connection to end the body.\n- Each sentence is a take, and the session's takes carry the voice and the\nhistory: the first take is the voice of every later sentence when no\n`voice` is given, and the last two are context.\n- The response is chunked PCM as for `stream: true` . Send`Accept: text/event-stream` to get the events above instead, with`done` carrying`{\"sentences\", \"seconds\", \"take_ids\"}` . The response ends once the\nbody has ended and its last sentence is spoken.\n- The session holds the engine from the request to the end of the response,\nso other requests get `429` meanwhile.\n\n`tools/live_speech.py` sends each line of its standard input as one chunk as\nsoon as it reads it, and writes the PCM to standard output:\n\n```\nsome-agent | tools/live_speech.py --voice Narrator |\n  ffplay -f s16le -ar 44100 -ch_layout mono -nodisp -\ntools/live_speech.py --events < script.txt      # prints each event with its arrival time\n```\n\n`curl -T -` also speaks a piped body, but it holds the piped lines back before\nsending them: on the mock engine its first audio came 1.7 s later than the\nsame feed through `tools/live_speech.py`.\n\nManage voices:\n\n```\ncurl -sS -H \"$AUTH\" localhost:8780/v1/voices\ncurl -sS -H \"$AUTH\" localhost:8780/v1/voices -F file=@clip.wav -F name=Narrator -F \"transcript=What the clip says.\"\ncurl -sS -H \"$AUTH\" -X DELETE localhost:8780/v1/voices/<id>\n```\n\n`tools/api_smoke.sh [base-url]` runs all of the above against a running server\nand checks each answer.\n\n`drama-mcp` is an MCP server over stdio (protocol 2025-06-18) that calls the\nHTTP API. `tools/make_app.sh` puts it at\n`DramaStudio.app/Contents/MacOS/drama-mcp`; `swift build` puts it at\n`.build/debug/drama-mcp`. When nothing answers, it opens Drama Studio and waits\nup to 90 seconds for `/v1/health`. It looks for the app in `DRAMA_APP_PATH`,\nthen the app that contains it, then `/Applications/DramaStudio.app`, then the\nrepository's `dist/DramaStudio.app`. `DRAMA_API_URL` overrides\n`http://127.0.0.1:8780`. It reads the token from the same file.\n\n| Tool | Does | \n|---|---|\n| `synthesize_speech` | `text` , with optional`voice` or`character` (`{name, description, scene?}` ),`output_path` ,`expressiveness` , and`play` . Writes a WAV, by default to`~/Library/Application Support/DramaStudio/Output/<timestamp>.wav` . Returns the path, seconds, frames, generation seconds, and`take_id` .`play` plays it with`afplay` . | \n| `retake_with_note` | `take_id` and`note` , with optional`output_path` and`play` . Generates that take's line again in the same voice with the note. Returns what`synthesize_speech` returns, with the new`take_id` . | \n| `list_voices` | The voice library. | \n| `add_voice` | Clones `audio_path` with its`transcript` under`name` . | \n| `delete_voice` | Deletes a voice by `id` . | \n| `studio_status` | Health, the model snapshot, and the API address. | \n\nThere is no MCP tool for streaming or live text input. An MCP tool call is one\nrequest and one result, so it cannot carry audio while text is still being\nwritten; an agent that wants that calls `POST /v1/audio/speech/live` itself.\n\nRegister it with Claude Code:\n\n```\nclaude mcp add drama-studio -- /Applications/DramaStudio.app/Contents/MacOS/drama-mcp\n```\n\nOr with Claude Desktop, in `~/Library/Application Support/Claude/claude_desktop_config.json`:\n\n```\n{\n  \"mcpServers\": {\n    \"drama-studio\": {\n      \"command\": \"/Applications/DramaStudio.app/Contents/MacOS/drama-mcp\"\n    }\n  }\n}\n```\n\n`drama-say` speaks through the running app the way `/usr/bin/say` speaks\nthrough macOS. It sends `POST /v1/audio/speech`, finds and opens the app the\nway `drama-mcp` does, and loads no model of its own. `tools/make_app.sh` puts\nit at `DramaStudio.app/Contents/MacOS/drama-say`.\n\n```\ndrama-say \"Hello there. [laughs] Good to see you.\"\ndrama-say -v Narrator -o hello.wav -f chapter.txt\ndrama-say --character \"Captain: gravelly old male voice\" --verbose \"Pray ye never find out.\"\ndrama-say --retake \"$TAKE\" --note \"softer and warmer\"\n```\n\nWith no `-o`, it plays the line as it generates: it sends `stream: true` and\nplays each completed chunk when it arrives. Startup time depends on model\nloading, the script, the voice and retries; it is not a fixed delay. `-o FILE` writes a WAV, M4A or PCM file, chosen by\nthe extension, and `-o -` writes raw PCM to standard output. With no text and\nno `-f`, it reads the text from standard input. `-v '?'` lists the voice\nlibrary. `--verbose` prints the take id that `--retake` takes. `drama-say --help` lists every option.\n\nIt prints nothing on success. It exits with 64 on a bad command line, 75 when\nthe studio is busy with another generation, after printing the `Retry-After`\nwait, and 1 on any other error, after printing the server's message.\n\n`DramaEngine` is a Swift library, version 2.1.2 (`DramaEngineVersion.string`,\n`CHANGELOG.md`). Add this repository as a package dependency and the\n`DramaEngine` product to a target; the app, the server and the CLI are not\nneeded. The public API includes `ModelLocation` and `DramaSetup` for the\ninstall, `DramaSynthesizer` (an actor) behind the `SpeechSynthesizer`\nprotocol, `SpeechRequest` with `ReferenceVoice`, `VoiceDesign`, `Take` and\n`GenerationConfig`, the streamed `SpeechEvent` s with `StreamedAudio` and\n`GenerationEstimate`, `PronunciationDictionary`, `DramaCodec` for standalone\nencode and decode, and `MockSynthesizer` for previews and tests. Everything\nelse in `Sources/DramaEngine` is `package`: the repository's targets use it,\nother packages cannot.\n\n``` python\nimport DramaEngine\n\nlet report = DramaSetup.check(location: .huggingFaceCache())\nguard let location = report.location, report.isReady else { fatalError(report.fixes.joined(separator: \"\\n\")) }\nlet engine = DramaSynthesizer(location: location, codecDirectory: DramaCodec.defaultDirectory)\nlet voice = try await engine.encodeReference(audioURL: clip, transcript: \"What the clip says.\", name: \"Narrator\")\nvar request = SpeechRequest(text: \"Hello there.\", references: [voice])\nrequest.config.streaming = .defaultLowLatency\nrequest.config.maxRetries = 0 // Allow partial playback before a chunk is validated.\nvar audio = StreamedAudio()\nfor try await event in engine.synthesize(request) {\n    play(audio.newSamples(from: event))\n}\n```\n\n- `DramaSetup.check` reads files only and names what is missing: a weight\nshard, the tokenizer,`codec.pth` , the Core ML codec models, or a tuned\nkernel schedule for this Mac, each with a fix.`DramaSetup.prepare` builds\nthe codec models with the`convert_codec.py` that ships in the engine's\nresource bundle, through a Python you name (it throws the exact commands\nwhen you name none and never downloads Python), and loads the model once to\ntune the kernels.\n- `Examples/SpeakCLI` is a separate package that depends on this one by path,\nclones a voice from a clip, streams progress, and writes a WAV in 67 lines.`Examples/README.md` has the Xcode steps and the entitlements: none beyond\nthe app's own, plus the microphone only to record clips.\n- The DocC catalog (`Sources/DramaEngine/DramaEngine.docc` ) has the landing\npage and the articles \"Getting the model\", \"Voices\" and \"Streaming\". Build\nit with`xcodebuild docbuild -scheme DramaEngine -destination 'generic/platform=macOS' -derivedDataPath .build/docbuild` ; no DocC plugin\nis needed.`swift test --filter SDKUsageTests` runs the landing page's\nexample on`MockSynthesizer` and fails if the page's copy drifts.\n- `tools/api_audit.py` (after`swift package dump-symbol-graph --minimum-access-level public` ) lists the public symbols and fails on one\nwithout a doc comment or a public value type that is not`Sendable` .\n- `tools/make_xcframework.sh` builds`dist/DramaEngine.xcframework` for macOS\non Apple silicon, with the Metal kernel sources and the conversion script\ninside the framework, then builds and runs`SpeakCLI` against it.\n\n- `Sources/DramaEngine/Weights` : the eight safetensors shards are memory-mapped\n(`MAP_SHARED` ) and wrapped as Metal buffers without copying. Experts stay in\nFP8 E4M3 with their 128x128 block scales and are decoded inside the kernel\nthrough a 256-entry table; every other weight is bf16 widened to fp32 in the\nkernel.\n- `Sources/DramaEngine/Kernels` : zero-centered RMSNorm, a bf16 GEMV for decode\nand a simdgroup-matrix GEMM for prefill, the Gated DeltaNet depthwise conv and\nrecurrence, GQA attention with an online softmax over an fp32 KV cache,\npartial RoPE (NeoX for the backbone, interleaved for the tower), the MoE\nrouter and expert-major FP8 kernels that dispatch only over active experts\n(one-row kernels for decode, tiled simdgroup-matrix kernels for prefill),\nthe GPU sampler, and the codebook tower's embedding mixing. Tile shapes,\nthreadgroup sizes and the attention key split are fields of a`KernelSchedule` that`Autotune` fills per machine (see below).\n- `Sources/DramaEngine/Model` : the backbone and tower runners, the reference's\nsampler (top-k, top-p, exponential-noise argmax, repetition-aware retry of the\nmain token) on the CPU and as Metal kernels that reproduce its picks and\ngenerator stream bit for bit, prefill and decode sessions, and the chunked\nframe generator. A decode frame is one command buffer, with the next frame's\nbuffer committed before the current one is waited for.\n- `Sources/DramaEngine/Prompt` : the`reference-bracket-v1` prompt builder, the\nreference's chunker and history policy, plus a sentence-per-turn mode.\n- `Sources/DramaEngine/Tokenizer` : a byte-level BPE tokenizer that matches\nHugging Face`tokenizers` on a 661-case corpus.\n- `Sources/DramaEngine/Codec` : Core ML wrappers for the DAC encoder (reference\nclips to codes) and decoder (codes to 44.1 kHz audio), both exported at one\nstatic 512-frame shape and run in windows with 256 frames of left context,\nplus the same decoder at a 64-frame shape for the silence stop's probe.\n- `Sources/DramaServer` : the HTTP API on`Network.framework` . Request parsing\nand routing are pure functions from an`HTTPRequest` to an`APIRequest` ;`APIService` answers them against any`SpeechSynthesizer` and`VoiceStore` ,\nso the app,`drama serve` , and the tests share it.\n- `Sources/DramaMCP` and`Sources/drama-mcp` : the MCP server, JSON-RPC 2.0\nover stdio, proxying the HTTP API.\n- `App/DramaStudio` : the SwiftUI app, with the API server and the menu bar\nextra in`App/DramaStudio/Server` .\n- `eval/` : the faithfulness suite (see`docs/EVAL.md` ).\n\nCurrent UI checks and responsiveness measurements are in\n[UI verification](https://github.com/maralski/drama-studio/blob/main/docs/ui-verification.md). Historical engine experiments below\nrecord their original configurations; they are not current latency guarantees.\n\nDesign notes and decisions are in `docs/DESIGN.md`; the decision trail of the\nbuild is `.audit/drama-studio.tsv`.\n\nThe engine is checked against the fish-speech v3.0.0-beta PyTorch\nimplementation running on the CPU in float32, teacher-forced on the same\nprompt and frames. Numbers from `eval/compare.py`:\n\n| quantity | relative L2 vs float32 oracle | top-1 agreement | mean KL | \n|---|---|---|---|\n| backbone hidden state, 121 positions (worst position 1.3e-6) | 5.7e-7 |  |  | \n| main-token candidate logits | 3.0e-7 | 1.000 | 1.3e-12 | \n| nine tower head logits | 4.4e-7 | 1.000 | 4.2e-12 | \n| residual stream after each of the 40 layers, and each of the 13 tower layers, at the first step | under 2e-6 each |  |  | \n\nFor scale, the same oracle run in bfloat16 on the CPU differs from its float32 run by a relative L2 of 5.2e-2 on the hidden state with a top-1 agreement of 0.846. The Metal engine sits at float32 precision; it keeps the FP8 expert weights exact and widens everything else, so it does not reproduce the CUDA release's W8A8 activation quantization, and no claim is made about that.\n\nThe sampler is checked separately: over 1174 sampled picks, none falls outside the reference's top-k / top-p set, and the count of rank-0 picks matches the expected value within noise.\n\nEnd-to-end, the sentences the model does speak transcribe cleanly under\nWhisper, but with the reference's chunking it often stops after the first\nsentence of a two-sentence prompt or trails into silence, and the reference\nPyTorch implementation does the same on the same text and seeds. The\nsentence-per-turn mode and the retry rule below exist for that reason.\nSee `eval/README.md` for how to reproduce every number.\n\nThe codec uses Core ML to access Apple's inference backends. In the recorded experiments on macOS 26.5 and an M3 Max, the Neural Engine was slower than the CPU:\n\n- The decoder's Neural Engine compile for the 512-frame shape ran for more than\n35 minutes without finishing; the 64-frame shape compiled in about 7 minutes\nand then ran three times slower on the ANE than on the CPU, because the\ncompute plan puts 72 percent of the decoder's cost (long-sequence\nconvolutions and the Snake activation's `sin` after upsampling) on the CPU\nanyway.\n- The ANE's native transposed convolution lost about 15 dB of accuracy, so the export rewrites transposed convolutions as polyphase convolutions.\n- The GPU path returns wrong audio for the chained residual units of the last decoder block (about 5 dB SNR); each unit alone is correct, which points at op fusion, though the cause was not isolated.\n- Core ML's enumerated shapes crash the CPU backend (SIGTRAP) on multi-channel convolutions here, so the models use one static shape.\n\nSo the shipped configuration runs the encoder in fp32 and the decoder in fp16\non the CPU through Core ML: 1.5 s per 512-frame window (23.8 s of audio) for\nthe decoder and 2.1 s for a 10 s reference clip in the encoder. The encoder\nmatches PyTorch's codes exactly; the decoder is 35 to 39 dB from the fp32\nreference. `DramaCodec.init(computeUnits:)` still accepts `.cpuAndNeuralEngine`\nfor anyone who wants to retry on a different machine or OS.\n\nMeasured with `eval/e2e.py` on six texts, plain and cloned, three seeds each\n(36 runs per row), scored by Whisper small.en word error rate, whether the model\nreached eos, trailing silent frames, and resemblyzer speaker similarity to the\nreference clip for the cloned runs.\n\n| configuration | mean WER | runs at or under 0.1 WER | eos reached | mean quiet tail | speaker similarity | \n|---|---|---|---|---|---|\n| reference chunking (300 bytes), no probe | 0.141 | 69% | 89% | 103 frames | 0.867 | \n| one sentence per turn | 0.063 | 86% | 100% | 30 frames | 0.867 | \n| sentence per turn, silence stop, short-chunk retry (previous defaults) | 0.016 | 92% | 100% | 2 frames | 0.867 | \n\nThe app uses the shipped chunking and retry policies. The `drama` CLI exposes\n`--chunking` for unchecked comparisons; checked generation always uses sentence\nturns. `--no-retry` and `--max-frames` remain available.\n\nUncloned outputs score 0.54 to 0.56 against the reference clip, so the cloned\n0.87 is the voice carrying over, not the metric's floor. The only misses in the\nlast row are Whisper rendering \"the 21st of March\" as \"21 March\" in three runs;\nevery other run transcribes exactly. The retry rule's threshold was chosen on\nthe same 72 runs of the first two rows and then applied to the third, so treat\nit as tuned to this sentence set. The word error rates above use the PyTorch\ncodec to decode. Decoding the same 36 frame sequences with the shipped Core ML\nfp16 decoder instead (`eval/coreml_vs_pytorch_wer.py`) gives the same mean\nword error rate, 0.063 against 0.063, with no run scoring worse.\n\nMedians of five interleaved runs against the previous engine on an idle\nmachine (`eval/perf_ab.sh`, raw runs under `eval/out/perf/ab_final/`). GPU\ntimes come from `drama bench`; the end-to-end rows are wall time from\n`drama generate --warm` on a 324-row prompt (a 10 s reference clip plus text)\nand 600 frames.\n\n| step | before | now | \n|---|---|---|\n| weights mapped and kernels compiled | 0.4 to 0.7 s | 0.4 to 0.7 s | \n| warm-up (page walk, pipelines, first command buffer) | 4 s cold, then a 0.37 s first forward | 4 s cold; the first forward then costs 0.05 s | \n| backbone prefill, 128 rows | 647 ms GPU | 213 ms GPU | \n| tower prefill, 128 rows | 171 ms GPU | 38 ms GPU | \n| backbone plus tower prefill, 300 rows in one chunk | three 128-row chunks | 382 plus 93 ms GPU | \n| end-to-end prefill of a 324-row prompt | 2.44 s | 0.72 s | \n| backbone decode, one frame | 17 to 18 ms GPU | 17 to 18 ms GPU | \n| tower decode, one frame | 6.7 ms GPU | 6.7 ms GPU | \n| decode, one frame | ten command buffers, sampling on the CPU between tower layers | one command buffer, 25.6 to 27.0 ms GPU, the next frame's buffer committed before the current one is waited for | \n| end-to-end decode | 31.5 frames/s (range 28.7 to 31.6) | 38.5 frames/s (range 37.0 to 39.1), 1.8x real time | \n\nPrefill runs dense projections through a simdgroup-matrix GEMM and the FP8\nexperts through tiled kernels that dequantize each weight tile once; at 128\nrows nearly every expert is active, so a chunk reads all 30 GB of expert\nweights once and the chunk is 512 rows so a 10 s reference clip is one pass.\nSmall chunks pay for that choice: an 8-row chunk takes 66 ms instead of 42,\nbecause one tile shape serves every row count so that a row's result is the\nsame in any chunk (the prefix cache depends on it). Decode is bound by weight\nbandwidth: the one-row backbone forward reads about 3.4 GB of weights in 17\nto 18 ms, about 200 GB/s averaged over its 600 small dispatches, and the\n`no-experts` bench delta puts the eight active experts at about 250 GB/s.\n`drama profile --rows N` prints the GPU time of each kernel group on the real\nweights (one layer repeated, so its numbers can be cache-warm), and\n`drama kbench` times the GEMM tile shapes and the expert kernels on synthetic\ndata. The before column comes from the same session as the now column; the\nsame baseline binary measured 583 to 586 ms for the 128-row prefill in two\nearlier series that day, so day-to-day drift of about 10 percent is normal.\n\nThe kernel changes keep every fp32 product and change only summation orders;\nthe faithfulness table above is from the new engine (hidden 5.7e-7, candidate\nlogits 3.0e-7, tower logits 4.4e-7 relative L2, top-1 1.0; it was 6.5e-7,\n3.3e-7 and 5.0e-7 before). The GPU sampler reproduces the CPU sampler's picks\nand generator stream: 160,000 chained picks identical in the long parity run\non the pre-merge code (`DRAMA_SAMPLER_PARITY_SAMPLES=20000`), 16,000 in the\ndefault run on the merged code, and every generation compared between the\nCPU and GPU samplers on the previous kernels produced the same frames.\nReordered sums do change the result on some prompts: in the end-to-end suite\n28 of 36 runs produced the same frames as before, one differed only in where\nthe codec silence probe stopped it, and 7 differed deterministically, with\nthe summary metrics unchanged (mean word error rate 0.016 in both, eos\nreached in all runs, speaker similarity 0.870 against 0.867). On the one such\nprompt examined (`eval/specs/qshort.json`), the two engines agree to 2e-7\nthrough layer 29 and part at layer 35, the shape of a discrete routing change\non a near-tie (the router ids themselves were not dumped). A fresh float32\noracle dump of that prompt takes the new engine's side: 9.5e-7 on the hidden\nstate, where the previous engine was off by 3.9e-3 and failed the suite's\nthresholds. That holds for this one prompt only; the faithfulness gate above\nruns on a single prompt and would not catch the same kind of divergence on\nanother.\n\nA second round on the decode frame, run as a hillclimb: seven experiments in\nsmall synthetic settings first, then one change at a time into the real\nframe, each kept only when the output stayed bit-identical and the frame got\nmeasurably shorter. The decision log is `.audit/hillclimb.tsv`.\n\nWhat the experiments established before anything changed. A decode frame was 1312 dispatches streaming 5.94 GB of weights in 21.7 ms, and its large GEMVs already ran at 346 to 378 GB/s against a measured read ceiling of 383 GB/s. At that rate the bytes alone take 15.5 ms, so the 30 percent target of this round (14.6 ms) sits below what any kernel change can reach; fewer bytes per frame were the only route, and the only route that keeps the output exact is lossless compression. The bf16 mantissas carry 6.97 of 7 bits and are incompressible; the exponents carry about 2.8 bits of 8, so a 3-bit exponent code with a per-tile window and byte escapes packs a weight into 11.3 bits with exact round trip. The kernel that unpacks it is compute-bound at about 285 GB/s of physical reads, so it wins only on the K=2048 shapes with wide N.\n\nWhat shipped, every step with a byte-identical dump and the faithfulness table unchanged (hidden 1.2e-6 relative L2 at most, top-1 1.0):\n\n| step | change | frame (bench rows 1, backbone + tower), under load | quiet GPU, 2026-10-23 | \n|---|---|---|---|\n| baseline |  | 21.1 ms | 21.0 ms (15.0 + 6.0) | \n| MoE layer | one decode row's mixture of experts as three dispatches (bf16 router and shared pass, routing plus routed gate-up, routed down plus shared down plus combine), FP8 decoded with integer ops instead of the lookup table | 20.2 | 20.1 | \n| encoder | a concurrent compute encoder with a barrier before every dispatch except the later members of a sibling group (the four GDN input projections, q k v, w1 and w3, and so on); q/k norm, rope and the KV writes in one dispatch | 19.7 | 19.5 | \n| samplers | 256-thread sampling kernels in place of one 1024-thread threadgroup; no change on a quiet GPU, and 47 to 22 ms per frame when another GPU client is active, which is what a user with an animating window sees; on a quiet GPU the kernels cost 0.67 ms per frame, about 0.2 ms more than the old shape, and the old shape stalls under the window server alone | 19.7 | not re-benched | \n| tower | embedding plus both concat norms in one dispatch, attention split over 8 threadgroups with an in-order combine and the gate folded in, the residual adds folded into wo and w2, w1 and w3 as one dispatch with the SiLU | 19.6 | not re-benched | \n| packed weights | eight weight families (GDN qkv and z, attention q, the semantic lm-head slice, tower wqkv, heads, w1 and w3) read from an exponent-coded copy at one row, 0.94 GB fewer bytes per frame | 18.2 | 18.1 (12.9 + 5.2) | \n\nReverted: a multi-matrix GEMV that grouped sibling projections into one dispatch (within noise once the concurrent encoder had taken the same gaps), more threadgroups for the MoE down kernel (each doubling cost 3 to 9 us per layer in its prologue), and the load folds of norms, gates and SiLU into the GEMVs (no gain on the tuned schedule). The frame went from 1312 dispatches to 759.\n\nThe packed copy lives under `~/Library/Application Support/DramaStudio/Packed/`,\n2.1 GB per checkpoint, built on the first load in about two seconds across\nall cores and verified by a full unpack before it is used; a cache that fails\nto build or load falls back to bf16 with the reason logged, and\n`DRAMA_PACKED=0` turns it off without a rebuild. Prefill, chunk invariance\nand the prefix cache never touch it: every path with more than one row reads\nthe bf16 shards as before.\n\nCaveats, and what a quiet session settled (2026-10-23). Every number in the\nhillclimb was taken while another process held the GPU (the window server\nand a virtual machine), so the ranges carried stalls and no end-to-end figure\nwas quoted. A second session rechecked the round with the virtual machine\nstopped and the app quit, one job at a time. The `ioreg` utilization counter\nreads 100 percent under the window server alone and is no gate; the gate was\nthe sampler stages of a traced 120-frame generation (`eval/quiet_label.sh`),\n0.7 ms per frame when the GPU is free, taken before and inside every batch.\nThe tune was redone there: the decode GEMV moved to 8 simdgroups, the narrow\nGEMV kept its tile this time, the MoE prefill tile went to 32x32x32, and the\n128-row forward went from 233 to 189 ms. The quiet column above is a\n10-round interleaved bench per pair except the baseline, 5 rounds. The\nconcurrent encoder stays: against a serial-encoder build of the same code it\nis 0.5 ms shorter on the bench line and 0.0 to 1.2 ms shorter per frame over\nthree interleaved 300-frame traces, with a fatter tail (p99 39 to 41 ms\nagainst 33 to 35), which playback's 64-frame buffer absorbs. The parked\nmulti-matrix GEMV was rebased (branch `hill/step3-gemv-decode-quiet`) and\nmeasured again: 0.1 ms, inside the A/A gap of the bench, so it stays parked.\nEnd to end, `eval/perf_ab.sh` over five 425-frame generations: 35.1 frames\nper second [31.7..36.4] against 20.7 for the baseline, whose 1024-thread\nsamplers stall under the window server alone. Short runs after idle reach\n48 to 50 frames per second (19.2 to 19.7 ms per frame); the sustained figure\nis lower because this Mac slows bandwidth-bound work by 20 to 40 percent\nafter minutes of GPU load and recovers over minutes of idle, so absolute\ndecode numbers depend on the GPU's recent history while interleaved A/B\ndeltas hold. The remaining ideas are small: the MoE down kernel's prologue,\nand the w2 and 4096x2048 families in the packed path.\n\nDefault sentence chunking keeps short complete sentences separate. It no longer folds a short opening sentence into the next one, which could let the model end after the opening and omit the rest of that chunk.\n\n`GenerationConfig.streaming` requests when decoded audio leaves the engine.\nWhen `maxRetries` and `minSpokenRatio` are positive, every request uses\n`.sentence` mode. Each chunk is decoded and checked before its selected attempt\nleaves the engine. Checked generation also verifies both complete neighboring\nchunks together before exposing their join. A mismatch retries only the current\nunplayed chunk within the existing attempt limit. Already emitted audio stays\nfixed. This applies to every voice, plain text, and performance cues.\n\nWith explicitly unchecked preview and retries disabled,\n`.lowLatency(stepFrames: 64)` sends partial audio every\n64 generated frames, about 3 seconds. The CPU decodes those frames while the GPU\ngenerates the next ones. Such a chunk cannot be replaced after playback begins.\n\nNormal generation waits for the model's end-of-speech signal, subject to its frame cap and loop guard. It does not treat silence as completion, because a voice can pause before the remaining words. Explicit silence probes run at fixed frame checkpoints and finish before sampling resumes, so decode timing cannot change the stop frame. Normal generation checks the decoded words with on-device speech recognition. Missing, changed, or uncertain words trigger a retry before audio enters playback or generation history. Cue retries alternate the requested sampling with broader replacements and ask for precise enunciation. Exhausted retries report a failure. Reference pace affects the time allowance, not the words that must be spoken.\n\nExplicit recognition labels such as `*gasp*` or `(gasps)` count as nonverbal\nannotations only when the script requests that cue family. Ordinary spoken\nwords remain required. The native checker selects an installed regional locale\nfrom your macOS language preferences, with a deterministic fallback.\n\nAcceptance policy revision 5 records the verified-sentence boundary rule. Archived takes retain their original check revision; playback does not upgrade their verification history. Opt-in local audits record the exact PCM identities, unit indices, token and confidence-span positions used by the boundary check.\n\nThe decode reuses the windows a full decode of the chunk would run\n(`DecodeWindow.covering`), so a chunk's partials join into exactly its audio.\nOn the Core ML codec, 600 frames decoded in 64-frame steps differ from one\nfull decode by a maximum absolute error of 0\n(`DRAMA_CODEC_SEAM=1 swift test --filter testSeamOnTheRealCodec`); decoding\neach step with a bare 256-frame left context instead differs by 0.027. All\nfull-window codec work of a run goes through one serial queue: a low-latency\nchunk's last frames decode while the next chunk prefills and generates. The\noptional silence stop reads fixed frame checkpoints through the 64-frame probe\nshape. Each reading completes before generation samples another frame. Readings\nstart after the voice-aware expected duration plus the configured silent tail.\n`drama generate --probe-every N` and `--probe-after-expected no` explicitly enable\nthe probe for measurement. In sentence mode, each attempt's full audio is decoded\nonce at its end.\n\nThe measurements below describe the earlier asynchronous streaming implementation.\nThey predate universal retry buffering and the removal of silence stopping from\nnormal generation. `eval/reliability/REPORT.md` records the earlier probe timing\nexperiments.\n\nMeasured on the M3 Max (16 cores, 12 performance), with the same two-sentence\nline, 9.4 s of audio in 202 frames, seed 7, in every run. The machine was not\nidle: other users' jobs kept the load average at 5 to 7 (two Python processes\nat a full core each), so every comparison interleaves its sides, and the raw\nrows are in `.audit/streaming.tsv`.\n\nTime to the first audio byte of `stream: true` through `curl -N`\n(`tools/streaming_ttfb.py`, 5 runs per side, alternating). Before is the\nengine at `9e645ef`, which decoded every chunk after the whole request was\ngenerated. Both sides returned the same 827,392 bytes, with HTTP 200 and curl\nexit 0 every time. Server CPU comes from `ps -o time=` before and after each\nrequest:\n\n|  | before | after (low latency) | \n|---|---|---|\n| first audio byte | 12.95 s (12.61 to 13.20) | 4.49 s (4.48 to 4.60) | \n| last byte | 14.48 s (14.14 to 14.72) | 8.57 s (8.49 to 8.68) | \n| server CPU per request | 39.0 s | 22.6 s | \n| average cores in use | 2.69 | 2.65 | \n| peak cores over any 0.5 s | 4.3 (3.8 to 4.7) | 4.0 (3.8 to 4.5) | \n\nThe first byte now comes after the prefill (0.7 s), 64 frames (1.8 s), and one decode of a 512-frame window (about 2 s). The request also uses less CPU: before, the silence probe decoded the whole chunk so far every 16 frames whenever no probe was running, and every chunk was decoded once more at the end; now each window is decoded once.\n\nGeneration speed with the decode on and off, in one process\n(`DRAMA_STREAMING_BENCH=1 swift test -c release -Xswiftc -enable-testing --filter StreamingBenchTests`, 5 runs per mode, modes interleaved, medians\nwith ranges). \"Decode after\" turns the silence stop off, so the codec is idle\nwhile frames land and runs once per chunk at its end:\n\n| mode | frames/s while frames land | end-to-end frames/s | first audio | wall | CPU cores | \n|---|---|---|---|---|---|\n| decode after | 34.9 (34.7 to 37.5) | 19.1 | 5.07 s | 10.58 s | 1.20 | \n| sentence | 33.0 (32.2 to 33.7) | 15.2 | 6.51 s | 13.30 s | 2.03 | \n| low latency, 64 frames | 32.7 (32.4 to 33.6) | 18.9 | 6.77 s | 10.68 s | 2.11 | \n| low latency, 32 frames | 32.9 (32.8 to 33.5) | 19.2 | 6.03 s | 10.50 s | 2.17 | \n\n- The CPU decode slows generation by about 6 percent while it overlaps the GPU (34.9 to 32.7 frames/s). Low latency still finishes with \"decode after\" and 2.6 s ahead of sentence mode, because the GPU never waits for the codec.\n- What limits the first audio is the decode, not the GPU. One window takes\n1.6 s alone (\"decode after\" spends 3.35 s on two) and 4.6 s while the GPU\ngenerates: in the event trace of the first low-latency run, the window was\nqueued at 2.50 s and its audio arrived at 7.09 s. The cause of the slowdown\nwas not isolated; contention for memory bandwidth or for the chip's power\nbudget is a guess. The test runner also sees slower decodes than `drama serve` (first audio 6.77 s here against 4.49 s through the server), and\nthat cause was not isolated either.\n- For this short first sentence (4.7 s of audio), low latency is no faster than sentence mode to the first audio. It pays off on a long first sentence: on one 12.5 s sentence (3 runs), low latency started at 6.19 s (6.09 to 6.94) against 23.5 s for sentence mode. Part of that gap is the retry: sentence mode judged the first attempt too short and generated the sentence again, while low latency kept that attempt (336 to 369 frames).\n- 32-frame steps start 0.7 s sooner but stutter: in every trace the first 1.5 s of audio ran out 1.1 to 1.5 s before the next partial arrived. With 64-frame steps every trace played without a gap, so 64 stays the default.\n\nThe app shows the time left beside the elapsed time while it generates\n(`GenerationEstimate`): the expected frames of the unfinished chunks (text\nlength at 16 characters per second, or the reference voice's slower pace,\ntimes 21.53 frames per second) less the\nframes of the current chunk, over a 3 s moving average of frames per second\nthat starts at 35, plus one prefill per chunk not yet prefilled (0.7 s until\nthe first is measured). It reads \"about 12 s left\", rounded to 5 s above 20 s,\nand \"finishing…\" or \"retrying\" once the frames stop predicting the end. The\nsame numbers are `X-Drama-Estimated-Seconds` and the `estimate` field of\n`/v1/health`.\n\nKernel schedule parameters (the GEMV token tile and rows per threadgroup, the\nGEMM tile, the MoE tile and K step, the one-row expert kernels' rows per\nthreadgroup, the attention key split, the GDN threads per state column) are\nnot constants any more. On the first load for a given (chip, macOS version,\nkernel source digest, model snapshot) the engine times each kernel family's\nreal workload on the resident weights, rotated over ten layers, for every\nvalue in its declared domain, keeps a candidate only when its output is within\na relative L2 of 1e-5 of the default's on the same inputs and it is at least 5\npercent faster over five interleaved rounds, then times whole forwards of 128\nand 300 rows and of one row on both schedules and drops a side's choices\nunless the tuned one wins by 2 percent at every row count. The result is a\nJSON report under `~/Library/Application Support/DramaStudio/Tuning/`; every\nlater load reads it and measures nothing. `drama tune` runs it from the\ncommand line and prints the table (about 30 s here); the app shows \"Tuning\nkernels for this Mac\" once; `drama schedule-ab` times any two schedule files\nagainst each other inside one process. The idea is Magnitude's (tune,\nvalidate, cache per device); the kernels, the domains and the measurement scripts are\nthis project's. Nothing in the numerics changes beyond summation order: the\ntuned engine scores hidden 5.8e-7, candidate logits 3.0e-7 and tower logits\n4.4e-7 against the float32 oracle with top-1 1.0 (`eval/out/perf/compare_tuned_final.txt`),\nand the A/B prompt below produced byte-identical frames with and without the\ntuned schedule in all fourteen compared runs.\n\nWhat it shipped on this M3 Max (`eval/out/perf/tune_final2.txt`): two decode\nchanges, a token tile of 1 with two simdgroups per threadgroup for the one-row\nGEMVs (27 percent faster per kernel) and a 16-way key split for one-row\nattention (27 percent). Everything on the prefill side stayed at the\nhand-picked values: the GEMM tile, the MoE tile and the GDN split never\ncleared the 5 percent bar, and the narrow-projection GEMV tile and the 4-way\nprefill attention split, 80 and 5 percent faster per kernel, moved the whole\n128- and 300-row forwards by 1.2 and 0.4 percent, under the margin, so the\nwhole-forward check reverted them. Two earlier versions of the measurement script would\nhave shipped a slower prefill: random activations route evenly over the 256\nexperts and favour a small MoE row tile that was 9 percent faster in the\nmeasurement script and 5 to 12 percent slower in the real prefill, and a 2 percent\nminimum gain measured at 128 and 32 rows let through a tile that lost 9\npercent at 300 rows. Both are in `.audit/autotune.tsv`, and the isolation\nthat caught the second is `eval/out/perf/schedule_ab_1.txt`.\n\nMeasured, with the shipped schedule against the default:\n\n| measurement | untuned | tuned | \n|---|---|---|\n| backbone forward, one row, GPU, `drama schedule-ab` in one process, 7 interleaved rounds | 19.2 ms [18.4..19.5] | 15.3 ms [14.8..15.5], faster in 7 of 7 | \n| same, `drama bench` in separate processes, two five-run series | 18.0 [17.2..18.3], 17.9 [17.5..18.2] | 15.1 [14.7..15.8], 15.4 [14.8..16.7] | \n| tower decode, one row, GPU, `drama bench` | 6.9, 6.8 ms | 6.1, 6.2 ms | \n| end-to-end decode, 425 frames after a 324-row prompt, `drama generate` , five-run medians | 32.9 [15.0..37.1], 29.7 [21.8..33.1] frames/s | 33.5 [14.9..42.2], 35.1 [32.3..36.8] frames/s | \n| same, GPU ms per frame, the three rounds of the first series where neither side was disturbed, then all five of the second | 26.9, 27.9, 30.4; 46.0, 34.6, 32.9, 33.7, 30.2 | 24.6, 23.7, 29.8; 31.0, 28.5, 27.2, 27.7, 29.7 | \n| backbone prefill, 128 and 300 rows | unchanged: the shipped schedule runs the same kernels ( `schedule-ab` 196.9 against 201.0 and 382.9 against 382.0 ms, the instrument's own spread) |  | \n\nThe one-row forward is the solid number: about 20 percent less GPU time in\nfour series (the two bench series, the in-process A/B and the tuner's own\ncheck at 17.1 against 14.6 ms), ranges not overlapping, from the decode GEMV\ntile and the attention split, both of which put more and smaller threadgroups\non the same bytes. The end-to-end frames-per-second rows are softer: the two\n`drama generate` series ran on a machine that was not idle (other sessions'\ntest jobs and the window server busy; load average 4.8 to 12), their ranges\nare wide, and in the first series two of five rounds were disturbed on both\nsides for a reason the load averages do not show, so the five-run medians\nmove only 2 and 18 percent while the per-frame GPU time is lower on the tuned\nside in every round where neither side was disturbed. Those two series ran\nwith an earlier cache whose decode fields equal the shipped ones and whose\nprefill fields have since been reverted. Prefill has no measured change and\nnone is expected: with the reverted fields the schedule is the default there.\nTo redo the end-to-end table on an idle machine: `eval/tune_ab.sh .build/release/drama 5 eval/out/perf/ab_tune_idle` then\n`eval/.venv/bin/python -I eval/perf_ab_summary.py eval/out/perf/ab_tune_idle`.\n\nCopyright 2026 Peter Marelas. Drama Studio's own code is under the Apache\nLicense 2.0; see [LICENSE](https://github.com/maralski/drama-studio/blob/main/LICENSE) and [NOTICE](https://github.com/maralski/drama-studio/blob/main/NOTICE).\n\nBuilt with Fish Audio. Drama Studio runs Fish Audio Drama 3, and the model\nweights and codec checkpoint are not included: the app downloads them from\n[fishaudio/drama-3](https://huggingface.co/fishaudio/drama-3), where Fish Audio\nlicenses them under the Fish Audio Community License, Version 1.0, copied here\nas [LICENSE-FISH-AUDIO](https://github.com/maralski/drama-studio/blob/main/LICENSE-FISH-AUDIO). That license governs the model and\nthe audio you make with it. It allows research and non-commercial use, and\ncommercial use while annual revenue is under USD 500,000; it forbids cloning a\nreal person's voice without their consent and the other uses its Section IV(b)\nlists, and it needs a separate Fish Audio license to offer speech generation to\nothers as a hosted service.\n\nSection IV(b) of that license asks anyone who makes speech generation available\nto others to bind them to its use restrictions, so the app's setup window shows\nthem and asks you to agree before it downloads or generates anything. Until you\nagree, the window, the HTTP API (403) and through it the MCP server and\n`drama-say` refuse to speak. The command-line `drama` tool is for developers\nbuilding from source and does not ask.\n\nThe fish-speech reference code copied under `eval/oracle/` and\n`Sources/DramaEngine/CodecConversion/dac_reference/`, the parts of DramaEngine\nreimplemented from it, the converted Core ML codec, and the example cast from\nFish Audio's Drama 3 Space stay under the Fish Audio Community License as well;\n[THIRD_PARTY_NOTICES](https://github.com/maralski/drama-studio/blob/main/THIRD_PARTY_NOTICES) lists each one and what changed.\ndescript-audio-codec code keeps its MIT notice there. This project is not\naffiliated with or endorsed by Fish Audio.\n\nDuring setup, you can choose to share anonymous usage information to help improve Drama Studio. Sharing is off unless you enable it. Only counts of app launches, generation outcomes and setup failures are included. Scripts, audio, voices, file names and personal identifiers are excluded. Change your choice at any time in Settings > General > Privacy.\n\nSee [usage monitoring configuration](https://github.com/maralski/drama-studio/blob/main/docs/usage-monitoring.md) for release setup\nand the data contract.", "url": "https://wpnews.pro/news/drama-studio-local-realistic-text-to-speech-synthesis-engine-for-apple-silicon", "canonical_source": "https://github.com/maralski/drama-studio", "published_at": "2026-11-03 07:23:37+00:00", "updated_at": "2026-11-03 07:51:30.024564+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "large-language-models", "ai-infrastructure"], "entities": ["Drama Studio", "Peter Marelas", "Fish Audio", "Drama 3", "Apple silicon", "Metal", "Core ML", "M3 Max"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/drama-studio-local-realistic-text-to-speech-synthesis-engine-for-apple-silicon", "markdown": "https://wpnews.pro/news/drama-studio-local-realistic-text-to-speech-synthesis-engine-for-apple-silicon.md", "text": "https://wpnews.pro/news/drama-studio-local-realistic-text-to-speech-synthesis-engine-for-apple-silicon.txt", "jsonld": "https://wpnews.pro/news/drama-studio-local-realistic-text-to-speech-synthesis-engine-for-apple-silicon.jsonld"}}