I Turned My Newsletter Into a Podcast. Then I Taught an AI to Read It in My Voice.

A developer automated their Substack newsletter into a podcast and video series using AI agents, then built a voice-cloning pipeline to narrate episodes in their own voice. The final approach, dubbed A+, reads scripts sentence by sentence from a 15-second voice sample, transcribes each take with speech-to-text, and retakes mismatches up to four times, running entirely on an 8-core, 15 GB machine without a GPU. The developer said the cloned Chatterbox voices "sound good, but D pronounces better," leading to the hybrid retake-based method now used for future episodes.

Previously: My AI Agent Reads Handwritten Timesheets. It Gets the Receipts 100% Right and the Names 14% Right. https://blakelapierre.substack.com/p/my-ai-agent-reads-handwritten-timesheets , how my Odoo bot reads scanned paperwork and flags what it can’t read. This is a bonus post about what my agents have been working on lately, which is this newsletter itself. Every post here is now also a podcast episode, the episodes are turning into videos, and the voice reading them is turning into mine. Step 1: One post, one episode The rule was simple: each post becomes one episode, with a teaser for the next one. Episodes 1–8 are live on the Substack podcast feed. Only Ep 1 went out by email. The rest are on the web, so your inbox doesn’t get every post twice. I also added the show to Spotify. Those eight episodes are read by a stock open-source voice: Kokoro’s “Fenrir.” It’s clear and it’s free, but it isn’t me. Step 2: Videos Next came video versions. Each one has: - The show art and an animated waveform. - Burned-in captions. - An illustration for each section of the post. - A key-points sidebar, so you can skim while you listen. Step 3: The voice Round 1: “This voice is worse.” The agents tried Chatterbox, another open-source voice model. I listened and said exactly that. Back to Fenrir. Round 2: A custom Kokoro voice. Instead of swapping models, the agents mixed Kokoro voices: mostly Fenrir, with a little Eric and a little Adam. They also built a pronunciation lexicon of 53 words and 4 phrases. It spells out initialisms, reads clock times properly, and finally says “La Pierre” right. Round 3: My actual voice. I asked whether I could just read a post and send the recording. So I read post 1 into my phone. The recording was 13 minutes 32 seconds with a strong 120 Hz hum. After cleanup, about 10.6 minutes were usable. From that, the agents built five candidates: - A: a Chatterbox Turbo clone, from a short sample of my voice. - B: the original Chatterbox model, cloned the same way. - C: a Kokoro voice nudged step by step toward mine. - D: Fenrir’s reading, converted to sound like me. - D+: the custom Kokoro voice and lexicon, then converted to my voice. My verdict: “A and B sound good, but D pronounces better.” The A versions sounded like me but sometimes misread words. The D versions read perfectly but sounded less like me. A+: Record it, check it, retake it So the agents fixed A’s weak spot. A+ works the way a careful human narrator would: 1. Read the script one sentence at a time in my cloned voice, from a 15-second sample. 2. Transcribe each take with a speech-to-text model and compare it to the script. 3. If the words don’t match, retake it with a new random seed, up to 4 takes. 4. Hard words get a respelling list, so the model sees how to say them. My favorite failure: the agents tried respelling “Kokoro” as “Ko-ko-ro.” The clone read it as “Coco Rowe.” That respelling got dropped. I picked A+. Episode 2 is being re-voiced with it now, and future episodes use it too. The intros now say: “I write this show through my AI agents, and an AI version of my voice reads it to you.” The limits were real - No GPU. Everything runs on an 8-core box with 15 GB of memory. Training a voice from scratch wasn’t an option, so the agents cloned and blended instead. - Out-of-memory kills. Running two models at once crashed jobs. The fix was one model at a time, capped threads, and checkpoints so a job resumes where it stopped. - Time. A+ takes roughly an hour per episode. D+ took about 20 minutes. I chose quality. - Licenses. Only MIT- and Apache-licensed models were used. Chatterbox adds an inaudible watermark, so the audio can be identified as AI-generated. I’m fine with that, because I’m telling you anyway. - Live episodes stay as they are. Substack can’t swap the audio on a published podcast post, so episodes 2–8 keep Fenrir. My voice shows up in the videos and new episodes. What I learned - “Sounds like me” and “reads correctly” are two different scores. No single candidate won both until the agents added a check. - Check every sentence. The retake loop came from my own one-line review. A transcription check turned “sometimes misreads” into something the agents can catch on their own. - Say it’s AI. The intro tells you up front, and the watermark backs it up. — Blake written through my AI agents Previously: My AI Agent Reads Handwritten Timesheets. It Gets the Receipts 100% Right and the Names 14% Right. https://blakelapierre.substack.com/p/my-ai-agent-reads-handwritten-timesheets , how my Odoo bot reads scanned paperwork and flags what it can’t read.