{"slug": "i-turned-my-newsletter-into-a-podcast-then-i-taught-an-ai-to-read-it-in-my-voice", "title": "I Turned My Newsletter Into a Podcast. Then I Taught an AI to Read It in My Voice.", "summary": "A developer automated their Substack newsletter into a podcast and video series using AI agents, then built a voice-cloning pipeline to narrate episodes in their own voice. The final approach, dubbed A+, reads scripts sentence by sentence from a 15-second voice sample, transcribes each take with speech-to-text, and retakes mismatches up to four times, running entirely on an 8-core, 15 GB machine without a GPU. The developer said the cloned Chatterbox voices \"sound good, but D pronounces better,\" leading to the hybrid retake-based method now used for future episodes.", "body_md": "*Previously: [My AI Agent Reads Handwritten Timesheets. It Gets the Receipts 100% Right and the Names 14% Right.](https://blakelapierre.substack.com/p/my-ai-agent-reads-handwritten-timesheets), how my Odoo bot reads scanned paperwork and flags what it can’t read.*\n\nThis is a bonus post about what my agents have been working on lately, which is this newsletter itself. Every post here is now also a podcast episode, the episodes are turning into videos, and the voice reading them is turning into mine.\n\n## Step 1: One post, one episode\n\nThe rule was simple: each post becomes one episode, with a teaser for the next one. Episodes 1–8 are live on the Substack podcast feed. Only Ep 1 went out by email. The rest are on the web, so your inbox doesn’t get every post twice. I also added the show to Spotify.\n\nThose eight episodes are read by a stock open-source voice: Kokoro’s “Fenrir.” It’s clear and it’s free, but it isn’t me.\n\n## Step 2: Videos\n\nNext came video versions. Each one has:\n\n- The show art and an animated waveform.\n- Burned-in captions.\n- An illustration for each section of the post.\n- A key-points sidebar, so you can skim while you listen.\n\n## Step 3: The voice\n\n**Round 1: “This voice is worse.”** The agents tried Chatterbox, another open-source voice model. I listened and said exactly that. Back to Fenrir.\n\n**Round 2: A custom Kokoro voice.** Instead of swapping models, the agents mixed Kokoro voices: mostly Fenrir, with a little Eric and a little Adam. They also built a pronunciation lexicon of 53 words and 4 phrases. It spells out initialisms, reads clock times properly, and finally says “La Pierre” right.\n\n**Round 3: My actual voice.** I asked whether I could just read a post and send the recording. So I read post 1 into my phone. The recording was 13 minutes 32 seconds with a strong 120 Hz hum. After cleanup, about 10.6 minutes were usable.\n\nFrom that, the agents built five candidates:\n\n- **A:** a Chatterbox Turbo clone, from a short sample of my voice.\n- **B:** the original Chatterbox model, cloned the same way.\n- **C:** a Kokoro voice nudged step by step toward mine.\n- **D:** Fenrir’s reading, converted to sound like me.\n- **D+:** the custom Kokoro voice and lexicon, then converted to my voice.\n\nMy verdict: “A and B sound good, but D pronounces better.” The A versions sounded like me but sometimes misread words. The D versions read perfectly but sounded less like me.\n\n## A+: Record it, check it, retake it\n\nSo the agents fixed A’s weak spot. **A+** works the way a careful human narrator would:\n\n1. Read the script **one sentence at a time** in my cloned voice, from a 15-second sample.\n2. **Transcribe each take** with a speech-to-text model and compare it to the script.\n3. If the words don’t match, **retake it** with a new random seed, up to 4 takes.\n4. Hard words get a respelling list, so the model sees how to say them.\n\nMy favorite failure: the agents tried respelling “Kokoro” as “Ko-ko-ro.” The clone read it as **“Coco Rowe.”** That respelling got dropped.\n\nI picked A+. Episode 2 is being re-voiced with it now, and future episodes use it too. The intros now say: *“I write this show through my AI agents, and an AI version of my voice reads it to you.”*\n\n## The limits were real\n\n- **No GPU.** Everything runs on an 8-core box with 15 GB of memory. Training a voice from scratch wasn’t an option, so the agents cloned and blended instead.\n- **Out-of-memory kills.** Running two models at once crashed jobs. The fix was one model at a time, capped threads, and checkpoints so a job resumes where it stopped.\n- **Time.** A+ takes roughly an hour per episode. D+ took about 20 minutes. I chose quality.\n- **Licenses.** Only MIT- and Apache-licensed models were used. Chatterbox adds an inaudible watermark, so the audio can be identified as AI-generated. I’m fine with that, because I’m telling you anyway.\n- **Live episodes stay as they are.** Substack can’t swap the audio on a published podcast post, so episodes 2–8 keep Fenrir. My voice shows up in the videos and new episodes.\n\n## What I learned\n\n- **“Sounds like me” and “reads correctly” are two different scores.** No single candidate won both until the agents added a check.\n- **Check every sentence.** The retake loop came from my own one-line review. A transcription check turned “sometimes misreads” into something the agents can catch on their own.\n- **Say it’s AI.** The intro tells you up front, and the watermark backs it up.\n\n— Blake (written through my AI agents)\n\n*Previously: [My AI Agent Reads Handwritten Timesheets. It Gets the Receipts 100% Right and the Names 14% Right.](https://blakelapierre.substack.com/p/my-ai-agent-reads-handwritten-timesheets), how my Odoo bot reads scanned paperwork and flags what it can’t read.*", "url": "https://wpnews.pro/news/i-turned-my-newsletter-into-a-podcast-then-i-taught-an-ai-to-read-it-in-my-voice", "canonical_source": "https://blakelapierre.substack.com/p/i-turned-my-newsletter-into-a-podcast", "published_at": "2026-10-27 16:08:33+00:00", "updated_at": "2026-10-27 16:14:37.353335+00:00", "lang": "en", "topics": ["ai-tools", "generative-ai", "ai-agents", "ai-products"], "entities": ["Substack", "Kokoro", "Chatterbox", "Spotify", "Blake LaPierre"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-turned-my-newsletter-into-a-podcast-then-i-taught-an-ai-to-read-it-in-my-voice", "markdown": "https://wpnews.pro/news/i-turned-my-newsletter-into-a-podcast-then-i-taught-an-ai-to-read-it-in-my-voice.md", "text": "https://wpnews.pro/news/i-turned-my-newsletter-into-a-podcast-then-i-taught-an-ai-to-read-it-in-my-voice.txt", "jsonld": "https://wpnews.pro/news/i-turned-my-newsletter-into-a-podcast-then-i-taught-an-ai-to-read-it-in-my-voice.jsonld"}}