Everybody's seen the demo: you call a number, an AI picks up, and you can suggest to hand over you to a real person.
It answers a real phone number, talks back in whatever language you speak to it, looks up answers to your questions, and hands you off to a real person when you ask.
Picture a few characters and a middleman.
You ──▶ Phone system ──▶ Our middleman ──▶ The AI
📞 (FreeSWITCH) (our code) 🧠
▲ │
└──── "transfer" ◀───┘ and: 📚 look up an answer
Why the middleman? Couldn't the phone system just talk to the AI directly? Not really and the reasons are basically the whole reason this project exists:
That's the shape. Now the interesting failures.
he old way to make a phone bot was a three-step relay race: turn the caller's speech into text, send the text to an AI, turn the AI's text back into speech. Three handoffs, and the meaning got a little more bruised at each one.
The new voice models do all three in one. You hand them audio, they hand you audio. And two lovely things fell out of that for free:
This is the one that ate my time.
Phone audio isn't a file you send all at once. The phone line expects that rhythm and nothing else.
Our AI, though, doesn't talk in a steady drip. It thinks for a beat and then hands you a big gulp of audio all at once. two seconds of speech in a single burst. If you just shove that gulp at the phone line as fast as it arrives, the line chokes. It has a tiny bucket (about a tenth of a second) and the rest spills on the floor. The caller hears the AI stutter and drop words.
The fix is almost embarrassingly simple once you see it: Catch the gulp in a queue, then release one small slice every 20 milliseconds, matching the heartbeat the phone line wants.
// Catch the AI's audio, then feed the phone line ONE small frame every 20ms.
const queue = [];
setInterval(() => {
const frame = queue.shift();
if (frame) phoneLine.send(frame); // one heartbeat's worth of sound
}, 20);
That little 20
is the difference between "sounds like a person" and "sounds like a robot losing signal." The lesson that generalizes: when something plays in real time, sending it as fast as you can is a bug. You send it as fast as it's meant to be heard.
(There's a second gremlin in the same neighborhood: the AI speaks at one audio "resolution" and the phone line runs at a lower one, so you also have to gently down-convert every slice — the audio equivalent of resizing a photo without making it blocky. Same idea, less drama.)
Real conversations have interruptions. You start answering, the other person jumps in, and you stop. A bot that plows through its whole sentence while you're talking feels like a hold-menu, not a helper.
The good news: the AI tells us the instant it hears you cut in. The moment it does, we dump everything we were about to say and go quiet.
The humbling news: you can only un-say the words you haven't handed over yet. Anything already dripping through the phone line will finish playing — you can't reach into the wire and grab it back. So the trick isn't a clever cancel button; it's never getting too far ahead of yourself. We hand the phone line as little audio as possible at a time, so when you interrupt, there's barely anything left in flight to talk over you.
The takeaway I keep coming back to: how fast you can stop is decided by how much you've already committed. That's true of a lot more than phone calls.
Handing the caller to a real human sounds trivial. It's where we found the most sharp edges.
Say goodbye before you leave. The AI wants to say "Connecting you to billing, one moment" — but its decision to transfer arrives a beat before those words do. If you transfer the instant it decides, you cut the caller off mid-sentence. So we hold the transfer, let the sentence finish playing, and then move the call. Decide now, act after the goodbye.
Unhook yourself first. Our middleman is quietly listening to the call. If we transfer without unhooking, we'd follow the caller right into their private conversation with the human. Not okay. So: stop listening, then transfer.
If the handoff fails, stay. The worst possible outcome is dropping someone into dead air. So if a transfer can't complete for any reason, we keep the caller connected and let the AI apologize and offer something else — rather than dumping them and vanishing.
If one line of code captures the whole spirit, it's this:
transfer(...).then((result) => {
if (result.error) return; // handoff failed → DON'T hang up; keep them with us
endOurSide(); // only leave once they're safely with a human
});
Fail toward the human, never toward the dial tone. That's the rule.
When you ask "what time is breakfast?", the AI doesn't secretly know your business. Here's the honest sequence:
The AI decides when to look something up; we own how good the lookup is. The one trap worth naming: if the search is even slightly misconfigured, it doesn't throw an error — it quietly returns confident nonsense, and the AI will happily say it out loud. So "the answers are subtly wrong" is a config bug, not an AI bug. Check the plumbing before you blame the model.
The AI charges for every second it's awake on a call. A caller who goes silent and wanders off is a slow money leak. So two humble timers earn their keep every single day:
Not glamorous. Absolutely essential. Put them in on day one, not after the first surprising invoice.
Built on FreeSWITCH (a free phone server), a small Node/TypeScript middleman, a voice-to-voice AI model, and a searchable knowledge base. Every hard part lived in the middleman — and none of it was really about the AI.