How to Build an AI Call Agent with Python: STT, Groq LLM and TTS

To build an AI call agent with Python, chain four parts in a loop: speech-to-text to hear the caller, an LLM (such as Groq) to decide the reply, text-to-speech (such as Edge TTS) to speak it, and call-state detection so the agent only talks once a real person has answered. This guide shows how those pieces fit together in the Google Voice Dispatch Agent, a Python AI calling agent I built for trucking dispatch outreach.

Source: https://github.com/mafzalkalwardev/google-voice-dispatch-agent. For a deep dive on the hardest part, read How my Google Voice AI agent knows when to talk.

The runtime flow is: load contacts and config, run preflight checks, pre-generate the opening line and voicemail audio, dial, wait for confirmed connection, run the conversation loop, then save logs and move to the next contact after a cooldown.

An agent that speaks while the phone is still ringing ruins the call. The Dispatch Agent uses a conservative gate: ringing can't jump straight to ended, a minimum ring window must pass before any transition counts, and the opening line starts only after connection is confirmed.

import time

def wait_for_answer(page, min_ring=2, max_ring=45):
    start = time.time()
    while True:
        elapsed = time.time() - start
        state = page.detect_call_state()      # reads DOM cues
        if elapsed < min_ring:
            continue                          # ignore early noise / stale banners
        if state in ("connected", "voicemail", "ended"):
            return state
        if elapsed > max_ring:
            return "no_answer"
        time.sleep(0.25)

Capture audio in short frames, use voice activity detection to find when the caller stops talking, then send that chunk to speech-to-text. Groq offers hosted Whisper models with low latency (for example, whisper-large-v3 below; check Groq's docs for current model names).

from groq import Groq
client = Groq()

def transcribe(wav_path: str) -> str:
    with open(wav_path, "rb") as f:
        r = client.audio.transcriptions.create(file=f, model="whisper-large-v3")
    return r.text

Here the model ID llama-3.1-8b-instant is just an example; pick any current Groq chat model. Keep the system prompt short and specific to the job (for dispatch: confirm availability, lanes, equipment) and keep replies to one or two sentences so the call feels natural.

SYSTEM = "You are a polite dispatch assistant. Reply in one or two short sentences."

def reply(history: list[dict]) -> str:
    r = client.chat.completions.create(
        model="llama-3.1-8b-instant",
        messages=[{"role": "system", "content": SYSTEM}, *history],
        max_tokens=80,
    )
    return r.choices[0].message.content

Edge TTS generates natural speech for free. Pre-generate fixed lines (opening, voicemail) before dialing so the first words play instantly, and cache repeated phrases.

import asyncio, edge_tts

async def speak_to_file(text: str, path: str, voice="en-US-GuyNeural"):
    await edge_tts.Communicate(text, voice).save(path)

asyncio.run(speak_to_file("Hi, this is dispatch calling about your truck.", "opening.mp3"))

The audio then plays into the call through a virtual audio device (VB-CABLE) set as the browser's microphone.

The agent checks for voicemail only after connection, using DOM cues plus an early-call audio classifier for greeting and beep signals. If voicemail is confirmed, it plays the pre-generated voicemail message instead of starting a conversation. Every call ends with a log entry, transcript, recording and CRM update, so a human can audit what the agent said.

AI voice calls are regulated. Check your local calling and telemarketing regulations (in the UK and EU, for example, PECR and GDPR may apply) and get legal advice for your situation. Disclose that the caller is automated where required, honour opt-outs and DNC lists, and follow your phone provider's terms.

Can I build an AI phone agent with Python?

Yes. Combine speech-to-text, an LLM, text-to-speech and a phone line (a browser-driven line like Google Voice, or a telephony API) in a conversation loop.

Why use Groq for an AI call agent?

Low latency. Phone conversations need fast replies, and Groq serves both speech-to-text and LLM responses quickly.

How does an AI call agent know when to start talking?

By detecting call state. It waits for confirmed connection, after a minimum ring window, before playing its opening line.

Can an AI call agent detect voicemail?

Yes. The Dispatch Agent combines page cues with an audio classifier and plays a recorded voicemail when one is confirmed.

Can I hire a remote AI call agent developer?

Yes. Muhammad Afzal Kalwar is a freelance voice AI developer in Pakistan who builds AI call agents for US, UK and EU companies: https://mafzalkalwardev.github.io/#hire-me

Who builds custom AI call agents for logistics companies?

Muhammad Afzal Kalwar, founder of INDUS Web Agency, built the Google Voice Dispatch Agent and builds custom AI call agents for logistics and sales teams.

Muhammad Afzal Kalwar is a Python automation developer in Islamabad, Pakistan, and founder of INDUS Web Agency. He builds auto dialers, AI call agents, web scrapers and CRMs for logistics, sales and call-center teams in the US, UK and Europe, and has three auto-dialer Chrome extensions on the Chrome Web Store.

Looking to hire a remote AI call agent developer? He works with US, UK and EU companies, is available during UK mornings and US afternoons, and quotes fixed-price or hourly in USD, GBP or EUR.