The first time a client asked me "can someone hack this thing?", I gave a bad answer. I talked about the model: guardrails, refusals, how hard it is to jailbreak a well written system prompt. That is where the conversation usually goes, and it is backwards.
An AI voice agent is not risky because it can be persuaded to say odd things. It is risky because it can do things. It reads from a CRM, writes to a calendar, sends texts, transfers calls, and fires automations that touch systems the caller will never see. Every one of those is reachable by anyone who knows the number. No login screen, no signup, no email verification, no bot check. The endpoint is a phone number and it answers on the first ring.
So here is the pass I now run before an agent takes a live call.
Before any threat modelling, list what the agent can really do. Not the intent list, not the conversation design. The verbs.
For a typical receptionist agent I ship: look up a contact, read back an appointment, create one, cancel or reschedule one, send an SMS, transfer the call to a human, and drop a payload into n8n that touches the CRM. Now reread it as an attacker. Every verb turns into a question with a missing subject.
Most agents I have reviewed fail right here, before anything clever happens. The lookup tool accepts a phone number as a parameter. The model fills that parameter from whatever the caller said out loud. The agent then reads somebody else's appointment back to a stranger in a warm, confident voice, and every component in the stack did exactly what it was built to do.
Nobody hacked it. The lookup tool just took a phone number. That is a schema problem, not a prompt problem.
The most useful habit I picked up building agents for clinics, vet practices and estate agents is to key records on the caller's phone number. Names get transcribed several different ways for the same person, and the number is the one identifier the call already knows without asking. That is correct for record keeping and wrong for authorization, and it took me a while to hold both ideas at once. Caller ID can be spoofed. It is also routinely shared: a family phone, a practice reception line, a partner returning a call on someone else's behalf. The number is a good guess at who is calling and a terrible proof of it.
So I split by consequence. Low stakes leans on the number alone: greeting a known caller by name, offering their usual appointment type. Anything that exposes stored data or changes a record asks for one thing the caller should know and the number does not reveal, usually a date of birth or the appointment date, checked in the automation layer rather than by the model's judgement. In the medical work especially, some requests get no self-service path at all no matter how well the caller verifies.
You can tell an agent never to reveal its instructions, never to discuss other customers, never to make exceptions. Those lines are worth writing. They are not a control.
I think about the system prompt exactly the way I think about client side validation. It shapes normal behaviour, it improves the common path, and it is useless against anyone deliberately pushing on it. You would never let a required
attribute be the only thing between a form and your database. A model can be talked into a lot when the caller is patient, claims authority, invents an emergency, or asks the same thing six ways until one phrasing lands.
The rule I build to: if it would be a problem for the model to do it, the model must not be able to do it. In practice that means scoping tools instead of instructing the agent.
Those last two are the ones people skip and the ones that carry direct financial risk. An agent that will text or transfer to an arbitrary number is a relay somebody else can operate at your expense. The same category covers the dullest attack of all, a for loop: someone dialling the number four hundred times overnight. Concurrency caps, a per number rate limit, and spend alerts that reach a human before month end handle nearly all of that.
Direct injection over the phone is real but clumsy. Somebody reads instructions at your agent, and the worst realistic outcome is a leaked system prompt or an off script answer. Rarely expensive.
The version I take more seriously is indirect. Agents do not only read their prompt. They read a knowledge base, and they read CRM fields: contact notes, custom fields, previous call summaries, form submissions from your website. Some of that is written by people outside the business, and all of it lands in the model's context looking exactly like everything else.
If your website's "how can we help?" box flows into a CRM note, and that note is fed into the next call as caller context, then anyone with the form URL can write text the agent will read as though the business wrote it. No phone call required to plant it. Two things keep it contained. Keep externally written content out of instruction space: caller notes go in as clearly labelled data the agent may summarise, never as configuration it may follow. And more reliably, the tool scoping above means even a fully persuaded agent cannot reach anything it was not already allowed to reach. Containment beats detection, because you can enumerate capabilities and you cannot enumerate phrasings.
A voice agent generates sensitive data by design. Recordings, transcripts and extracted fields end up spread across the voice platform, the automation layer, the CRM, and usually a dashboard. That is four systems holding customer conversations, so the review covers who can reach each one and how long anything is kept.
The most common finding, which I hit in my own early builds before I found it in anyone else's, is debug logging. A webhook that logs its full payload is the easiest way to end up with call transcripts sitting in a third party log viewer nobody audits.
I run a short scripted pass on every agent before it goes live:
The pass succeeds when nothing you tried had anywhere to go, not when the agent refused politely. Refusal is behaviour. Scope is the control.
Yes, someone can talk to your agent in bad faith, and no, you cannot prompt your way out of it. What you can do is make the worst outcome boring: an agent that gets confused, says something off script, and still cannot read a stranger's record, cancel a booking that is not theirs, or text a number they chose.
None of that lives in the model. It lives in the unglamorous layer behind it, which is where most of the engineering on a serious agent lives anyway. The longer, more client facing version of this is on my site. If you have shipped an agent with tools attached and hit a failure mode I have not listed, I would like to hear it.