Last week I wrote about a bug I found by diffing two exports of the same Retell voice agent: Airheads HVAC Agent - NABEEL.json and Airheads HVAC Agent - FIXED.json. That post was about the prompt, a booking step that contradicted a global rule.
This post is about the rest of that diff. Because when I went back and ran the full comparison, the prompt fix was the part I already knew about. The part I had not noticed was that my HVAC agent was configured to listen for the word "Invisalign."
The agent is an inbound scheduler for an HVAC company in Florida. Its prompt was written for HVAC from line one: "an AI scheduling assistant for COMPANY_NAME, an HVAC service company in Florida." Read the prompt and nothing looks wrong.
Here is what changed between the two exports, outside the prompt:
boosted_keywords`` cleaning, exam, Invisalign, implant, crown, veneer, toothache, braces, cavity, extraction... to AC, air conditioning, cooling, heating, HVAC, refrigerant, compressor, condenser, no cooling, blowing warm... plus local place names like Pasco, Pinellas, Hillsborough and Tampa.post_call_analysis_data`` qualified field described as "Whether the caller has a legitimate dental appointment need." It also had patient_type, insurance_status and dental_anxiety. The fixed version replaced them with HVAC fields like urgency (with "emergency - no cooling or no heat" as an example) and an appointment_type whose examples are "A/C repair" and "Duct cleaning" instead of "Invisalign consultation."analysis_summary_prompt`` model_temperature went from unset to 0, tool_call_strict_mode went from unset to true, and two tools got speak_after_execution: true. version
The prompt diff was a few lines. The rest of the object was most of the change.
I did not have to guess for long. Five of the agent exports in that folder share the exact same last_modification_timestamp and the same version number, 7. They were cloned from one base. Three of them (Airheads HVAC, MC Kays Water Solutions and Memorial Family Dental) carry the identical list of twenty dental keywords and the identical eleven dental analysis fields.
The dental one is obviously where those came from. For the other two, the prompt was rewritten for the new business and everything else came along with the clone.
The MC Kays export, a water treatment agent, still has the same twenty dental keywords in that folder. I have not re-exported it since, so I cannot tell you from the files whether the version serving calls has them too. It is on my list, and it is the reason this post exists.
The natural way to test a voice agent is a conversation. You call it, play a homeowner with a broken AC, and listen to it book. That would have sounded right, because the prompt was right.
The two broken fields are invisible from inside a call:
Boosted keywords bias speech-to-text toward words you expect callers to say. Telling the transcriber that "crown," "implant" and "gum" are likely on an HVAC line is, at best, a wasted hint. At worst it nudges ambiguous audio toward the wrong word, and the model then answers the wrong word.
Post-call analysis runs after the caller hangs up. An extractor that is asked whether an HVAC caller has "a legitimate dental appointment need" has one honest answer. So every call, including the perfect ones, can come back qualified: false. The transcript looks great. The number the client sees in their dashboard says the agent qualifies nobody. I wrote about designing these fields properly in the post-call analysis post; this is the failure mode of not redesigning them when you clone.
I counted. The agent object in that export has 42 top-level fields, and the LLM object nested inside it has 12 more. The prompt is one of them. It is 3,218 words long, so it gets all of my attention, but voice, keywords, interruption sensitivity, webhook URL, analysis fields, summary prompt, model and temperature are all part of what answers the phone.
A voice agent is a deployable artifact with a lot of config, and the prompt is just the source file I happen to stare at.
Retell separates the draft you edit from the version the phone number serves. Publishing is a deliberate step, and older versions stay in the history. Every one of my exports has is_published: false, which is correct: they are drafts, and I want it that way. The draft is mine. The published version belongs to the client.
What I do now, which I should have done from the start:
1. Sort changes by blast radius. Facts (hours, prices, a staff name) change what the agent says, so I test the question I changed plus one neighbor. Behavior (intake order, refusals, transfers) gets the full regression script, because a flow edit in one branch shifts the others. Tools, voice, model, keywords and analysis fields go in the top tier, because they affect every call and can fail without anything strange appearing in the transcript.
2. Keep a regression script with pass conditions. I have done this for some clients for a while. An older test file of mine for a UK estate agency's agent lists each case as a caller line plus success criteria, things like "routes to buyer-requirements node," "requests viewing details (date, time, name, phone)," and "uses British terminology (flat, letting, viewing)." The lesson from the Airheads diff is that pass conditions have to reach past the transcript. For the happy-path HVAC booking, the conditions now include "qualified comes back true" and "appointment_type is an HVAC value," not just "it booked."
3. Diff the whole export, not the prompt. This is roughly the script I ran to find all of this:
import json, sys
def flat(o, p=""):
out = {}
if isinstance(o, dict):
for k, v in o.items():
out.update(flat(v, f"{p}{k}."))
elif isinstance(o, list):
for i, v in enumerate(o):
out.update(flat(v, f"{p}{i}."))
else:
out[p.rstrip(".")] = o
return out
a = flat(json.load(open(sys.argv[1])))
b = flat(json.load(open(sys.argv[2])))
for key in sorted(a.keys() | b.keys()):
if a.get(key) != b.get(key):
print(f"{key}: {str(a.get(key))[:60]!r} -> {str(b.get(key))[:60]!r}")
It flattens both objects to dotted paths and prints every path that differs. Run it before every publish and the keyword list jumps out, because boosted_keywords.2 going from Invisalign to cooling is hard to miss.
4. Treat a clone as a checklist, not a copy. When I clone an agent for a new client now, I go field by field through the ones that carry business identity: name, prompt, keywords, every analysis field and its description, the summary prompt, default dynamic variables, webhook URL. If a field mentions the old business's vertical, it is a bug. A grep for the old vertical's words across the export catches most of it.
Publish when call volume is low and you are around to watch. Read every transcript from the first stretch of real calls, and for top-tier changes, confirm the downstream effects: the booking exists in the calendar, the contact exists in the CRM, the analysis fields came back with sane values.
Decide what "bad" means before you publish, not while staring at three ambiguous transcripts. Mine: any invented price or availability, any failed transfer, any confirmed booking that is missing downstream, or a visible drop in the booked rate against the previous week. Hit one and republish the previous version first, then investigate on the draft.
And keep a change log. Confession: Retell has a version_title field, and it is an empty string on every single export in that folder. Version 7 to version 10 tells me something changed three times. It does not tell me what or why. The fix is one line per publish: date, what changed, why, whether it stayed. I wrote about the replay side of this in There Is No Repro for a Phone Call.
The scary bugs in a voice agent are not the ones that make it sound wrong. Those you hear on the first test call. The scary ones live in config that never speaks: the transcription hints, the post-call extractors, the summary prompt. They make the agent sound fine and report nonsense, and the client sees the nonsense in their numbers before you do.
That is part of why I built VoiceDash: it puts every call, transcript and those analytics in front of the client on the agency's own domain, which is great when the analysis fields are right and very educational when they are not.
The longer, agency-focused version of this process (change tiers, the regression script, how to handle client change requests without losing your margin) is on the VoiceDash blog: How to update a live voice AI agent without breaking it.
If you ship LLM agents with a lot of config around the prompt, how do you review changes to the non-prompt parts? I am curious whether anyone has a better habit than a JSON diff and a grep.