I'm a Honda service advisor in South Florida. My day is a service drive: customers dropping off cars, phones ringing, and the same fifteen questions over and over. How much is a brake job? Is this covered? Can I come in Saturday? What does the wrench light with the "A1" mean?
So I automated the repetitive half of my own job and kept the human half. It's an SMS agent — customers text the dealership's number over iMessage, and an LLM system answers from the actual owner's manual, checks their service contract, books appointments conversationally, and escalates to my Telegram when it needs a human. It's been live with real customers for months. Six languages, because my customers speak six languages.
Here's a real exchange, after hours, in Spanish:
Customer:Hola, quiero hacer una cita
Agent:Claro — ¿qué tipo de servicio necesitas para tu Honda Ridgeline? Recomiendo en la mañana para que tengas tu vehículo de vuelta el mismo día.
Customer:El sábado en la mañana
Agent:Lo siento, no estamos abiertos los sábados. ¿Te vendría bien el viernes? Abrimos a las 7 — ¿te funciona a las 8:00 AM?
The agent knew my schedule, refused the closed day, offered a concrete alternative, and never left the customer's language.
I built all of it by hitting problems and fixing them. Then I read Chip Huyen's AI Engineering, and had the strange experience of finding names for things I'd invented in the dark — plus one glaring gap I hadn't noticed at all.
This is the mapping, and the gap.
If I could hand one sentence to someone shipping their first agent, it's that one. It cost me three separate production bugs to write it down.
A customer with an expired service contract asked whether a repair was covered. The agent told them yes. It was reading the coverage data correctly — the expiration date was right there in the context — and it still got the comparison wrong.
I did what everyone does first: I fixed the prompt. Put today's date in the system message. Spelled out "compare the expiration date to today's date." Told it to be careful.
It failed 3 out of 3.
So I stopped asking. Now the date comparison happens in Python and the model only gets to relay the verdict:
def _annotate_expirations(in_md: str) -> str:
"""Stamp a computed past/active verdict next to every expiration date in the
portal data. LLMs are unreliable at date comparison, so the comparison is
done here and the model only relays the verdict."""
...
if d < today:
return line + " ⚠️ [COMPUTED: this date has already PASSED — no longer valid]"
return line + " [COMPUTED: date is in the future — still active]"
Every dated line in the coverage data arrives at the model pre-judged. There is no date math left for it to get wrong. That function now has seven unit tests, and they cost zero API calls to run.
Customer asks for three services and then, "so what's the total out the door?" The agent quoted three correct prices and then produced a total that was simply wrong — shop supplies and Florida sales tax applied to a subtotal it had added up by itself.
Same fix. The fee formula lives in code exactly once:
def compute_total(prices: list[float]) -> dict:
"""Sum quoted service prices into an out-the-door breakdown, rounded to cents.
Single source of truth for the fee formula so it never drifts between the
prompt and the code:
shop supplies = min(subtotal × 10%, $59.50)
tax = (subtotal + shop supplies) × 7%
"""
The model's job is to notice that the customer asked for a total and to say the resulting numbers like a human. Not to be a calculator.
The booking agent returns structured output — a Pydantic schema, enforced by the API — with a complete
flag. Early on, a booking saved with no service type in it. The model had marked itself complete, and I believed it.
Now the server independently re-checks that every required field is really filled before anything is written or emailed. The model's self-assessment is a suggestion. The check is the authority.
Three different bugs, one shape: any claim the model makes that code can verify, code should verify. Everything downstream of that rule got more reliable — and cheaper to test, because assertions on pure functions don't need an API key.
Reading the book was less "here's something new" and more "oh, that's what I've been doing":
[BOOKING_COMPLETE]
" is not a thing you want a customer to receive.)None of that made the book a waste of time. The opposite: I paid for each of those lessons in production incidents, with real customers on the other end. The book costs thirty dollars. If you're about to build this, buy it first — you'll still hit your own bugs, but you'll hit better ones.
Here's what reading it actually changed.
My quality signal, for months, was "a customer complained, or a scripted terminal conversation happened to catch it." That's not a signal. That's luck with a feedback delay.
So I built the deterministic layer first — no LLM judges, no fancy framework, one file:
python evals.py # full suite
python evals.py --runs 3 # more runs → better flakiness signal
python evals.py --only orchestrator
Over 130 cases now. Every single one is either a real production bug or a behavior I can assert exactly. The sections: date annotation, orchestrator classification, tech/pricing answers, the outreach flow, appointment confirmation, price gating, total math, routing, and maintenance-code decoding. Most of them are pure functions — they run in seconds and cost nothing.
Three things about it worth stealing.
1. The hallucinated-price detector. My favorite eval in the suite, and it's nine lines. Regex every dollar amount out of the agent's reply, and assert each one exists in the pricing file:
def _known_amounts() -> set:
with open(PRICING_PATH) as f:
return set(_PRICE_RE.findall(f.read()))
def _hallucinated(reply: str) -> list:
known = _known_amounts()
return [a for a in _PRICE_RE.findall(reply) if a not in known]
Deterministic hallucination detection for the one category of hallucination that would actually cost my dealership money. No judge model, no rubric, no ambiguity. If you have a domain where the ground truth is a file, you can write this today.
2. It found a dormant bug in its first hour. The orchestrator can classify a short follow-up ("Why?", "How come?") as follow_up
, which sticky-routes back to whichever agent answered last. Nice feature. Except follow_up
was missing from the intent registry, so the validator silently coerced every one of those verdicts to tech
— the entire sticky-routing path was dead code, and had been for weeks. Nothing crashed. No customer wrote in to say "your routing table has a missing enum member." It took writing down what I expected to happen for the gap to become visible.
3. Flakiness is a result, not a nuisance. Each LLM case runs N times and reports a pass rate. A case that passes 4/5 is telling me something a binary green check would hide. Cases where the model is unreliable but code catches it downstream are marked known_flaky
— they report their rate and don't fail the suite. That's honest: the model is flaky there, and the guard is the thing I'm actually relying on.
The next layer is an LLM judge over full transcripts with binary rubric questions — is the reply native-sounding in the customer's language? is it grounded in the provided manual context? was escalating the right call? — calibrated against transcripts I label myself, since I'm both bilingual and the domain expert. That's the nice thing about building for a job you actually work: I am the ground truth.
The code is on GitHub: Service-Advisor-AI-Agent. Architecture diagram, the eval suite, and a demo video of a real conversation are all in the README.
I'm a service advisor teaching myself to be an engineer in public, and I'm looking for applied-AI / conversational-AI work. If you build in this space — dealer tech especially — I'd like to hear from you.****