# A Week as an AI Integration Consultant

> Source: <https://dev.to/lamingsrb/a-week-as-an-ai-integration-consultant-4p27>
> Published: 2026-08-13 07:32:45+00:00

Most weeks I don't write much new code. I read other people's systems, draw arrows on a whiteboard, and try to figure out which of the twelve places customer data lives is the one I should actually trust. That is the honest shape of AI integration work in a B2B company that has been shipping since 2014. The LLM is the easy part. The plumbing is the job.

Here are field notes from a recent week doing exactly this, plus the checklist I wish every founder had in hand before they hire anyone (me included) to bolt an LLM onto their stack.

The client wanted "an AI assistant that answers customer questions from our knowledge base and CRM." That is a sentence, not a specification. My first day is almost always the same: I map where data actually lives, who writes to it, and how stale it is by the time anyone reads it.

For this client the map ended up looking like this:

| System | Role | Write frequency | Trust level |
|---|---|---|---|
| Salesforce | Accounts, opportunities | Sales reps, daily | High but partial |
| NetSuite | Invoices, entitlements | Nightly batch | High, delayed |
| Zendesk | Tickets, macros | Agents, real-time | High for current, weak for history |
| Confluence | Internal KB | Product team, weekly | Medium, drifty |
| A 2019 MySQL app | Product config per customer | Nightly cron | Source of truth but ugly |

The MySQL app is the one nobody wanted to talk about. It is also the only place that knows which features a given customer is actually entitled to. If the assistant answers "yes, you have access to X" without reading that database, it will hallucinate entitlements and create a support fire.

**Rule I now apply on day one:** find the ugliest system in the stack. That is almost always the real source of truth. The pretty SaaS on top is a view, not a fact.

I don't touch a prompt until I have this map signed off. If the CTO can't tell me which system wins in a conflict between Salesforce and NetSuite on the same field, we are not ready to add an LLM. We are ready to have a meeting about data governance dressed up as an AI project.

By Tuesday I know the systems. Now I have to pick the glue. This is where most integrations quietly fail six months in, because someone reached for the wrong tool on day one and everything after that inherited the mistake.

My rough decision tree, after doing this on and off for a decade:

For this client I went with an EventBridge bus in front of Lambda workers, with Zendesk and Salesforce webhooks feeding in, and one small agent loop for a specific case (drafting a reply that needs to check entitlements, then pull the right KB passage, then decide whether to escalate). Everything else is a linear workflow, because linear workflows are debuggable at 2am and agent loops are not.

**The brittle trap I avoid:** a prompt sitting inside a Zapier or Make step, calling an LLM, and writing back to a CRM with no queue, no retries, no idempotency key, and no audit log. It demos beautifully. It falls over the first time the LLM returns malformed JSON, and nobody can tell you which record got corrupted.

Wednesday is always retrieval. The client had 4,200 Confluence pages, 18,000 closed Zendesk tickets, and a product manual as a 380-page PDF. "Just point the AI at it" is a six-month project disguised as a sentence.

What I actually built this week:

`tsvector`

column for full text search on the same rows.The number that matters here: on this client's evaluation set of 140 real historical questions, pure vector retrieval got 61% top-5 recall. Hybrid + rerank got 89%. That 28 point gap is the difference between "the assistant is useful" and "the assistant is a liability." It is also the reason I no longer take retrieval seriously if it is just a vector database and vibes.

**Cost-side note:** keeping retrieval in Postgres (instead of a dedicated vector DB) saved this client roughly $600 to $900 per month at their volume, and removed one vendor from the security review. For a mid-market B2B company that matters more than the theoretical benchmarks.

Thursday is the day I earn my rate. Anyone can wire an LLM to a CRM. Making it not embarrass the company is the actual skill.

Three things I insist on before anything goes to production:

**1. A structured output contract, enforced twice.** The prompt asks for a specific JSON schema. The response is validated against a Pydantic or Zod schema before it touches a downstream system. On failure, the worker retries with an error-corrective prompt, up to two times, then drops to a dead-letter queue with the full trace. No silent failures, no half-written records.

**2. An eval set that lives in the repo.** For this client, 140 questions with expected behaviors, run on every prompt change and every model change. Not just "does it answer correctly," but categorical: did it refuse when it should have refused, did it cite the right document, did it escalate the entitlement question. I run this in CI. A prompt change with a 3-point regression on the eval set does not ship.

**3. An audit log, always.** Every LLM call gets stored with input, output, model, cost, latency, retrieval context IDs, and the outcome downstream. This costs almost nothing in Postgres and it is the single most valuable artifact you will have when a customer emails asking why the assistant told them something wrong three weeks ago.

The kind of thing that goes into a worker looks roughly like this:

``` python
def handle_ticket_event(event):
    ticket = fetch_ticket(event["ticket_id"])
    context = retrieve(ticket.body, customer_id=ticket.customer_id)
    entitlements = get_entitlements(ticket.customer_id)  # from MySQL, not CRM

    draft = llm.generate(
        prompt=REPLY_PROMPT,
        context=context,
        entitlements=entitlements,
        schema=DraftReply,  # enforced
    )

    audit.log(event, context, draft)  # always, before any write

    if draft.confidence < 0.7 or draft.needs_human:
        assign_to_agent(ticket.id, draft=draft.text)
    else:
        post_internal_note(ticket.id, draft.text)  # never auto-send week 1
```

Note the last comment. Week one, the assistant never sends anything to a customer. It writes internal notes for agents to review. Week four, once the eval set and the audit log show it is behaving, we flip the switch on the low-risk categories. This is how you ship an LLM into a real business without a Slack channel full of angry executives on day two.

Friday is documentation and handover. This is where independent consultants lose clients (by writing nothing) and where I keep them (by writing too much, honestly).

What I hand over on every engagement:

Clients who run the 30-60-90 reviews get compounding value. Clients who don't tend to let the whole thing quietly rot within a year. I now write this into the contract.

If you are a founder or CTO thinking about hiring an AI integration consultant, run through this before you write the brief. It will save you and your consultant a painful week.

Start with the smallest workflow that touches real money or real customer time. One ticket type. One report. One approval step. Wire it end to end with a queue, a schema, an eval set, and an audit log. Ship it as an internal draft first, promote it to customer-facing only when the numbers say so. Then, and only then, look at the next workflow.

The teams that get AI integration wrong try to boil the ocean with a chat interface. The teams that get it right pick one boring workflow, instrument it obsessively, and let the ROI compound. Everything I have shipped that survived past a year followed the second pattern.

If you are wrestling with an integration like this and want a second pair of eyes, or you have already tried the prompt-in-a-Zap route and want to do it properly, get in touch at [lazar-milicevic.com/#contact](https://lazar-milicevic.com/#contact). More field notes from production systems live on the [blog](https://lazar-milicevic.com/blog).