How we centralised a fragmented support operation and handed the most repetitive part of triage to an LLM, while keeping every judgement call with a human.
The problem: support that leaked #
Every growing company outgrows the systems it started with, and support is usually where the cracks show first. Ours was split across two tools that never talked to each other. Customer tickets came into Zendesk. Business requests came in as tasks in Microsoft Planner - and anything a customer raised that needed engineering had to be copied from Zendesk into Planner by hand. A request would land and sit there until a support specialist happened to pick it up, with little visibility for the person who raised it. One ticket sat unassigned long enough to blow past every reasonable response time before anyone noticed. When the same issue can arrive through several different doors, "someone will pick it up" quietly becomes "no one does."
So we did two things. First, we centralised, pulling all of it into one system: Jira. Then we handed the repetitive, judgement-light part of the job to AI and, deliberately, kept the judgement-heavy part with people.
Step 1: one shape for everything #
Everything now runs in one system, Jira, collapsed into a single three-tier model:
L1, Customer Support. Everything from customers still arrives by email, but now lands in a single Jira queue instead of a separate help-desk tool. An inbound email becomes a ticket automatically; the customer gets one acknowledgement and stays in the same thread. L1 owns the customer relationship end to end.L2, Product Support. Internal and business users raise product issues through a single Jira Service Management portal. L2 reproduces, validates, and decides whether something is a genuine defect.L3, Engineering. A small number of engineering areas, each a Jira board with a clear owner. They fix; progress syncs back up automatically, from the engineering board to the Product Support ticket and, when it started with a customer, on to the customer-facing ticket, all inside Jira.
Centralising alone fixed the “which door did this come through?” problem. It did not fix the toil. Someone still had to read every incoming issue, decide which engineering team it belonged to, open a ticket on the right Jira board, copy the details across, attach the screenshots, and link the two so nothing got lost, on every ticket that came in. That is classification, not judgement, which is exactly what an LLM is for.
Step 2: AI as the router #
We did not ask AI to resolve tickets, to talk to customers, or to make product calls. We gave it one job: read a new product-support ticket and decide which engineering domain it belongs to.
When a ticket is created in L2, a Jira automation fires an AI step that reads its summary and description and returns exactly one label: one of the engineering domains, or UNSURE
.
Confident classification. The automation creates a linked ticket on the right board, assigns the domain lead, copies the attachments across, and links it back to the original. The reporter’s context travels with it. No human touched a keyboard.Not confident. Nothing is auto-routed. The ticket stays in the L2 queue for a person to triage, exactly as before.
The UNSURE
branch is the design. Everything else is plumbing. LLMs sound confident even when they are wrong, so we never asked the model for a confidence score. Instead we told it to return a domain only when the ticket clearly belongs to one, and to say UNSURE
otherwise. The failure mode we care about, a ticket auto-filed onto the wrong team’s board, is far more expensive than the one we accept, an ambiguous ticket waiting a few minutes for a human. So we tuned the prompt to be conservative and let people catch the tail.
Confident-but-wrong still happens, and the design assumes it will. Because every auto-routed ticket stays linked to the one it came from and lands on a real person’s board, a bad route is visible and one action away from being bounced back or re-escalated by hand. The AI never gets the last word; it just saves a human from typing when the answer is obvious.
What AI does, and what it deliberately doesn’t #
The line we drew:
AI does the mechanical routing described above. High-volume, low-judgement, and easy to check.AI does first-line self-service. A knowledge-grounded assistant sits on the Jira support portal and answers common “how do I…” questions from a curated internal handbook, so business users can help themselves before raising anything. It only answers from approved content, respects each person’s access so it cannot surface something the asker is not allowed to see, and, like the router, it says so plainly when it does not know.AI does That line is enforced in the prompt, not left as a hope, and those calls stay firmly with people. In a regulated industry, that boundary is not optional.notown the customer, resolve on its own, or auto-answer anything that looks like regulated advice.
What we learned #
Use AI as a router, not a decider. The value was not a clever autonomous agent; it was removing a hundred small classify-copy-paste chores a day. Narrow the job until it is checkable.Design the “I don’t know” path first. The confidence gate matters more than the classifier. Make the safe fallback the default and only act on clear cases.Keep a human owning the relationship. Automation gets work to the right place faster; it does not get to decide the customer is done.The knowledge base is the product. An AI assistant is only as good as what it is grounded in, and only as safe as its permissions.Measure the tail, not the average. The ticket that sits for three days is the one that hurts. Alerting on at-risk items, and routing them the moment they arrive, is where the real recovery came from.
Centralising gave us one front door. AI gave us a doorman for the post, as long as we stay honest about the letters he does not get to open.