While supporting our clients who are using our Voice platform, we often find ourselves falling into repetitive work.
For example, let’s observe the example support ticket with the following summary:
Our call center agent claims that our customers never hear the first few seconds when the agent greets them, what gives?
Of course, with a description like this, you first have to play an initial round of ping-pong with the client to gather enough information to know where to start, for example asking for any kind of identifiers, such as call IDs, so we can start looking up what went wrong. When we get the identifier, we start looking into the issue.
Seniors know which facts matter, and that knowledge takes years to build #
On a happy day, we would take a look at the call records, and assuming that the issue is caused by some expected behavior, we usually leave enough breadcrumbs here that tell us what happened. This gives us everything we need to provide an answer to our client, and continue with our day.
However, what if the answer isn’t found here? That’s usually when the fun begins and seasoned support engineers make the iconic “lean forward”.
Depending on the type of issue, you might think you should first check a few Grafana dashboards and check for anomalities, spikes, weird patterns, and hopefully they will vaguely match the timestamps in the client complaint and have an a-ha moment. Perhaps the observed packet loss and low bandwidth budget has something to do with the reported missing audio? But considering that this is visible only in one of the five provided examples by the client, you quickly dismiss it and think about what’s next.
Depending on how desperate you are, at this point you might consider looking at application logs and try to walk through the process, assuming you know it yourself, and look for the missing piece that will help you understand what happened. At this point, you are already hours into troubleshooting and have talked to several engineers to help you navigate through the forest known as communication platform with many components that are split across many microservices that somehow supposedly work together.
Fast forward across the 10 tabs you have HEPIC, Grafana, backend logs, API docs and various configuration pages. You managed to understand the clients use case, how they consume our APIs to orchestrate their call center and you might as well apply for a job at their startup, but you just can’t find anything that would cause their pain.
At this point the client is losing patience and thus the senior customer support engineer gets involved. Takes a look at the ticket, picks up your mouse and checks a few timestamps.
They immediately recognize that the issue is caused due to the client call orchestrator connecting both parties in the call via a conference only once both call legs are established, causing unpredictable audio clipping on both sides. Barely ten minutes of troubleshooting and the definitive answer was found.
If this was your first month on the job, you are probably exhausted at this point and wonder when you will catch up with all these things. It’s not true to say that these things aren’t documented, as the API behavior is very well documented in our awesome Voice API docs, but connecting all these dots and knowing the implications simply requires more time…
Okay, so how can we accelerate this?
Just use AI? #
With AI being all the craze, it was only natural to consider integrating it somehow into this process to accelerate our troubleshooting. And more than anything, reducing the difficulty of entry for new engineers, since just knowing where to look is a skill that has to be developed with time and experience.
At the time, AI powered chat bots were really popular, especially for providing support, but citing documentation to support engineers isn’t really helping, it’s actually annoying. No one wants a colleague (AI or not) who just drops links to various docs whenever you ask them to help with something, so this approach was a no-go for us.
Take the previous example, our support engineer wouldn’t even know what exactly to ask the AI, as the issue reported is often just a by-product of something else entirely. We need something that will work with us on figuring out what the issue is, instead of just being an interactive encyclopedia.
Thus, the criteria for building an AI tool that is actually useful for us is simple: if it’s not doing the same work as me, for me, I don’t want to use it.
We need something that will receive the same input as our supporters, pull up its artificial sleeves and get to work.
What we built #
Introducing Lynx – an internal support tool that tries to be something rather more than a simple prompt-in, answer-out flow with an LLM behind the curtain. At its’ core, it’s a tool that goes through a structured investigation loop, leveraging LLM models that make extensive use of tools, both MCP and locally defined ones.
Small tangent: what are tools? #
While this article aims to cover the overall implementation of Lynx, we will also provide a “10 sentence or less” crash course for certain topics.
Tools are essentially just commands that the AI is allowed to invoke. For example, you might want to allow your LLM to do a web search, lookup a phone number, or simply do some algebra.
You would do this by defining tools. MCPs are one very popular example of such tools, but when developing an “LLM wrapper” on the backend, you can expose certain parts of your application directly to the AI by defining local tools.
Now, the aim of Lynx is to allow our support engineer to initiate an analysis using human language, providing any info they gathered thus far, or simply forwarding whatever they received themselves. Lynx enters the investigation loop, starting with a system prompt that we curate, giving it an overall goal and instructions how to behave. It then usually starts using the tools at its’ disposal to gather as much data as possible.
We provided it tools that connect it to everything any other supporter has access to: HEPIC, Prometheus, application logs, reports, etc.
The crucial point to remember here is the fact that the AI doesn’t “know” voice infrastructure. It knows how to look things up, correlate them, and attempt to reason about what it finds. And digesting huge amount of textual data is something where LLM really shine.
However, since just giving an LLM a bunch of tools won’t really help it know what to look for, it was important to develop a list of articles how to approach certain problems, what to look, what to use, and how certain things are supposed to function. For this, we came up with a knowledge base for the AI to search for.
Also, around this time, RAG systems became hugely popular for putting between your AI assistant and your knowledge base, so we obviously had to incorporate that as well.
Small tangent: what is RAG? #
Retrieval-Augmented Generation (RAG) is an AI framework that allows LLMs to pull relevant text from an knowledge base.
There are plenty of amazing articles about this topic, but essentially, it allows the AI to search for things inside your knowledge base, and getting results based on some calculated similarity/relevancy score.
A simplified overview of how this looks like can be something like this:
Looks fine and dandy, right? We just need a few volunteers to write the knowledge base, how wants to go first?
Matter of fact is that not many people want to write documentation, especially not for a handbook to be used by an experimental AI companion.
So…. Just use AI? #
Feels like Deja Vu, but yes, we tackled this problem with AI as well. We realized that years of real investigation already lives in our internal communication channels, e.g. our slack channels dedicated to connecting our supporters with our developers.
We did the following experiment:
- Dump every historical support conversation, initial support query on one side, human analysis on the other
- Have Lynx investigate each reported issue independently, using its tools
- Ask Lynx to compare its’ analysis against what the human engineer actually found
- Ask Lynx to create or edit existing knowledge base articles based on what it learned
The result? Thousands of articles, based on actual investigations and issues encountered by clients. We suddenly had a manual how to handle all issues encountered in the past few years!
We captured exactly the kind of knowledge that we were looking for. The edge cases, the non-obvious correlations, the slack threads that spun for days investigating some obscure issue.
For example, if someone uninformed were to ask Lynx about something we refer to as ASTW, they would have gotten the following answer:
Since the AI model used (Claude Sonnet 4.5, in this specific instance) has only general knowledge and knows nothing about our procedures and abbreviations, all it can do is guesswork, and sadly often with misplaced confidence.
However, now after we constructed our knowledge base, it’s a completely different story:
As you can see, we now have an initial version of our knowledge base built from real queries, and their answers are used by the LLM for future questions and investigations.
Earlier we mentioned what tools are and that Lynx has a bunch of them, but how did we decide what to plug into Lynx was intentionally left vague.
While creating the initial knowledge base, aside from giving Lynx the tools to write articles, we also defined a nifty tool called “request_tool_integration”.
@Tool(name = "request_tool_integration", description = """
Requests integration of a new tool or data source. USE THIS TOOL when you identify a need for accessing data that is not currently available. Creates a request ticket for the development team. """)
public String requestToolIntegration(
@ToolParam(description = "Name for the proposed tool") String toolName,
@ToolParam(description = "Description of what the tool should do") String description,
@ToolParam(description = "Justification for why this tool is needed") String justification
) {...}
This tool would be used by Lynx to “file a formal complaint” whenever it would notice our human supporter was able to check something Lynx wasn’t able to see for itself. For example, consider the following support thread on our slack channel:
The prompt was simple, and the human supporter gave a straightforward answer, but when Lynx tried to analyse it itself, its conclusion was wrong.
The initial analysis provided by Lynx was vague and mostly guesswork. However, since this was the learning phase, it actually invoked the tool mentioned above with the following description:
Integration request – Resource: Account Configuration Service
Identifiers: account_id: NUMBER, recording_configuration, sftp_server_configured
Reason: Cannot determine if an account has SFTP recording delivery configured, which is essential for diagnosing why API recording downloads fail with HTTP 500 errors.
The support team identified this configuration as the root cause, but I have no way to query it independently.,
Value: Would enable independent diagnosis of recording access method failures. When a customer reports HTTP 500 on recording downloads,
I could query their account configuration and immediately determine if it's a SFTP-based account (expected behavior) vs CDN-based account (potential bug).
This would reduce support turnaround time and improve first-contact resolution rate. Currently, only account administrators can access this configuration.
What happened here is Lynx told us exactly what it needed access to in order to do its’ job, so that’s actually how we formulated our initial list of tools that need to be integrated into Lynx. Ideally, we wanted to plug Lynx into everything via MCPs so it can access anything a normal supporter would want. This didn’t turn out to be as straightforward as initially anticipated. Instead, it was big enough of a learning experience that I’ll write a longer article about it soon.
Back to the original objective, there is a support ticket waiting #
Now, for any given user complaint, Lynx is able to query for previous troubleshooting experiences and quickly try to apply that knowledge on the current investigation. Also, it is able to access all relevant information across all our components to quickly digest all records and hopefully find a correlation that will result in an accurate analysis.
Once done, it provides a compact report, split into distinct sections, ranging from a quick summary about what it found, to a evidence section containing traces to records from which the conclusions were derived from, allowing the supporter to verify any claim and actually learn from the analysis itself.
The result? The difficulty of Voice related troubleshooting significantly dropped, domain specific knowledge is no longer a barrier, and diagnostic data is no longer overwhelming.
Pictured: An example investigation by Lynx of a failed call. The issue was due to a misconfiguration while setting up the client account. Immediately identified by Lynx with provided steps how to remedy it.
Aside from having a tool for support engineers, what we ended up with is actually also useful for anyone needing a quick answer regarding our voice platform. For example, one day someone from Infobip was wondering what a specific client was up to, and simply asked Lynx the following:
Although the account traffic was already flagged by our other systems internally, it was still a non-technical-user friendly way to request some technical investigation and get an answer rapidly, without having to bother an actual engineer.
What the future brings for Lynx #
All in all, Lynx is quickly becoming an immense time saver all around, both for technical employees for who the tool is originally aimed at, but also for non technical employees who would like to get a quick answer from a voice engineer. AI tools often struggle going from “that’s cool” to “that’s actually useful”, and we hope that Lynx will serve as one real world example where we managed to make the cut.
The closed learning loop of automated learning of troubleshooting cases where human troubleshooters had to “jump in” ensures the knowledge base keeps growing and improving.
Regarding next steps, there is still lots to do in terms of improving Lynx, especially regarding improvements in the knowledge base utilization, and overall testing and quality of response improvements.
Aside from simply talking about giving it access to more integrations, there is also a lot of discussion to be had how and when an analysis is triggered.
Although currently while in testing it’s always triggered by an human prompt, we aim to broaden the scope, for example by integrating it in our monitoring pipeline, so an analysis can be executed as soon as our alerts trigger. No matter the issue, having an Lynx analysis already provided and ready for review is like having an always on duty support engineer, always at your side, and hopefully reduces the amount of times our on-duty supporters have to be woken up.