When you build agents for clients, support tends to follow you into every messaging app. A client sends a screenshot in WhatsApp. Someone else reports a failed task in Telegram, while a Slack thread contains the only explanation of a problem you thought you had fixed last week.
Before you can investigate, you have to find the conversation, work out which agent and version produced it, and check whether it has happened before. As you take on more customers, keeping track of their feedback becomes a job of its own.
And that only covers the failures someone reports. What about the agent that quietly gives the wrong delivery time, or claims it completed a task after a tool failed? You want to catch those before the client has to send you another screenshot.
Latitude brings both kinds of evidence into the same place. Flaggers inspect conversations and leave annotations when they find problems. Client reviews can come in as scores or custom signals. Signals group related findings so you can investigate a recurring failure with its examples attached. Through Latitude’s MCP, your coding agent can query that evidence across the fleet, produce reports, and compare releases.
I built a Hermes fleet to show how this works for a consultancy managing agents for several clients. The example has five client deployments and two development projects, with 845 conversations across a six-week window. I’ll use it throughout this guide.
The monitoring flow is designed to stay familiar as you go from five agents to fifty or more. Add a deployment, configure its checks, and bring it into the same review and reporting process. More customers will still mean more responsibility. The aim is to avoid making every new conversation another item on someone’s reading list.
Set up each agent so you can find its problems later #
For this fleet, I used one Latitude project per agent. Traveler Logistics has three: an operations copilot in Slack, an invoice chaser in Microsoft Teams, and a support triage agent in WhatsApp. Nimbus Retail and Harbor & Sage Accounting each have one. Two more projects hold development traffic. That arrangement keeps a deployment’s conversations, costs, and signals together. When a client reports a problem with the invoice chaser, I know where to look. When I want a fleet report, I can query across projects. This is an organizational convention for the example; choose organization access separately according to who should be allowed to see each client’s data.
The Hermes telemetry plugin sends conversations and tool activity to the selected project. Give each deployment a clear name and record its release version from the start. Those details save you from reconstructing which agent did what when a complaint arrives two weeks later.
Connect your coding agent to Latitude’s MCP as well. This gives you a way to ask questions about the whole fleet without opening each project in turn. You’ll still use the console to review conversations and settings, but routine investigation can start with a request in plain language.
Have flaggers make the first pass #
The first job is to find problems without waiting for a client complaint. Latitude’s flaggers inspect completed sessions and annotate the trace where a failure appears. Those annotations feed signal discovery, so a finding can become part of a recurring issue rather than sitting unnoticed in a transcript.
There are two kinds of checks to configure:
- Deterministic checks catch failures such as empty responses, tool errors, malformed JSON, and low cache usage. Enabled checks run on every eligible session without an LLM charge.
- LLM checks look for behavior that takes more interpretation: frustration, forgotten context, incomplete work, unnecessary refusals, or an agent pretending a failed tool call succeeded. These incur AI usage and use sampling, with a 10% default.
Latitude also uses screening evidence to trigger relevant LLM checks outside the random sample. Treat sampling as a way to tune review volume and coverage, not as a hard spending cap. As traffic grows, review the usage alongside the findings.
Use the presets as a starting point for each agent’s job, then adjust the selection. A support triage agent and a structured extraction agent should not necessarily have identical checks. You can enable flaggers through the API and connected tools; sampling controls live in the project settings. This is work you do when onboarding or changing a deployment. The checks then run as conversations arrive. Taking on another client doesn’t require assigning someone to read a sample of that client’s chats every morning.
Give client feedback a place beside automatic findings #
Clients will keep using WhatsApp, Telegram, and Slack. The monitoring process needs to accommodate that habit.
When a client reports a failure, bring the feedback into Latitude as a score or a custom signal. You can submit a review as a custom score against the relevant conversation, with the reviewer’s note attached, or capture the reported issue directly as a custom signal. Either route requires a way to transfer the feedback; Latitude does not automatically read every client messaging channel.
Failing scores can feed signals alongside other evidence; custom signals let you record the issue without first turning the review into a score. Keep the client’s description and relevant conversation references with the finding so your team can investigate it, even if the original report came through a different channel.
The example fleet includes a review about a reminder sent to a customer who had already paid. The reviewer wrote: “reminder sent to a customer who paid. again. this is the one thing that makes us look bad”.
That note gives the failure context a generic error rate cannot. Keep it with the evidence. Also keep the source clear: custom describes how a score entered Latitude, not whether a human or a model produced it. An imported client review is still human feedback; an automated score is not.
Investigate patterns, with the conversations attached #
Signals group related findings into named patterns. They show occurrences, affected sessions and users, trends, and the underlying evidence. A recurring tracking failure becomes one investigation with several examples to inspect.
In the support triage project, the findings included tracking lookups that lost leading zeros, delivery windows quoted in UTC but labeled as local time, and escalations that never received a human follow-up. Those require different owners and different fixes. The signal list makes them visible together.
Read the conversations before accepting a signal title as the diagnosis. One signal in this example said customers were not told what evidence to provide for damage claims. The conversations showed a more specific problem: the agent repeated the same evidence instructions regardless of the claim type. It even asked a customer to photograph damage when the pallet was missing.
Adding another instruction to request photos would have made that worse. The fix needed to distinguish a missing shipment from a damaged one.
Signals help you choose where to spend attention; the traces help you decide what to change. A useful request to your coding agent is:
Review the open signals across this fleet. Prioritize problems by severity and affected users. For each, show the project, representative conversations, and what evidence I should review before deciding on a fix.
Choose a person to own each investigation. Grouping repeated findings saves review work, but someone still needs to decide whether the answer is a tool fix, a prompt change, or a conversation with the client about their process.
Let urgent findings reach the team #
Connect Slack at the Latitude organization level and configure notification routes. The integration can send signal and monitor notifications to channels your team already watches. Routes support severity thresholds, so you can reserve immediate attention for high and urgent findings.
In the Hermes example, Latitude sent a notification for the damage claim signal with its description, project, and a link to investigate.
This is how automatic detection becomes useful in daily work. You don’t have to remember to visit every project’s signal page to discover that something needs attention.
Keep the notification setup deliberate. Routes are configured by notification group at the organization level, and connecting Slack alone is not enough: groups left on “Don’t send” stay silent. A Slack workspace can connect to only one Latitude organization.
As the fleet grows, tune what deserves an interruption. Sending every finding into a busy channel can recreate the same scattered-feedback problem you started with. Use alerts for triage and a regular fleet review for the rest.
Ask for a fleet report, then a client report #
The consultancy needs to know where attention is required across its customers. A client needs to know what their agents handled and what is being fixed. Both reports can use the same data.
Through MCP, a coding agent can query Latitude and turn the results into an HTML report. For example:
Build a fleet report for July 20 through September 4, 2026. Include every project, conversations handled, recorded AI cost, tool error rates, and open signals. Separate development projects from client deployments and link findings to their evidence.
The report for this example covered 845 conversations, with a 4.66% tool-call error rate and 15 open signals. Of those conversations, 831 belonged to the five client deployments and 14 to development projects.
Give clients an update they can use
Your client may only hear about an agent when someone complains. They forward the problem, you investigate, and most of the work that follows stays out of sight. By the next check-in, they are still asking whether that issue was fixed and what they are paying you to look after.
A regular service report gives them a fuller picture. It shows how much their agents are being used and explains what has improved, with the remaining problems and next steps in plain language. The person responsible for the agents can share it internally without having to explain traces or walk colleagues through your monitoring dashboard. Conversation counts alone don’t prove a return on investment, but they give the client a starting point for judging usage alongside reliability and the work their staff still need to do.
I used the same Latitude data to produce a report for Traveler Logistics, whose three agents handled 763 conversations. The report explains the tracking fix and its measured results, then separates work for the consultancy from action needed on the client’s side. For example, the support agent was creating escalation tickets, but nobody was picking them up. The client needs to know who should fix that handoff, not just that an escalation metric changed.
You can ask your coding agent to prepare this through MCP:
Draft a service report for Traveler Logistics using only its projects and the agreed reporting period. Explain agent usage, verified improvements, and unresolved problems in language the client can share with colleagues. Separate our next steps from decisions or action needed from the client. Remove internal console links and flag any claim or delivery date I need to confirm before sending.
The opening summary of the Traveler Logistics service report. Later sections explain fixes, outstanding issues, and who needs to act. Costs here cover model usage, not the consultancy’s service fee.
Always specify the date window and use aggregate queries where available. Have the report distinguish recorded model usage from the full cost of delivering the service. The small AI costs in this example are not a budget for running a consultancy; they exclude your team’s work and other operating costs.
For the consultancy, this makes regular client updates practical as the fleet grows. Reuse the reporting request with a different client and period, then review the evidence and commitments before sending. You still owe each customer a considered update, but you don’t have to assemble every report from dashboards and old chat threads. The client gets a record of progress without having to chase you for one.
Check what happened after a fix #
Once you ship a fix, use an experiment to compare release-tagged traffic. This makes “is it better?” a repeatable investigation with a defined window and comparable metrics.
The support agent had a tracking problem caused by converting IDs to integers and losing leading zeros. I rewrote the tool description and compared v1.7.0 with v1.6.0:
| Metric | v1.6.0 | v1.7.0 |
|---|---|---|
| Conversations | 223 | 138 |
| Tool-call error rate | 11.23% | 2.63% |
| Conversation error rate | 9.42% | 2.17% |
| Handoffs to a human | 15.25% | 7.25% |
The results support the fix, but this was a comparison of live traffic, not a controlled A/B test. The releases handled different numbers of conversations and could have faced different requests. Compare rates and inspect examples of the specific failure you intended to address.
Here, the targeted signal fell from five occurrences to one. Meanwhile, the share of conversations affected by signals rose from 5.83% to 10.87%, driven by timezone and damage claim problems. The tracking result looked better; the other problems still needed work.
An MCP request can make that distinction explicit:
Compare v1.6.0 and v1.7.0 on the support triage project over the experiment’s date range. Check the tracking failure first, then error rates and handoffs. Explain differences in the populations and any other signals that worsened.
Use the result to decide whether to close the investigation and what to tell the client. An attractive headline on its own isn’t enough.
Turn important failures into ongoing evaluations #
Flaggers cover common failure categories. When you find a problem specific to one deployment, you can turn its signal into an evaluation that keeps checking new traffic. For the invoice chaser, that means looking for reminders sent after payment.
Latitude requires a human annotation on at least one of the signal’s traces before it generates the evaluation. Annotating a clear failure gives the evaluator an example of what you consider wrong. The evaluation then runs on eligible traffic; this example uses 10% sampling.
The evaluation generated from a confirmed client failure, sampling 10% of eligible traffic.
Review the evaluator against representative conversations, including cases that should pass. A detector that agrees with one example has very little evidence behind it. As more reviewed examples accumulate, alignment helps you see whether its judgments match yours.
This saves you from repeatedly searching for the same known behavior by hand. It still needs maintenance: an agent’s tools, instructions, and client requirements can change. The evaluation belongs to its project. When onboarding another agent, assess which checks it needs rather than assuming an existing project’s evaluation covers it.
What changes when you take on more customers #
With five agents, you might remember every open issue. With fifty, you need clear ownership and a review routine that works when nobody remembers what was said in Tuesday’s Telegram thread.
The flow remains the same. Bring each deployment’s telemetry into Latitude, enable appropriate checks, attach client feedback to conversations, and investigate the signals that deserve attention. Use MCP to assemble the fleet view and client reports. After a change, compare the results and keep monitoring the failures that matter.
Your costs and responsibilities will grow. More traffic means more processing and potentially more evaluation spend. New clients bring new failure modes, and serious incidents still require people. What you can avoid is multiplying all the routine work: transcript reading, copying complaints between tools, checking dashboards, and collecting numbers for reports.
For a consultancy, that is capacity you can put toward another customer. The next deployment joins an established monitoring process, and the team spends its attention on understanding and improving the agents it has promised to look after.