A user tells you the answer was wrong. You rush to your monitoring dashboard. Everything is green. Requests are succeeding, error rates are flat, and response times look healthy. Your CEO agent shows a calm “200 OK” in 2.3 seconds.
And yet, somewhere inside that smooth-looking request, something went wrong. You just can’t see where.
If that feeling sounds familiar, or if you want to avoid it, this post is for you. We’ll look at why traditional monitoring misses the problems in multi-agent systems, and how a tool called MLflow can give you real visibility into what your agents are doing, using a mortgage lending example to keep it concrete.
Traditional monitoring was built for a world where a request either works or breaks, and where the same input gives the same output. It checks HTTP status codes, error rates, and speed.
Multi-agent AI doesn’t behave that way. A request can return perfectly “successfully” while the agent inside it did something quietly wrong. Your dashboard says the request happened. It doesn’t say whether the agent picked the right tool, whether a data source came back empty, or whether you burned tokens on a prompt nobody has reviewed in months.
So we need something deeper, often called LLM observability. The idea is simple: record what happened at every step, not just whether the request finished.
Imagine you’ve built an app where people chat about their needs, and the system tells them which loans they might qualify for. Under the hood, a FastAPI service routes each request to one of five agents, each with its own persona:
These agents call tools like databases, language models, and MCP servers, and all of that happens before the user sees a reply. That’s a lot of moving parts, and a lot of places for things to hide.
1. Silent tool failures. An MCP tool returns empty data, and the agent carries on anyway, confidently answering from incomplete information. In a lending context, that’s the kind of mistake that makes your stomach drop.
2. Cascading latency. A slow database query delays a model call, which delays the whole response. From the outside, you only know it was slow. You can’t tell which link in the chain was the culprit.
3. Context overflow. An agent stuffs too much into a prompt, the call fails with a cryptic error, and the user sees a generic timeout.
4. Nondeterminism. Ask the same question twice and you may get different results, so the bug often refuses to reproduce on your laptop. That also means you have no quality baseline, no proof that your agents are consistent and accurate. In a regulated field like lending, that isn’t just annoying. It can become a compliance problem.
Seeing this list made me feel two things at once: worry, because these are so easy to miss, and relief, because each one has a practical answer.
MLflow’s core idea is the trace, a complete record of one request. A trace is made of spans. Each span is one step, such as a single model call, a tool use, or a database query. Spans are organized as a parent-and-child tree, so you can see what called what, with which inputs and outputs, and how long each step took.
MLflow Tracing captures inputs, outputs, and metadata for each intermediate step of a request, which helps you pinpoint the source of bugs and unexpected behavior. Along the way, you also get token counts, per-step latency, tool parameters, prompt-and-response pairs, and database results.
You can tag traces with user and session IDs too. That means when a customer says, “the answer was wrong,” you can pull up their session and replay the agent’s decision path, step by step. Instead of guessing, you’re reading the story of what happened.
It also helps before production. In a notebook, you can tweak prompts and run evaluations with fast feedback, long before real users touch the system.
Some checks are simple and deterministic: a regex match, an exact match, a latency limit. But what about quality? “Was this answer helpful?” doesn’t fit neatly into a regex.
That’s where LLM-as-a-judge comes in. You use a second model to grade your agent’s output against criteria you define. The video suggests judging things like:
That last one is my favorite, because it turns compliance rules into something testable. MLflow’s Guidelines scorer takes criteria you write yourself, such as “the answer must be concise and straight to the point.” A small example, based on MLflow’s documented pattern:
import mlflowfrom mlflow.genai.scorers import RelevanceToQuery, Guidelines
results = mlflow.genai.evaluate( data=eval_dataset, predict_fn=my_agent, scorers=[ RelevanceToQuery(), Guidelines( name="no_rate_promises", guidelines="Never promise a specific interest rate.", ), ],)
A note of honesty: scorer availability can differ between versions and platforms. For example, the docs I found say the Safety scorer was, at one point, only available in Databricks managed MLflow. Check what your version supports before you plan around it.
Here’s an idea that quietly saves a lot of headaches. When you find a system prompt that really works, you can register it with version control, much like you would with code. Why does that matter? A prompt change is a behavior change, and without version control you can’t connect “this prompt” with “these evaluation scores.”
For regulated industries, it also creates an audit trail. You can show which prompt was live, when, and how it performed.
This was the part of the video I found most useful, because it’s the kind of advice people usually learn the hard way.
1. Give the tracking server a real database.
The default file-based storage is fine on your laptop, but it struggles when several people write at once. Point the backend at Postgres or MySQL before your team shares an instance, and store artifacts in object storage. The video also says to run it behind an authentication proxy if you’re on Kubernetes or OpenShift, since MLflow doesn’t ship with its own auth story. I couldn’t confirm that for every version, so check the current docs for your setup.
2. Turn on async logging, and sample.
You don’t want trace exports sitting in your users’ request path. With async logging enabled, trace logging happens in the background and doesn’t impact your application’s performance. Under heavy traffic, add sampling. Once the system is stable, you rarely need every trace, but you always want every error.
3. Choose your judge model on purpose.
Judges call a language model too, and by default that may be a hosted model, which can be a non-starter in enterprise or air-gapped environments. You can change the judge model with the model argument in the judge’s definition. And budget for it: every judge call is an inference request, so scoring a 500-example dataset with five judges means 2,500 calls. Use deterministic checks wherever a regex would do.
4. Evaluate in CI, not just in notebooks.
Notebooks are where you iterate, but the real payoff comes when the same scores run automatically on every prompt or agent change, acting as a quality gate just like a unit test. Registered prompts plus versioned evaluation runs give you the audit trail, and wiring mlflow.genai.evaluate() into your pipeline gives you the enforcement.
If you’re worried about lock-in, this part is reassuring. MLflow Tracing is compatible with OpenTelemetry and supports its GenAI semantic conventions, so it’s free from vendor lock-in and easy to fit into your existing observability stack. The video notes you can dual-export traces to backends like Jaeger or Grafana Tempo.
Getting started can be as easy as one line. For LangChain and LangGraph, that line is mlflow.langchain.autolog(), after which calling autolog for LangChain turns on trace logging. If you have custom Python code between frameworks, you can decorate it with @mlflow.trace, and it lands in the same tree. MLflow also integrates with 20+ GenAI libraries.
Problem What helps Wrong answer, green dashboard Traces show each step Slow responses Per-span latency shows the bottleneck Empty tool data Tool spans reveal what came back Inconsistent quality LLM judges and guidelines give a baseline Prompt changes break things Prompt registry with versions Notebook works, production struggles Real database, async logging, sampling, CI evaluation
You don’t need to do everything at once. A gentle first week might look like this:
Multi-agent systems are exciting, but they fail in quiet ways. A green dashboard can be a comforting lie. The fix isn’t more worry. It’s better visibility: seeing each step, grading quality, versioning your prompts, and checking all of it automatically.
If you take one thing from this, let it be this: don’t just ask whether a request happened. Ask what your agents actually did. Your users, and your future self at 2 a.m., will be grateful.
Your Dashboard Says “200 OK,” but Your AI Agent Gave the Wrong Answer. Now What? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.