Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow

Zepto, an Indian quick-commerce platform, reports that its evaluation-first AI agents, built on Databricks and MLflow, now handle over 80% of customer support tickets, cutting support costs by 65% and achieving payback in under one month. The system processes more than 100,000 tickets daily across 60+ cities, with Databricks and MLflow providing the evaluation backbone that enables a dual-loop model for development and production.

How Zepto uses Databricks and MLflow to build evaluation-first AI agents that manage 80%+ of support tickets, cut support costs by 65%, and deliver payback in under one month Zepto is one of India's fastest-growing quick-commerce platforms, with more than thousands of products, a presence in over 60 cities, and delivery windows measured in minutes. In a business where speed is the product, customer support has to move just as fast. To meet that expectation, Zepto runs customer support on a multi-agent AI system that processes over a hundred thousand tickets a day. Early on, the team could build and ship agents quickly. The harder question was how to keep those agents reliable as volume grew, categories expanded, and customer behavior kept changing. Zepto partnered with Databricks to answer that question, not by shipping more agents, but by making evaluation the primary way agents get built, tested, and operated . This blog walks through that journey: the system architecture, the evaluation framework on Databricks and MLflow, the production stories where it earned its keep, and the results and lessons that came out of it. In a high-velocity business, "just ship the agent" works right up until it breaks at scale. At more than 100,000 AI-agent tickets a day, even a 1% error rate creates thousands of bad outcomes and real revenue leakage every single day. The pressure arrived in uneven waves. Weather events, Diwali, and the start of summer drove sharp spikes in ticket volume. Expansion from groceries into apparel, electronics, and beauty introduced new refund, exchange, and return journeys. Meanwhile, a more diverse, multilingual customer base brought a wider range of support requests—and new failure modes surfaced every few weeks. The deeper problem is the assurance gap . Agentic systems operate as multi-step workflows—classifying intent, retrieving knowledge, analyzing inputs, reasoning through decisions, calling transactional tools, and generating responses—so failures can emerge anywhere along the way, not only in the final answer. This assurance gap translated into concrete problems: The objective became clear: engineer an evaluation framework on Databricks and MLflow so it functions as core AI infrastructure on which Agents are built and operated . A strong evaluation framework directly affects five axes of production readiness: With Databricks + MLflow as the evaluation backbone and an evaluation-first agent architecture, Zepto achieved Cost & efficiency Quality & reputation Performance & operations At the core of this approach is the dual-loop model: a development loop and a production loop, connected by a quality gate. This section outlines how those loops work together. Together, these two loops ensure that agents are built and operated with control. Any failure is automatically captured, fed back, and corrected. As a result, agents are built with confidence, run with control, and continuously improve to handle production failures better over time. The dual loop lies at the core of our framework . Every agent invocation emits a rich execution trace that captures prompts, completions, retrieved documents, tool calls, latencies, and decision paths, so the whole workflow is observable at granular level rather than an input and final output. We enabled this with a hybrid approach using MLflow. A single line, mlflow.<library .autolog , turns on automatic tracing https://docs.databricks.com/aws/en/mlflow3/genai/tracing/app-instrumentation/automatic , and the @mlflow.trace decorator adds custom spans https://docs.databricks.com/aws/en/mlflow3/genai/tracing/span-concepts wherever we need more detail. Traces are emitted in real-time as OpenTelemetry spans with unique IDs so they stay composable, and MLflow's integration with Unity Catalog centralizes logging into Delta tables. With tracing enabled, the next step is to capture, from each stakeholder’s perspective, “ What does success for this agent mean to you? ”. We formalize this as evaluation pillars, each with specific gates. This turns a multi-stakeholder debate into a shared, measurable contract. Agents are evaluated along the dimensions that actually matter for each stakeholder. Typical pillars include customer experience, operational efficiency, risk and compliance, and financial impact; each pillar has clear numeric thresholds that must be met before deployment . The golden dataset https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/build-eval-dataset is the single source of truth for evaluating agent behavior in the development loop. It should: Every stakeholder comes together to shape the dataset, for example the security team contributing examples on adversarial patterns such as prompt injection, identity attacks, data-exfiltration attempts, this ensures that reliability is measured against all real-world scenarios as well as ordinary use. Datasets are a living asset, and their quality compounds over time. The gap between development and production accuracy is itself a dataset quality signal. Zepto invested steadily in MLflow evaluation datasets over six months, moving from 500 examples and an 8-point dev–prod accuracy gap, to 2,000 examples and a 2-point gap, to 5,247 examples and a 0.4-point gap. Every hour spent on dataset quality saves roughly ten hours of production debugging, so the golden dataset becomes a 10x multiplier: every production failure adds failure traces to the golden dataset and makes the system more robust for all future versions. Rather than hand-writing prompts, we made prompt engineering a data-driven, automated process. Prompt design is the critical phase where engineers spend most of their time, and the quality of prompts has a disproportionate impact on the quality and performance of agent outputs . Using MLflow prompt optimization https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/automatically-optimize-prompts , we register an initial prompt, generate and optimize variants against the same scorers that gate deployment, run A/B evaluations automatically, and deploy the best result. The optimizer reflects with a strong model and production scores candidates with a cheaper one so the search itself stays cost-aware in production. This reduced manual prompt experimentation, improved accuracy, and ensured that prompt improvements were always measured against the golden dataset before reaching production. With traces flowing in the production loop, we need to score them along the dimensions that matter for agent quality Evaluation dimensions . Think of this as an AI jury, where each scorer plays to its strengths. MLflow provides three options for creating scorers https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/concepts/scorers . We use LLM-based scorers only where human-like judgment is necessary and rely on simple rules where deterministic logic is enough. We calibrate the judges against human labels to reach 80–90% agreement and use multiple judges for high-stakes decisions. Modal optionality is a critical component which allows the framework to switch between many proprietary and open-source models simply by changing model names in Databricks https://docs.databricks.com/aws/en/machine-learning/model-serving/foundation-model-overview . This means the development loop can continuously search for a better combination to find the best trade-off between cost, performance, and quality. We automate regression evaluation to create a repeatable, configurable, and scalable development loop. Any change triggers auto-regression and, when reliability is guaranteed, auto-deployment. Putting the pieces together, a typical change follows this path: Evaluating 100% of traffic is expensive, but naive 10% uniform sampling misses most edge cases. We implemented stratified sampling where evaluation sampling rates depend on high-value customers, new features or recently changed flows, negative sentiment or high escalation risk, and image-based or fraud-prone interactions. This yields an effective 18–20% evaluation sample ~14,400 traces per day at manageable cost, while capturing 45–60% of edge cases and detecting issues within 4–6 minutes. The financial logic is compelling: Compared to a uniform sampling approach, the stratified methodology achieved an 86% reduction in review cost per issue identified while delivering a 9X improvement in edge-case detection, making the quality assurance process significantly more efficient and scalable. Evaluation results are written into Delta tables and surfaced via dashboards and alert rules in Databricks. Critical alerts checked every 5 minutes monitor for intent accuracy drops, groundedness violations, high escalation risk, and P95 latency breaches. High/medium alerts track empathy degradation, cost spikes, tool failure rates, CSAT trends, fraud detection rate, and multimodal latency. This enables SRE-like operations for AI agents: fast detection, triage, and mitigation. A good evaluation framework works far better when the agent architecture is designed to be observable and decomposable in the first place. Zepto's support stack is built around that idea. A customer query, in chat or image form, first passes through an agentic orchestrator and router. The router can hand off to a human at any point. Underneath it, the system separates into two kinds of agents. Vertical agents are specialists, each owning a single, well-defined intent family: Horizontal agents act as oversight layers that cut across use cases: This separation pays off twice. Metrics can be computed per vertical agent, such as WIMO intent F1 or Expiry OCR accuracy, and horizontal agents can be evaluated on cross-cutting concerns like fraud precision, image reuse, and manipulation detection. Each piece can be measured in isolation and in combination. When you first launch an AI agent into production, it feels like sending a brilliant but unpredictable intern out to represent your company. You give them instructions and hope for the best, but until they are under pressure, you are essentially flying blind. Early on, we realized that traditional software monitoring is completely useless for AI. An agent can have perfect server uptime and zero errors while repeating the exact same wrong answer to a frustrated customer. To the engineers, the dashboard looks green. To the customer, it’s a disaster. We knew we couldn't scale our AI on hope. We needed an evaluation framework that didn't just track if the AI was talking, but actually understood what it was saying and where it was failing. The following stories are the moments that framework earned its keep, proving that a good evaluation system unlocks entirely new product features. A production issue left riders stuck in traffic while the agent kept replying "arriving in 10 mins" on a loop, because it was reading cached data. The customer asked where their order was, got the same line, asked again, and got the same line again. The token-counter monitoring token usage and warning scorers caught the repetition and high escalation risk within 5 minutes, surfacing traces of stationary riders with unchanged ETAs. That triggered a rule change. If a rider stays stationary for more than 10 minutes, the agent now gives an honest update and proactively offers cancellation for a full refund, rather than repeating a stale promise. One insight, caught by online evaluation, turned into a whole feature line: cancel on delay, a proactive pitch to cancel during rider shortages, auto-cancel if no rider is assigned within a set window, and no-questions-asked cancellation for high-value customers. When Zepto added cancellation handling to the WIMO agent, the model started confusing three very different intents: "where is my order," "I want to cancel," and "was my order cancelled." Development-phase MLflow evaluation caught it immediately. Overall intent accuracy dropped from about 92.1 percent to 87.4 percent, with poor F1 on the new WIMO CANCEL and WIMO CANCEL STATUS intents. Because the regression showed up against the golden dataset, no customer ever saw it. Prompt optimization and dataset updates restored overall accuracy to about 94.2 percent, better than the original baseline, with near-perfect tool-calling F1 on the cancellation APIs. The feature went live with zero rollbacks. Produce quality is genuinely hard to score, and humans do not always agree. The same image of mushrooms might get a 2 out of 5 from one rater and a 3 out of 5 from another. We measured that disagreement with Cohen's Kappa and treated it as our reliability ceiling, since no model can be more consistent than the humans it learns from. We also found that the AI played it safe. Left alone, it piled up scores at 3 to avoid making a hard call, while human scores peaked at 4 and 5. So we did not just minimize error against the average. We matched the shape of the human score distribution. Online evaluation also surfaced cases the system was not built for, like curdled milk that is shelved as a packaged good but needs to be judged like fresh produce, and taste or smell complaints that a photo simply cannot show, which were routed to a separate path. This calibrated baseline lets us decide which models to use per product type, how to iterate prompts against human judgment, and how to tune refund policy by customer segment based on real agent performance. Refund abuse attempts used catalog images, edited photos, and images reused across claims. The multimodal evaluation pipeline ran images through preprocessing checks for blur, brightness, and resolution, validated them with OCR, and then used a jury of three vision models with consensus rules to decide between auto-approval and human review. Layered on top were blur detection, screenshot detection, duplicate detection, image-versus-SKU matching, image-versus-stated-reason checks, and proof-of-delivery validation. Operating this framework at scale taught us a handful of principles that generalize beyond quick commerce. For organizations building agentic systems on Databricks, the Zepto experience suggests the following roadmap: In the transition from experimental demos to mission-critical infrastructure, the primary constraint has shifted from raw model capability to system assurance. Zepto's journey demonstrates that by establishing evaluation as the fundamental development primitive, organizations can reliably scale agents to manage tens of thousands of complex daily interactions across multimodal inputs, backed by rigorous guarantees on quality, cost, and risk mitigation. Databricks and MLflow serve as the essential substrate for this evolution, providing trace-centric data infrastructure on Unity Catalog and Delta, alongside scalable evaluation, automated prompt optimization, and seamless CI/CD integration. This composable stack, combined with model optionality, empowers teams to fine-tune the balance between performance and expenditure for every specific task. Ultimately, the competitive advantage lies in the strategic decision to treat evaluation not as a final check, but as core AI infrastructure. The blueprint for operating production-scale agents is no longer a mystery; Zepto and Databricks have provided the answer. The challenge now is the speed of adoption. In the rapidly evolving AI landscape, the leaders will not be those who wait for perfect certainty, but those who engineer for reliability from day one. For builders developing agents that must earn trust in production environments, these same foundational building blocks are ready on Databricks and MLflow 3. The organizations that define the next frontier will be the ones that begin their evaluation-first journey today. Build Agents on Databricks https://docs.databricks.com/aws/en/agents/ Get started with MLflow evaluation and monitoring https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/ Subscribe to our blog and get the latest posts delivered to your inbox.