{"slug": "evaluation-first-ai-agents-how-zepto-scales-customer-support-on-databricks-and", "title": "Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow", "summary": "Zepto, an Indian quick-commerce platform, reports that its evaluation-first AI agents, built on Databricks and MLflow, now handle over 80% of customer support tickets, cutting support costs by 65% and achieving payback in under one month. The system processes more than 100,000 tickets daily across 60+ cities, with Databricks and MLflow providing the evaluation backbone that enables a dual-loop model for development and production.", "body_md": "How Zepto uses Databricks and MLflow to build evaluation-first AI agents that manage 80%+ of support tickets, cut support costs by 65%, and deliver payback in under one month\n\nZepto is one of India's fastest-growing quick-commerce platforms, with more than thousands of products, a presence in over 60 cities, and delivery windows measured in minutes. In a business where speed is the product, customer support has to move just as fast.\n\nTo meet that expectation, Zepto runs customer support on a multi-agent AI system that processes over a hundred thousand tickets a day. Early on, the team could build and ship agents quickly. The harder question was how to keep those agents reliable as volume grew, categories expanded, and customer behavior kept changing. Zepto partnered with Databricks to answer that question, not by shipping more agents, but by **making evaluation the primary way agents get built, tested, and operated**.\n\nThis blog walks through that journey: the system architecture, the evaluation framework on Databricks and MLflow, the production stories where it earned its keep, and the results and lessons that came out of it.\n\nIn a high-velocity business, \"just ship the agent\" works right up until it breaks at scale. At more than 100,000 AI-agent tickets a day, even a 1% error rate creates thousands of bad outcomes and real revenue leakage every single day.\n\nThe pressure arrived in uneven waves. Weather events, Diwali, and the start of summer drove sharp spikes in ticket volume. Expansion from groceries into apparel, electronics, and beauty introduced new refund, exchange, and return journeys. Meanwhile, a more diverse, multilingual customer base brought a wider range of support requests—and new failure modes surfaced every few weeks.\n\nThe deeper problem is the **assurance gap**. Agentic systems operate as multi-step workflows—classifying intent, retrieving knowledge, analyzing inputs, reasoning through decisions, calling transactional tools, and generating responses—so failures can emerge anywhere along the way, not only in the final answer.\n\nThis assurance gap translated into concrete problems:\n\nThe objective became clear: **engineer an evaluation framework on Databricks and MLflow so it functions as core AI infrastructure on which Agents are built and operated**.\n\nA strong evaluation framework directly affects five axes of production readiness:\n\nWith Databricks + MLflow as the evaluation backbone and an evaluation-first agent architecture, Zepto achieved\n\nCost & efficiency\n\nQuality & reputation\n\nPerformance & operations\n\nAt the core of this approach is the dual-loop model: a development loop and a production loop, connected by a quality gate. This section outlines how those loops work together.\n\nTogether, these two loops ensure that agents are built and operated with control. Any failure is automatically captured, fed back, and corrected. As a result, agents are built with confidence, run with control, and continuously improve to handle production failures better over time. **The dual loop lies at the core of our framework**.\n\nEvery agent invocation emits a rich execution trace that captures prompts, completions, retrieved documents, tool calls, latencies, and decision paths, so the whole workflow is observable at granular level rather than an input and final output.\n\nWe enabled this with a hybrid approach using MLflow. A single line, `mlflow.<library>.autolog()`, turns on [automatic tracing](https://docs.databricks.com/aws/en/mlflow3/genai/tracing/app-instrumentation/automatic), and the `@mlflow.trace` decorator adds [custom spans](https://docs.databricks.com/aws/en/mlflow3/genai/tracing/span-concepts) wherever we need more detail. Traces are emitted in real-time as OpenTelemetry spans with unique IDs so they stay composable, and MLflow's integration with Unity Catalog centralizes logging into Delta tables.\n\nWith tracing enabled, the next step is to capture, from each stakeholder’s perspective, “**What does success for this agent mean to you?**”. We formalize this as evaluation pillars, each with specific gates.\n\nThis turns a multi-stakeholder debate into a shared, measurable contract. Agents are evaluated along the dimensions that actually matter for each stakeholder. **Typical pillars include customer experience, operational efficiency, risk and compliance, and financial impact; each pillar has clear numeric thresholds that must be met before deployment**.\n\nThe [golden dataset](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/build-eval-dataset) is the single source of truth for evaluating agent behavior in the development loop. It should:\n\nEvery stakeholder comes together to shape the dataset, for example the security team contributing examples on adversarial patterns such as prompt injection, identity attacks, data-exfiltration attempts, this ensures that reliability is measured against all real-world scenarios as well as ordinary use.\n\nDatasets are a living asset, and their quality compounds over time. The gap between development and production accuracy is itself a dataset quality signal. Zepto invested steadily in MLflow evaluation datasets over six months, moving from 500 examples and an 8-point dev–prod accuracy gap, to 2,000 examples and a 2-point gap, to 5,247 examples and a 0.4-point gap. Every hour spent on dataset quality saves roughly ten hours of production debugging, so the golden dataset becomes a 10x multiplier: every production failure adds failure traces to the golden dataset and makes the system more robust for all future versions.\n\nRather than hand-writing prompts, we made prompt engineering a data-driven, automated process. Prompt design is the critical phase where engineers spend most of their time, and the **quality of prompts has a disproportionate impact on the quality and performance of agent outputs**.\n\nUsing [MLflow prompt optimization](https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/automatically-optimize-prompts), we register an initial prompt, generate and optimize variants against the same scorers that gate deployment, run A/B evaluations automatically, and deploy the best result. The optimizer reflects with a strong model and production scores candidates with a cheaper one so the search itself stays cost-aware in production. This reduced manual prompt experimentation, improved accuracy, and ensured that prompt improvements were always measured against the golden dataset before reaching production.\n\nWith traces flowing in the production loop, we need to score them along the dimensions that matter for agent quality (Evaluation dimensions). Think of this as an AI jury, where each scorer plays to its strengths. MLflow provides three options for creating [scorers](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/concepts/scorers).\n\nWe use LLM-based scorers only where human-like judgment is necessary and rely on simple rules where deterministic logic is enough. We calibrate the judges against human labels to reach 80–90% agreement and use multiple judges for high-stakes decisions.\n\nModal optionality is a critical component which allows the framework to switch between many proprietary and open-source models simply by changing model names in [Databricks](https://docs.databricks.com/aws/en/machine-learning/model-serving/foundation-model-overview). This means the development loop can continuously search for a better combination to find the best trade-off between cost, performance, and quality.\n\nWe automate regression evaluation to create a repeatable, configurable, and scalable development loop. Any change triggers auto-regression and, when reliability is guaranteed, auto-deployment.\n\nPutting the pieces together, a typical change follows this path:\n\nEvaluating 100% of traffic is expensive, but naive 10% uniform sampling misses most edge cases. We implemented stratified sampling where evaluation sampling rates depend on high-value customers, new features or recently changed flows, negative sentiment or high escalation risk, and image-based or fraud-prone interactions. This yields an effective 18–20% evaluation sample (~14,400 traces per day) at manageable cost, while capturing 45–60% of edge cases and detecting issues within 4–6 minutes.\n\nThe financial logic is compelling: Compared to a uniform sampling approach, the stratified methodology achieved an 86% reduction in review cost per issue identified while delivering a 9X improvement in edge-case detection, making the quality assurance process significantly more efficient and scalable.\n\nEvaluation results are written into Delta tables and surfaced via dashboards and alert rules in Databricks. Critical alerts (checked every 5 minutes) monitor for intent accuracy drops, groundedness violations, high escalation risk, and P95 latency breaches. High/medium alerts track empathy degradation, cost spikes, tool failure rates, CSAT trends, fraud detection rate, and multimodal latency. This enables SRE-like operations for AI agents: fast detection, triage, and mitigation.\n\nA good evaluation framework works far better when the agent architecture is designed to be observable and decomposable in the first place. Zepto's support stack is built around that idea.\n\nA customer query, in chat or image form, first passes through an agentic orchestrator and router. The router can hand off to a human at any point. Underneath it, the system separates into two kinds of agents.\n\nVertical agents are specialists, each owning a single, well-defined intent family:\n\nHorizontal agents act as oversight layers that cut across use cases:\n\nThis separation pays off twice. Metrics can be computed per vertical agent, such as WIMO intent F1 or Expiry OCR accuracy, and horizontal agents can be evaluated on cross-cutting concerns like fraud precision, image reuse, and manipulation detection. Each piece can be measured in isolation and in combination.\n\nWhen you first launch an AI agent into production, it feels like sending a brilliant but unpredictable intern out to represent your company. You give them instructions and hope for the best, but until they are under pressure, you are essentially flying blind.\n\nEarly on, we realized that traditional software monitoring is completely useless for AI. An agent can have perfect server uptime and zero errors while repeating the exact same wrong answer to a frustrated customer. To the engineers, the dashboard looks green. To the customer, it’s a disaster.\n\nWe knew we couldn't scale our AI on hope. We needed an evaluation framework that didn't just track if the AI was talking, but actually understood what it was saying and where it was failing. The following stories are the moments that framework earned its keep, proving that a good evaluation system unlocks entirely new product features.\n\nA production issue left riders stuck in traffic while the agent kept replying \"arriving in 10 mins\" on a loop, because it was reading cached data. The customer asked where their order was, got the same line, asked again, and got the same line again.\n\nThe token-counter (monitoring token usage) and warning scorers caught the repetition and high escalation risk within 5 minutes, surfacing traces of stationary riders with unchanged ETAs. That triggered a rule change. If a rider stays stationary for more than 10 minutes, the agent now gives an honest update and proactively offers cancellation for a full refund, rather than repeating a stale promise.\n\nOne insight, caught by online evaluation, turned into a whole feature line: cancel on delay, a proactive pitch to cancel during rider shortages, auto-cancel if no rider is assigned within a set window, and no-questions-asked cancellation for high-value customers.\n\nWhen Zepto added cancellation handling to the WIMO agent, the model started confusing three very different intents: \"where is my order,\" \"I want to cancel,\" and \"was my order cancelled.\"\n\nDevelopment-phase MLflow evaluation caught it immediately. Overall intent accuracy dropped from about 92.1 percent to 87.4 percent, with poor F1 on the new WIMO_CANCEL and WIMO_CANCEL_STATUS intents. Because the regression showed up against the golden dataset, no customer ever saw it. Prompt optimization and dataset updates restored overall accuracy to about 94.2 percent, better than the original baseline, with near-perfect tool-calling F1 on the cancellation APIs. The feature went live with zero rollbacks.\n\nProduce quality is genuinely hard to score, and humans do not always agree. The same image of mushrooms might get a 2 out of 5 from one rater and a 3 out of 5 from another. We measured that disagreement with Cohen's Kappa and treated it as our reliability ceiling, since no model can be more consistent than the humans it learns from.\n\nWe also found that the AI played it safe. Left alone, it piled up scores at 3 to avoid making a hard call, while human scores peaked at 4 and 5. So we did not just minimize error against the average. We matched the shape of the human score distribution. Online evaluation also surfaced cases the system was not built for, like curdled milk that is shelved as a packaged good but needs to be judged like fresh produce, and taste or smell complaints that a photo simply cannot show, which were routed to a separate path.\n\nThis calibrated baseline lets us decide which models to use per product type, how to iterate prompts against human judgment, and how to tune refund policy by customer segment based on real agent performance.\n\nRefund abuse attempts used catalog images, edited photos, and images reused across claims. The multimodal evaluation pipeline ran images through preprocessing checks for blur, brightness, and resolution, validated them with OCR, and then used a jury of three vision models with consensus rules to decide between auto-approval and human review. Layered on top were blur detection, screenshot detection, duplicate detection, image-versus-SKU matching, image-versus-stated-reason checks, and proof-of-delivery validation.\n\nOperating this framework at scale taught us a handful of principles that generalize beyond quick commerce.\n\nFor organizations building agentic systems on Databricks, the Zepto experience suggests the following roadmap:\n\nIn the transition from experimental demos to mission-critical infrastructure, the primary constraint has shifted from raw model capability to system assurance. Zepto's journey demonstrates that by establishing evaluation as the fundamental development primitive, organizations can reliably scale agents to manage tens of thousands of complex daily interactions across multimodal inputs, backed by rigorous guarantees on quality, cost, and risk mitigation.\n\nDatabricks and MLflow serve as the essential substrate for this evolution, providing trace-centric data infrastructure on Unity Catalog and Delta, alongside scalable evaluation, automated prompt optimization, and seamless CI/CD integration. This composable stack, combined with model optionality, empowers teams to fine-tune the balance between performance and expenditure for every specific task.\n\nUltimately, the competitive advantage lies in the strategic decision to treat evaluation not as a final check, but as core AI infrastructure. The blueprint for operating production-scale agents is no longer a mystery; Zepto and Databricks have provided the answer. The challenge now is the speed of adoption. In the rapidly evolving AI landscape, the leaders will not be those who wait for perfect certainty, but those who engineer for reliability from day one.\n\nFor builders developing agents that must earn trust in production environments, these same foundational building blocks are ready on Databricks and MLflow 3. The organizations that define the next frontier will be the ones that begin their evaluation-first journey today.\n\n[Build Agents on Databricks](https://docs.databricks.com/aws/en/agents/)[Get started with MLflow evaluation and monitoring](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/)\n\nSubscribe to our blog and get the latest posts delivered to your inbox.", "url": "https://wpnews.pro/news/evaluation-first-ai-agents-how-zepto-scales-customer-support-on-databricks-and", "canonical_source": "https://www.databricks.com/blog/evaluation-first-ai-agents-how-zepto-scales-customer-support-databricks-and-mlflow", "published_at": "2026-09-09 03:00:00+00:00", "updated_at": "2026-09-09 03:19:56.823191+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "machine-learning"], "entities": ["Zepto", "Databricks", "MLflow", "Unity Catalog"], "alternates": {"html": "https://wpnews.pro/news/evaluation-first-ai-agents-how-zepto-scales-customer-support-on-databricks-and", "markdown": "https://wpnews.pro/news/evaluation-first-ai-agents-how-zepto-scales-customer-support-on-databricks-and.md", "text": "https://wpnews.pro/news/evaluation-first-ai-agents-how-zepto-scales-customer-support-on-databricks-and.txt", "jsonld": "https://wpnews.pro/news/evaluation-first-ai-agents-how-zepto-scales-customer-support-on-databricks-and.jsonld"}}