You ship an agent. The offline eval set (the scored test cases you run before launch) looks green. Two weeks later, the same prompts miss new cases. Photos that should skip enhancement get sent for edits anyway. Bad images slip through. The model did not “forget” — the world moved.
That is the problem Soumya Gupta and Jai Chopra from Uber’s computer vision team put on stage: how do you keep a multimodal agent (an AI that reads images and text together) sharp when production never stops changing?
Their answer is not a better one-shot prompt. It is a closed loop — evals, human labels, diagnosis, and config-driven autotuning (settings rewritten from feedback, not by hand) that keep every stage of the pipeline alive after launch.
Uber Eats is huge. Jai cites roughly a $90 billion run rate, millions of items added each month, ~20% year-over-year growth, and about 10,000 cities. Delivery is as big as mobility on Uber today. A photo is often the first signal a customer gets about a merchant.
Smaller merchants lack time, know-how, and budget for pro shoots. Menus keep changing. Consumers want authentic food photos — and a meaningful fraction distrust anything that looks AI-generated. Open the app to a feed of “AI slop” and trust dies.
So the team threads a needle. Stay faithful to the original image. Preserve the merchant’s brand. Avoid one shared prompt that makes every dish look the same — that collapses diversity. Quality varies globally: soft focus, bad framing, weak color, plus a long tail of user-generated content.
“We want to be able to improve quality selectively… optimize globally for the entire marketplace. We don’t want to cannibalize certain merchants.”
Design goals sound familiar if you build agents anywhere: preserve trust, enhance only when needed, ship safely, learn continuously, and stay cost-efficient at scale. Rules-based systems are controllable but brittle. Fully free agents are creative but unsafe. The principle is the middle path — agency with guardrails — and that principle drives how they design both agents and evals.
Understand, route, edit, publish, log
Think of a night kitchen with a head chef who decides which plates need a remake. First an understanding agent describes the photo with a multimodal AI language model (LLM). That structured view goes to a router: enhance or skip. Skip keeps the original.
If the router says enhance, an editing agent runs in a loop with a quality-assurance (QA) agent. It self-corrects for up to K tries. Still failing? Do not publish — take the coverage hit. Passes? A final post-processing and publish-ready QA gate runs. Then everything is logged. Logging is not an afterthought. Uber uses a flat JSON structure for the full orchestration so engineers and product folks can inspect one case or roll up aggregates. Jai’s rule: start with logging. Without it you have nothing to optimize — and no self-learning loop. At Uber they use Arise for this observability layer (the tooling that records what each agent did).
If you do not log the full agent path, you cannot build a closed loop later.Flat, shareable traces beat pretty dashboards you cannot drill into. The router takes image, text description, and metadata. It grades against a pass/fail rubric, then chooses enhance or skip. Eval looks like a classic confusion matrix — a grid of right vs wrong routing calls (true positives, false negatives) that yields precision and recall (how often enhance/skip is correct, and how often bad images are caught).
For routing, Uber’s guardrail metric is recall — catching bad images so they do not skip enhancement and reach eaters. More complex routers might send some images to a smaller, lower-latency model to save cost. Then you need an N×N matrix that checks whether each branch was chosen correctly — not just enhance vs skip. Human labels are the golden source of truth — the scored examples you align the agent to. Collect a representative set across geographies, dish types, and image quality. Give labelers objective guidelines to cut subjective noise. Tune the agent against that gold set until guardrails pass — then ship.
Failures teach why static offline models die. A sharp cheeseburger photo routed to enhance wastes compute and risks degrading a good image. A six-wing photo labeled “eight pieces” that the router approved can push the editor to hallucinate two extra wings — a faithfulness miss. Offline green does not mean production green.
Sample, diagnose, reflect, synthesize, ship
Soumya’s drift loop is simple to say and hard to skip. Sample production on a fixed cadence. Send the same objective guidelines to human labelers. Compare agent output to labels. On mismatch, an umbrella diagnosis agent localizes the fault and triggers autotuning.
Autotuning itself uses two sub-agents. A reflect agent studies mismatches, strips noise, and finds systemic issues. A synthesize agent rewrites the agent config from that feedback, then benchmarks again. Pass the golden set? Register the new config. Next production run picks it up.
Soumya stresses the closed loop needs no human in the tuning path — but it does need observability, guardrails, and quick rollback. One static offline model is not enough. This loop is what keeps the system alive.
Enhancement is three steps: generate a prompt specific to this image (using the description plus router directives), enhance, then a multi-dimensional QA gate (plating, faithfulness, colors, and more). Fail? Feed QA notes back into prompt generation and try again.
Two endings: pass within K and publish, or never enhance that image. Their sweet-potato-fries example failed first on portion size and unrealistic plating, then passed on the second try. The metric is pass@K — pass rate at iteration K. Ideally it rises as feedback accumulates.
Generation evals use pairwise comparison of input vs output: is the edit better? “Better” is proprietary in detail, but Uber bakes in product design, policy, and legal. Checks include faithful, complete, natural, realistic. The gate returns yes, no, or unsure.
Faithfulness fail: the editor adds shrimp that was never there. Completeness fail: sauce under sushi disappears. Reward hacking (gaming the QA score): a creative edit gets rejected, then the agent oversteers into a generic ceramic bowl — pixels change a lot, meaning changes little. Object coherence and physics issues from frontier image models leak into the product use case.
Multimodality matters at the gate too. If you cannot count eight wontons in the frame, the system should answer “unsure” and reject — not force a publish. After the edit loop comes publish-ready QA: policy plus broader quality. Why another gate? Jai frames it as a Swiss cheese model — stacked layers so a hole in one check is blocked by another. Overlapping layers cut the chance a failure reaches production. Redundancy is intentional.
The model loop handles drift against offline human gold. Uber also dogfoods before launch — internal teams try the product first — and pulls live feedback after. A higher abstraction — the diagnoser — accepts signals from any of those loops, decides which agent config to fix, and routes the change. One agent or many.
Internal thumbs up/down and free-form notes from merchants, design, and product teams replay as good and bad examples, then re-benchmark before a new config ships. Marketplace health metrics close the story — conversion, add-to-cart, completed orders — sliced by geo, device, and dish type so tuning can target weak segments.
What is a closed-loop eval for an AI agent?
It is a system that keeps measuring the agent in production, comparing outputs to human labels or product signals, diagnosing which stage failed, and updating prompts or configs automatically. Offline evals ship the first version. The closed loop keeps that version from going stale.
Why optimize the router for recall instead of precision?
Uber’s routing guardrail is recall so bad images are less likely to skip enhancement and reach eaters. False enhances cost compute and can hurt already-good photos, but a missed bad image is the higher product risk they call out for that stage.
What does pass@K mean in an image editing loop?
Pass@K is the share of cases that clear the QA gate by attempt K. Each failed pass feeds feedback into the next prompt. If the case still fails after K tries, you skip publishing the edit and accept lower coverage instead of shipping a bad plate.
Offline evals get you to ship. Closed loops — labels, diagnosis, and config autotune — are what keep multimodal agents honest in production.
“A static offline model gets you launched. The closed loop is what keeps the system alive.”
This story was written with the assistance of an AI writing program.
How Uber’s Computer Vision Team Stops AI Slop Before It Ships was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.