{"slug": "how-nubank-ships-ai-agents-in-days-not-weeks", "title": "How Nubank Ships AI Agents in Days, Not Weeks 🤖", "summary": "Nubank's AI team published a research paper, 'Screen Before You Serve,' describing a pipeline in which simulated customers hold multi-turn conversations with candidate support agents and an LLM judge scores the transcripts before any live A/B test. On Nubank's highest-volume support agent in Brazil, the method cut release time to 4.4 days from 21.2 days, and a Qwen open-weight model that replaced GPT-5.2 after screening across 29 configurations and 16,000 simulated conversations raised self-service by 8.8 points while cutting p95 latency 25%. The team cautioned that a simulator earns its keep by ranking agent versions correctly, not by sounding realistic — a simpler frontier-model simulator ranked the best agent version last.", "body_md": "👋 *Hey, Linas here! Every day, I break down 3 stories shaping the future of FinTech & Artificial Intelligence - plus the money movements and trends worth tracking. First time here? 400k+ FinTech and AI leaders get this daily. Join them:*\n\n**Before a new Nubank AI agent talks to a real customer, it talks to thousands of fake ones.**\n\nThat’s the method behind *Screen Before You Serve*, a new research paper from the digital bank’s AI team. Simulated customers hold multi-turn conversations with each candidate agent. An LLM judge then scores the transcripts. Only versions that beat the current one on a pass criterion set in advance go on to a live A/B test.\n\nThe results, measured on Nubank’s highest-volume support agent in Brazil:\n\n- **Faster releases.** Each new agent version took 4.4 days, down from 21.2 days for the previous agent.\n- **Better outcomes.** In a live A/B test, the agent scored 4.9 points higher on self-service rate and 36.7 points higher on transactional NPS, its satisfaction score, than the one it replaced.\n- **An open-weight model in production.** A[Qwen](https://linas.substack.com/p/qwen3-8-27b-local-guide) model replaced GPT-5.2 after screening across 29 configurations and 16,000 simulated conversations. Self-service rose a further 8.8 points, satisfaction held, and p95 latency fell 25%.\n\nThis is a perfect illustration that the constraint on AI agent products has moved from the model to the test loop. Models are good enough for most support work. What separates teams now is **how quickly and cheaply** they can tell whether this week’s version beats last week’s.\n\nOf course, Nubank’s answer comes with a *warning*. A simulator earns its keep by ranking agent versions correctly, not by sounding like real customers. When the team built a simpler simulator on a frontier model, it ranked the best version of the agent last.\n\nThe paper also fits a bigger pattern. Earlier this month we showed [how Revolut cut AI costs 90% and started shipping faster](https://linas.substack.com/p/revolut-ai-coding-costs-open-source-models) by moving 2,000 engineers onto a self-hosted open-weight model. Nubank has now done the customer-facing version, and published the test that made it safe.\n\nWe carefully analysed the whole paper, watched Rohan Ramanath, Nubank’s General Manager of AI, walk through it on video, and checked every headline number against the tables behind it. Some of those numbers hold up. Some are softer than they look. Inside:\n\n- **The table Nubank’s announcement left out.** 3 of 4 changes its engineers made to the agent made it worse. What that base rate means for any team shipping on pure intuition.\n- **The one test that shows whether an AI agent simulator can be trusted** , and why realism is the wrong thing to measure here.\n- **How Nubank replaced GPT-5.2 with an open-weight AI model** , and what routine model swaps do to pricing power.\n- **8 places the headline numbers are softer than they look.** Make sure to read this before citing them in a pitch or an investment memo.\n- **A 6-step playbook for founders** building or selling AI agents.\n- **6 diligence questions for investors** , plus which positions look stronger and weaker after this paper.\n\nNubank’s team now gets nearly five attempts at a better agent in the time it used to take to ship one. Here’s exactly how, starting with what it built.", "url": "https://wpnews.pro/news/how-nubank-ships-ai-agents-in-days-not-weeks", "canonical_source": "https://linas.substack.com/p/nubank-ai-agents-simulated-customers", "published_at": "2026-11-17 10:29:06+00:00", "updated_at": "2026-11-17 19:35:04.880880+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research", "ai-tools", "mlops"], "entities": ["Nubank", "Qwen", "GPT-5.2", "Rohan Ramanath", "Revolut"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-nubank-ships-ai-agents-in-days-not-weeks", "markdown": "https://wpnews.pro/news/how-nubank-ships-ai-agents-in-days-not-weeks.md", "text": "https://wpnews.pro/news/how-nubank-ships-ai-agents-in-days-not-weeks.txt", "jsonld": "https://wpnews.pro/news/how-nubank-ships-ai-agents-in-days-not-weeks.jsonld"}}