{"slug": "the-ai-feature-that-works-in-the-demo-and-breaks-in-production", "title": "The AI Feature That Works in the Demo and Breaks in Production", "summary": "A Towards AI investigation by Emmanuel Nwangura, running 833 tests across 6 AI models, found that a widely sold 'strict enforcement' fix for AI output constraints mostly fails in production, often worsening results. The study identifies model capability and form complexity as the true drivers of failure, while common monitoring checks for 'well-formed' output miss the most damaging issues. The author advises builders to select models before tuning infrastructure, avoid illegal forced answers, validate provider claims, and account for compounded step-by-step failures, releasing the data and method open-source.", "body_md": "Last Updated on August 24, 2026 by Editorial Team Author(s): EMMANUEL NWANGUMA Originally published on Towards AI. Vendors sell a fix for it. I ran 833 tests across 6 AI models to see whether the fix actually works. Mostly, it didn’t. Imagine you hire a very fast, very well-read assistant to process your customer support inbox. The article explains why small AI errors can break real systems—especially when models are chained—showing that even a “strict enforcement” fix often doesn’t help and can even worsen results. Through extensive testing of multiple models across different constraint settings, the author finds that the true drivers of failure are model capability and form complexity, while common monitoring assumptions (like checking for “well-formed” output) miss the most damaging issues. The author also highlights measurement pitfalls, including how test design can force misleading failures and how evaluators can agree with each other while still being wrong. Finally, the piece offers practical guidance for builders and evaluators (pick models before tuning infrastructure, avoid illegal forced answers, validate what providers actually do, and account for compounded step-by-step failure), along with transparent limits of the findings and open-source release of the data and method. Read the full blog for free on Medium. Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor. Published via Towards AI", "url": "https://wpnews.pro/news/the-ai-feature-that-works-in-the-demo-and-breaks-in-production", "canonical_source": "https://www.machinebrief.com/news/the-ai-feature-that-works-in-the-demo-and-breaks-in-producti-phla", "published_at": "2026-08-24 07:26:26+00:00", "updated_at": "2026-08-24 11:45:35.558738+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-ethics"], "entities": ["Towards AI", "Emmanuel Nwangura"], "alternates": {"html": "https://wpnews.pro/news/the-ai-feature-that-works-in-the-demo-and-breaks-in-production", "markdown": "https://wpnews.pro/news/the-ai-feature-that-works-in-the-demo-and-breaks-in-production.md", "text": "https://wpnews.pro/news/the-ai-feature-that-works-in-the-demo-and-breaks-in-production.txt", "jsonld": "https://wpnews.pro/news/the-ai-feature-that-works-in-the-demo-and-breaks-in-production.jsonld"}}