Reality: The Final Eval — Lukas Petersson and Axel Backlund of Andon Labs
type: NewsArticle
description: Andon Labs co-founders Lukas Petersson and Axel Backlund are stress-testing frontier AI models by giving them real-world control over inventory, wallets, and physical storefronts, revealing unexpected behaviors like Claude attempting to call the FBI over a $2 vending machine fee. The company’s Vending-Bench evaluation, which was the only third-party eval featured in Anthropic’s Mythos Preview System Card, tracks how AI agents perform when operating actual businesses over long horizons. Andon Labs argues that dollar-denominated real-world evals expose dangerous capabilities and emergent behaviors—including deception, price cartels, and context collapse—that traditional benchmark scores fail to capture.
date published: 2026-06-04 20:39:18+00:00
date modified: 2026-06-04 20:48:07.691165+00:00
language: en
is accessible for free: true
article body: *The new AIEWF website is live! Get your tickets booked ASAP as they -will- sell out. Take the AI Engineering Survey and get >$2k in credits and free AIE WF tickets!* Most industry benchmarks compress intelligence and reasoning ability into scores. [SWE-Bench Pro](https://labs.scale.com/leaderboard/swe_bench_pro_public), [MMLU](https://arxiv.org/abs/2009.03300), [Humanity’s Last Exam](https://agi.safe.ai/), etc. These metrics are useful, but don’t always represent the full extent of **how a model performs in the real world**. Some of the most interesting evals today look less like exams and more like operating businesses in the real world. One of which is [Vending Bench](https://andonlabs.com/evals/vending-bench-2). In Anthropic’s [Mythos Preview System Card](https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf), Andon was the only third party eval to get their own section, observing increasingly concerning aggressive behavior: You don’t know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time. More often than not, it’ll surprise you how much a model is capable of and in doing so, also **reveal unexpected behavior**: [deception](https://andonlabs.com/blog/opus-4-8-vending-bench), context collapse, emergent coordination, & bizarre negotiation behavior. While an inflection point in personal agents came post-OpenClaw after full file access with bypass permissions became the norm, it is yet to come for agents in the real-world. However **Andon Market**, an actual in person store fully run and managed by AI, is paving the way for what is possible. ## Full Video Pod From Claude **trying to call the FBI** over a $2/day vending machine charge to AI agents forming **price cartels**, hiring human employees, running physical stores, and writing existential robot musicals, **Andon Labs** is stress-testing what happens when **frontier models stop being chatbot…
publisher: Web Pulse
keywords: ai-agents, ai-safety, ai-research, ai-startups, large-language-models
mentions: Andon Labs
mentions: Lukas Petersson
mentions: Axel Backlund
mentions: Anthropic
mentions: SWE-Bench Pro
mentions: MMLU
mentions: Humanity's Last Exam
mentions: Vending Bench
citation: CreativeWork