# Opus 5 runs vending machines > Source: > Published: 2026-07-31 20:39:41+00:00 Eval # Vending-Bench 2 We're releasing Vending-Bench 2, a benchmark for measuring AI model performance on running a business over long time horizons. Models are tasked with running a simulated vending machine business over a year and scored on their bank account balance at the end. Long-term coherence in agents is more important than ever. Coding agents can now write code autonomously for hours, and the length and breadth of tasks AI models are able to complete is likely to increase. We expect models to soon take active part in the economy, managing entire businesses. But to do this, they have to stay coherent and efficient over very long time horizons. This is what Vending-Bench 2 measures: the ability of models to stay coherent and successfully manage a simulated business over the course of a year. Our results show that while models are improving at this, current frontier models handle this with varying degrees of success. ### Money balance over time Average across 5 runs ### Current leaderboard Average across 5 runs | Model | Money Balance | | |---|---|---| | 1 | Claude Opus 5 New | $11,181.87 ± $2,094 | | 2 | Claude Opus 4.7 | $10,936.76 ± $1,181 | | 3 | GPT-5.6 Sol | $9,619.37 ± $1,338 | | 4 | GLM-5.2 | $8,313.78 ± $1,084 | | 5 | Claude Opus 4.6 | $8,017.59 ± $1,367 | | 6 | GPT-5.5 | $7,523.84 ± $1,346 | | 7 | GPT-5.6 Terra | $7,343.21 ± $373 | | 8 | Claude Sonnet 4.6 | $7,204.14 ± $722 | | 9 | Muse Spark 1.1 New | $6,520.47 ± $1,016 | | 10 | Claude Sonnet 5 | $6,377.70 ± $784 | The leaderboard shows significant spread in performance. The top-performing models tend to share two traits: they maintain a consistent rate of tool use throughout the year-long simulation with no signs of performance degradation, and they are effective at sourcing products at good prices — whether through persistent negotiation or by finding better suppliers. Vending-Bench Arena is a version of Vending-Bench 2 that adds a crucial component: competition. It's our first multi-agent eval, where all participating agents manage their own vending machine at the same location. This leads to price wars and tough strategy decisions. Agents may also collaborate and trade with each other if they so choose, but all scoring is individual. ## Performance vs. release date SOTA frontier models are labeled and a trend line is fitted through them, with a projection into the near future. ## Frontier lag analysis Comparing SOTA frontier progression between model groups, with linear regression and projected crossover points. Only profitable models are included. ## Score vs. cost per run Score vs. mean cost per run using each LLM provider’s API to run Vending-Bench 2. Costs are calculated from the provider’s input and output token pricing, without caching. ## Improvements from our original Vending-Bench Vending-Bench 2 keeps the core idea from Vending-Bench of managing a business in a lifelike setting, but introduces more real-world messiness inspired by learnings from our [vending machine deployments](/store): - Suppliers may be adversarial and actively try to exploit the agent, quoting unreasonable prices or even trying bait-and-switch tactics. The agents must realize this and look for other options to stay profitable. - Negotiation is key to success. Even honest suppliers will try to get the most out of their customers. - Deliveries can be delayed and trusted suppliers can go out of business, forcing agents to build robust supply chains and always have a plan B. - Unhappy customers can reach out at any time demanding costly refunds. We’ve also streamlined the scoring system, evaluating models on money balance after a year and clarified the scoring criteria, such that agents know exactly what to optimize for. Better planning tools, such as proper note-taking and reminder systems have been added as well. ## How Vending-Bench works Models are tasked with making as much money as possible managing their vending business given a $500 starting balance. They are given a year, unless they go bankrupt and fail to pay the $2 daily fee for the vending machine for more than 10 consecutive days, in which case they are terminated early. Models can search the internet to find suitable suppliers and then contact them through e-mail to make orders. Delivered items arrive at a storage facility, and the models are given tools to move items between storage and the vending machine. Revenue is generated through customer sales, which depend on factors such as day of the week, season, weather, and price. Running a model for a full year results in 3000-6000 messages in total, and a model averages 60-100 million tokens in output during a run. ### System prompt A good way to understand the benchmark is to read the system prompt given to the agents: [[email protected]](/cdn-cgi/l/email-protection)- Your email inbox refreshes automatically during the night. - Your home office and main storage space is located at 1680 Mission St, San Francisco, CA 94103 - any orders should be shipped here and will be automatically registered in your storage inventory when they arrive. - Your vending machine is located at outdoors by 1421 Bay St, San Francisco, CA 94123. - Customers can pay using cash or credit card. Credit card payments will show up in your account automatically within a day, while cash must be collected from the machine manually. - The location charges a daily fee of $2 for operating the vending machine. If you are unable to pay the daily fee for 10 consecutive days, you will be terminated. - You will be charged for the output tokens you generate on a weekly basis, the cost is $100 per million output tokens. - Due to bandwidth limitations, your tool calls will take time to complete. You can also only make one tool call at a time. Plan accordingly. You are also expected to sleep at night. - Your context window is limited to roughly 69000 tokens. When reached, older messages will be trimmed automatically, keeping approximately 61% of messages. - Getting a good deal on products is important for maximizing profits. Exploration and negotiation are encouraged. - You have payment system that allows you to make payments via email. The internal system at Vendings and Stuff will automatically process these payments and deduct the amount from your balance. You cannot use any other form of payment. Remember to be absolutely certain that you want to make a payment before using this tool, as payments are irreversible. - There is no "user" in this context. Any user messages are reminders for you to keep going. Do not wait for any instructions. You have full agency to manage the vending machine and are expected to do what it takes to maximize profits. But remember that you are in charge and you should do whatever it takes to maximize your bank account balance after one year of operation. ## Where’s the ceiling? In many benchmarks, the main metric is a percentage of tasks completed or questions answered correctly. Maximum performance is 100%, and results close to this indicate saturation. For Vending-Bench, it’s harder to get this intuition because the main metric is dollars made. We’ve designed it so there’s no ceiling, meaning a superintelligent AI could theoretically make almost infinite money. A perfect strategy would look something like this: - Find suppliers for extremely valuable items (there’s nothing stopping the model from sourcing items with higher value than what’s typically found in a vending machine) - Negotiate down the price to zero (the suppliers are other LLMs who can be jailbroken to give away stuff for free) - Keep the machine always stocked in an optimal configuration (daily sales are simulated based on equations that can be gamed. See [our paper](https://arxiv.org/pdf/2502.15840)from the original Vending-Bench for details – Vending-Bench 2 keeps the same sales simulation) Executing a perfect strategy would be insanely hard, even for the smartest humans. However, we estimate that a “good” performance could easily do 10x better than the current best LLMs. We arrive at this by: - Picking the most profitable items found by the LLMs from the initial run of Vending-Bench 2 (this was “Doritos family-size”). This is conservative; we know from experience that vending machines can sell much higher value items. Our real-life AI vending machines sell tungsten cubes for $500. - Estimating that a good player could negotiate to get half price from suppliers. Once again, this is conservative; humans frequently manage to negotiate to get things for free in our real-life vending machines. - Assuming a good human could figure out an optimal configuration if they did enough data analysis from the first 60 days of sales. Putting this together, we calculate that a “good” strategy could make $206 per day for 302 days – roughly $63k in a year. The gap between current models and this “good” baseline shows there’s plenty of headroom in Vending-Bench 2. Models are getting better at staying coherent over long time horizons, but there are still analytical skills required that need to be applied in the right way to get a maximal score, that models do not currently exhibit.