{"slug": "vals-ai-s-rctbench-catches-ai-decision-models-failing-basic-budget-control", "title": "Vals AI's RCTBench Catches AI Decision Models Failing Basic Budget Control", "summary": "Vals AI's RCTBench, a new benchmark that has nine decision models run a RollerCoaster Tycoon 2 park in OpenRCT2 for 96 weekly turns, found that only three of nine models reached the 1,000-guest goal, each in just one of three runs. Perplexity's Decider V1.3 topped the leaderboard at 70.96% average score at $0.29 per run, using a repeated borrow-and-spend strategy of three $5,000 loans within the first 19 weeks of every run. Five additional frontier models could not be evaluated at all because a mature park's state reaches roughly 44,000 tokens, exceeding their providers' input or option-count limits.", "body_md": "# Vals AI's RCTBench Catches AI Decision Models Failing Basic Budget Control\n\nNine frontier decision models were given three in-game years to grow a RollerCoaster Tycoon 2 park to 1,000 guests. Six failed.\n\n- Vals AI launched [RCTBench](https://www.vals.ai/benchmarks/rctbench?utm_source=alphasignal) , scoring decision models on running a RollerCoaster Tycoon 2 park.\n- Perplexity's Decider V1.3 tops the leaderboard at 70.96%, averaging just $0.29 per run.\n- Only three of nine models ever hit the 1,000-guest goal, and only once each.\n- Five frontier models could not play because park state exceeds their API limits (~44K tokens).\n- Model confidence (37-48% on chosen move) did not correlate with winning or losing.\n- Full decision-by-decision traces available in the [writeup](https://www.vals.ai/blogs/building-rctbench?utm_source=alphasignal) .\n\nVals AI’s new [RollerCoasterTycoonBench](https://www.vals.ai/benchmarks/rctbench?utm_source=alphasignal) (RCTBench) asks a decision model to build a park from an empty plot in [OpenRCT2](https://openrct2.io/?utm_source=alphasignal) and attract 1,000 guests within three in-game years. Nine models completed three runs apiece, producing 27 runs. Only three reached the guest target, each in one run; the others often exhausted their cash, stalled, or repeated low-value actions.\n\nPerplexity’s Decider V1.3 led the benchmark with an average score of 70.96%. Its [borrow-and-spend strategy](https://www.vals.ai/blogs/building-rctbench?utm_source=alphasignal) was consistent: borrow $5,000 early, spend it immediately, and repeat. The model took three such loans within the first 19 weeks of every run, each time while holding less than $1,600 in cash. Its parks finished with 965 to 1,042 guests.\n\n## The harness makes every choice stick\n\nRCTBench maintains a continuing game state across every model call. Given a text representation of the park and a fixed list of available actions, the model assigns a probability to each action. The harness executes the highest-ranked choice, advances the simulation when instructed, and feeds the resulting state into the next call.\n\nDuring each weekly turn, the model can open as many as six game windows, one at a time, and make one change in each before choosing “done.” The windows cover construction, paths, land, staff, attractions, marketing, finances, and research. The live game generates every option, ensuring that actions are legal and affordable, and the harness shuffles their order on each call.\n\nEach state includes the information available to a player: finances, ride statistics, staff, guest thoughts, a text map, and construction choices with prices. A run lasts 96 weekly turns. Half of the score reflects progress toward 1,000 guests and a park rating of 600; the other half reflects company value divided by $100,000. A park with 900 guests, a qualifying rating, and $40,000 in company value scores 65%.\n\n## Decider V1.3 leads a noisy table\n\n| Average scores across three runs and reported cost per run |  |  |  | \n|---|---|---|---|\n| Rank | Model | Score | Cost per run | \n|---|---|---|---|\n| 1 | Decider V1.3 | 70.96% | $0.29 | \n| 2 | Decider V1.2 | 67.06% | $0.21 | \n| 3 | Jev 1.13.0 | 59.22% | $2.18 | \n| 4 | Drex Max | 56.87% | $0.37 | \n| 5 | GPT-6 Luna Decisions | 55.44% | $3.76 | \n| 6 | Jev 1.14 | 42.74% | $1.07 | \n| 7 | Gemini Decision 1 | 29.13% | $1.12 | \n| 8 | Decider Flash V1.3 | 21.43% | $0.10 | \n| 9 | Decider Flash V1.2 | 5.31% | $0.53 | \n\nFive additional models could not enter the evaluation because their providers capped either input length or option count below the environment’s requirements. A mature park can produce a state of roughly 44,000 tokens, while its action list can exceed per-call limits. The leaderboard excludes those models and assigns them no score.\n\n## Five ways agents lose the park\n\n- **Optimization loops.** Decider Flash V1.2 changed ride-inspection intervals 439 to 469 times per run but never hired a mechanic. Most rides had broken down by the end, and its parks finished with fewer than 120 guests.\n- **Up-front overbuilding.** In its first run, Decider Flash V1.3 built 17 rides and stalls within four weeks, then stopped building. It hired 99 staff, including 65 entertainers, and finished $75,967 overdrawn.\n- **Unmanaged debt.** Twenty of the 27 runs borrowed money, and none of those borrowers repaid any principal.\n- **Affordable inaction.** Drex Max passed 43 to 49 of the 96 weeks without opening a window. During 28 to 35 of those idle weeks, it had enough cash to build.\n- **Premature coasting.** Decider V1.2 passed 1,000 guests in week 34 of its first run, then skipped 38 of the final 48 weeks while its cash balance rose to $15,826.\n\n## Confidence misses the outcome\n\nThe models’ reported probabilities for their selected actions averaged 37% to 48%, with no clear separation between high-scoring and low-scoring systems. Decider V1.3’s successful choices and Decider Flash V1.2’s damaging choices occupied similar confidence ranges. These probabilities express relative action preference, with no demonstrated calibration to eventual performance.\n\nAn ablation that hid construction prices reduced average guest counts for seven of the nine models. Drex Max averaged 836 guests when prices were visible and 409 when they were omitted. Price data lets a model connect borrowing or spending to specific construction opportunities before committing to an action.\n\n## Prompt design moves the policy\n\nVals replayed 100 decisions for each of six models under controlled variations. Resending an identical request changed 0% to 9% of selections, establishing a baseline for model variability. Shuffling the option order changed 0% to 34%, revealing substantial position sensitivity in some systems.\n\nRemoving the stated role and objective changed 19% to 35% of selections for four of the six models, exceeding the effect of option shuffling in those cases. The measured policy therefore includes the model, prompt, option order, state representation, and action schema.\n\n## Long horizons expose control failures\n\nTheme-park management combines pricing, staffing, debt, marketing, construction, maintenance, and delayed feedback in one evolving state. An early loan can expand capacity, while a missed mechanic hire may damage attendance weeks later. Random seeds also change guest behavior and ride breakdowns between runs.\n\nVals reports mean scores with standard errors, but three runs per model leave small ranking gaps inconclusive. The traces provide stronger evidence about recurring behavior: repetitive actions, uncontrolled spending, prolonged inactivity, unmanaged debt, and weak continuation policies after milestones.\n\n## What an agent harness should catch\n\n- **Hidden costs.** Expose prices and resource consequences alongside each available action.\n- **Action loops.** Track repeated choices and trigger replanning when they fail to improve the state.\n- **Budget drift.** Require explicit limits, approvals, or repayment rules for borrowing and rapid spending.\n- **Idle turns.** Flag repeated no-op decisions when affordable actions remain available.\n- **Ambiguous completion.** Define terminal conditions and post-milestone objectives so an agent does not stop progressing too early.\n\nVals publishes the complete decision explorer for all 27 runs on the [benchmark page](https://www.vals.ai/benchmarks/rctbench?utm_source=alphasignal). The traces reveal the behavior behind each score and support closer evaluation of models intended for long-running, budget-sensitive workflows.", "url": "https://wpnews.pro/news/vals-ai-s-rctbench-catches-ai-decision-models-failing-basic-budget-control", "canonical_source": "https://alphasignal.ai/news/vals-ai-s-rctbench-catches-ai-decision-models-failing-basic-budget-control", "published_at": "2026-11-19 17:00:01+00:00", "updated_at": "2026-11-19 20:12:19.721690+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "artificial-intelligence"], "entities": ["Vals AI", "RCTBench", "Perplexity", "Decider V1.3", "OpenRCT2", "RollerCoaster Tycoon 2", "Decider Flash V1.2", "Jev 1.13.0"], "also_reported_by": [], "takeaway": "Across 27 runs, only three of nine decision models hit RCTBench's 1,000-guest target once each, and model confidence of 37-48% on the chosen move did not correlate with winning or losing.", "recommendation": {"advice": "Test decision agents in a stateful harness that shuffles action order and caps turns, and check whether confidence scores track outcomes; on RCTBench they did not.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/vals-ai-s-rctbench-catches-ai-decision-models-failing-basic-budget-control", "markdown": "https://wpnews.pro/news/vals-ai-s-rctbench-catches-ai-decision-models-failing-basic-budget-control.md", "text": "https://wpnews.pro/news/vals-ai-s-rctbench-catches-ai-decision-models-failing-basic-budget-control.txt", "jsonld": "https://wpnews.pro/news/vals-ai-s-rctbench-catches-ai-decision-models-failing-basic-budget-control.jsonld"}}