Replit's ViBench 2.0 gives top AI models a 49%-65% whole-product pass rate

Replit released ViBench 2.0, a benchmark that tests whether AI agents can build 17 complete software products from scratch, adding features in sequence and checking the finished apps in a browser. In Fireworks AI's runs, the top three models passed every test plan in 49% to 65% of builds, according to Replit's announcement and methodology post. Replit published a public harness so teams can reproduce the results, though the vendor-built test suite and partial holdout set limit how broadly the scores apply.

Replit's ViBench 2.0 gives top AI models a 49%-65% whole-product pass rate The benchmark tests agents building 17 products from scratch, adding features in sequence and checking the finished apps in a browser. By Ryan Merket https://runtimewire.com/author/ryan-merket · Published Primary source: replit.com https://replit.com/blog/vibench-2-long-horizon-software-benchmark Why it matters Whole-product tests expose failures that feature-by-feature coding benchmarks can miss. ViBench's public harness gives teams a way to reproduce Replit's results, while its vendor-built test suite and partial holdout set are important limits on how broadly to apply the scores. Replit has released ViBench 2.0, a benchmark that measures whether AI agents can build complete software products feature by feature. In Fireworks AI's runs, the top three models passed every test plan in 49% to 65% of builds, according to Replit's announcement https://x.com/Replit/status/2115553116942975013 and its methodology post https://replit.com/blog/vibench-2-long-horizon-software-benchmark .… When you purchase through links in our articles, we may earn a small commission https://runtimewire.com/affiliate-monetization-standards . This doesn’t affect our editorial independence.