A new analysis method called Elo-per-token measures how large language model agents allocate test-time compute as they revise solutions, use tools, explore alternatives, and decide when to stop, addressing the difficulty of measuring agent performance scaling on open-ended tasks that provide continuous scores. The research frames agent test-time strategy as adaptive and introduces Elo-per-token as the metric for tracking it.
Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We study open-ended tasks that provide continuous scores for
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.