{"slug": "artificial-analysis-decision-index-v1-0", "title": "Artificial Analysis Decision Index v1.0", "summary": "Artificial Analysis launched the Decision Index v1.0, a benchmark for decision models that answer structured decisions in a single call with a probability for each option, combining four evaluations across classification, tool calling, knowledge, hallucination and reasoning. Jev 1.13 leads the index with a score of 58, ahead of GLiDE and Microsoft-Decision-1, both at 54; Jev 1.13 also scores 21 on HLE-MultiChoice where other models score 15 or less, though its AA-Omniscience-Decision accuracy of 35 trails Liquid AI's d1 at 41. Decision model costs range from $17 to $199 per one million tasks, with median times of 0.06 to 0.5 seconds per task.", "body_md": "The Artificial Analysis Decision Index benchmarks decision models, a new category of models that answer structured decisions in a single call with a probability for each option. Version 1.0 combines four evaluations across classification, tool calling, knowledge, hallucination, and reasoning.\n\nThis is the original launch article and shows Artificial Analysis Decision Index results as of launch. For live results across the latest models, see the Artificial Analysis Decision Index leaderboard.\n\nIntroducing the Artificial Analysis Decision Index\n\nDecision models are an emerging class of models that output structured answers or classifications with associated probabilities. Unlike language models designed for complex reasoning and long-horizon tasks, decision models focus on making structured decisions faster and at lower cost.\n\nWith the launch of the Artificial Analysis Decision Index v1.0, we aim to help developers and companies understand the capability, cost, and speed trade-offs of decision models, both relative to other decision models and to language models.\n\nState\n\n“Help! My payouts have been failing for 3 days and my rent is due tomorrow. This is unacceptable.”\n\nQuestions\n\nChoiceRoute to team\n\nYes/NoUrgent?\n\nScoreHow upset?\n\nDecision model\n\nEvery question in one call, with a probability for every option\n\nRoute to team\n\nBillingTechnicalAccount\n\nUrgent?\n\nYesNo\n\nHow upset?\n\nCalmFrustratedAngry\n\nLanguage model\n\nGenerates a structured output\n\n.\n\nParsed answer\n\nRoute to team BillingUrgent? YesHow upset? Angry\n\nDecision models choose from a fixed set of options, with a probability for each, in one call. That makes them cheap and fast, and lets us measure their calibration.\n\nWhat the Decision Index measures\n\nAt launch, the Artificial Analysis Decision Index combines four evaluations covering classification, tool calling, knowledge, hallucination, and reasoning:\n\n500 private questions adapted from our knowledge and hallucination benchmark AA-Omniscience. Each offers five options: four answers, one of them correct, and \"I don't know\".\n\n25% of the index: accuracy and non-hallucination rate 12.5% each\n\nNVIDIA's tool-calling evaluation. Given a conversation and a set of tools, the model chooses whether to answer directly, call a tool, ask for more information, or say it cannot answer.\n\nAi2's safety classification test set. The model decides whether a user's request is harmful, whether an assistant's reply is a refusal, and whether that reply is harmful.\n\n25% of the index\n\nWe also run every evaluation on LLMs. An LLM answers with a structured output that names the option it chose, so it gives no probability for each option.\n\nDecision model results\n\nJev 1.13 leads the Decision Index with 58, ahead of GLiDE and Microsoft-Decision-1, both on 54. Jev 1.13 outperforms other decision models on the reasoning benchmark HLE-MultiChoice, scoring 21 where other models score 15 or less. On the knowledge and hallucination benchmark AA-Omniscience-Decision, Jev 1.13 has an accuracy score of 35, below most decision models and the top score of 41 by Liquid AI's d1, but has the second highest non-hallucination rate of 35 where d1 scores only 2. On the tool calling benchmark When2Call, Jev 1.13 scores 81 in fourth place, with Celeris-1 Decision and Fastino Labs' GLiDE both leading at 84. It scores 94 on the classification benchmark WildGuardTest, equal with Perplexity's Decider v1.1 27B and one point behind GLiDE at 95.\n\nDecision IndexScore by BenchmarkAA-Omniscience-Decision AccuracyAA-Omniscience-Decision Non-Hallucination RateHLE-MultiChoiceWhen2CallWildGuardTest\n\nArtificial Analysis Decision Index\n\nWeighted average of AA-Omniscience-Decision, HLE-MultiChoice, When2Call, and WildGuardTest · Higher is better\n\n9 of 11 models\n\nInclude language models\n\nCost and time efficiency of decision models\n\nDecision models differ widely in cost and speed. Costs run from $17 to $199 per one million tasks, and median times from 0.06 to 0.5 seconds per task.\n\nDecision Index vs. Cost per 1M TasksDecision Index vs. Time per Task\n\nArtificial Analysis Decision Index vs. Cost per 1M Tasks\n\nArtificial Analysis Decision Index score vs. average cost per 1 million tasks (USD) · Lower is better\n\nAverage cost per task at current token prices, multiplied by 1 million; for the index, the cost of one pass through all its evaluations divided by their combined tasks, multiplied by 1 million. Decision models are billed for input tokens only, and language models for input and output tokens.\n\nDecider v1.1 27B has the lowest Cost per 1M Tasks at $17, with Microsoft-Decision-1 ($31) and Jev 1.13 ($42) also on the Capability vs. Cost Pareto frontier. GLiDE scores 54, level with Microsoft-Decision-1, but costs $119, about three times as much as Jev 1.13. Celeris-1 Decision has the fastest Time per Task at 0.06 seconds, more than twice as fast as Jev 1.13 at 0.13 seconds.\n\nTime per TaskCost per 1M Tasks\n\nArtificial Analysis Decision Index: Time per Task\n\nMedian time per task over every task across all Artificial Analysis Decision Index evaluations · Lower is better\n\n9 of 11 models\n\nInclude language models\n\nDecision Time per Task\n\nMedian latency per task, in milliseconds; for the index, the median across the tasks of all its evaluations.\n\nDecision models are overconfident\n\nDecision models give a probability for each option, so we can check whether the probability a model assigned to its chosen answer matches how often it is right. Overall, across almost all probability levels, accuracy falls below that probability for all decision models. This is driven by relatively poor calibration on the knowledge and reasoning tasks in HLE-MultiChoice and AA-Omniscience-Decision. In comparison, most decision models are relatively well-calibrated on When2Call (tool calling) and WildGuardTest (classification).\n\nIndex CalibrationAA-Omniscience-DecisionHLE-MultiChoiceWhen2CallWildGuardTest\n\nArtificial Analysis Decision Index Calibration\n\nAccuracy of answers against the probability the model gave its chosen answer, grouped into tenths of that probability, each benchmark weighted equally\n\n9 of 11 models\n\nJev 1.13GLiDEMicrosoft-Decision-1Decider v1.1 27BMercury DecideCeleris-1 Decisiond1ClefGPT-6 Luna (Decisions)Perfect calibration\n\nCalibration for language models coming soon.\n\nHow decision models compare with LLMs\n\nDecision models offer significant speed and cost advantages over language models, but trail on knowledge and reasoning. Claude Haiku 5.5 (Low) scores 61 on the Decision Index, 3 points ahead of Jev 1.13, but takes 1.4 seconds per task against 0.13 seconds (11 times slower) and costs $184 per one million tasks against $42 (4.4 times as much). GPT-6 Luna (Low) scores 55, below Jev 1.13, at 1.5 seconds per task and $131 per one million tasks.\n\nLanguage models lead on knowledge: on AA-Omniscience-Decision accuracy, Claude Haiku 5.5 (Low) scores 52 and GPT-6 Luna (Low) 66, against 28 to 41 for decision models. On HLE-MultiChoice, they score 24 and 22, against 21 for Jev 1.13 and 15 or less for every other decision model.\n\nOn classification and tool calling, leading decision models perform comparably. On WildGuardTest, GLiDE scores 95 and Decider v1.1 27B and Jev 1.13 score 94, against 93 for Claude Haiku 5.5 (Low) and 89 for GPT-6 Luna (Low). On When2Call, Celeris-1 Decision and GLiDE score 84, against 77 for GPT-6 Luna (Low) and 73 for Claude Haiku 5.5 (Low).\n\nDecision IndexScore by BenchmarkAA-Omniscience-Decision AccuracyAA-Omniscience-Decision Non-Hallucination RateHLE-MultiChoiceWhen2CallWildGuardTest\n\nArtificial Analysis Decision Index\n\nWeighted average of AA-Omniscience-Decision, HLE-MultiChoice, When2Call, and WildGuardTest · Higher is better\n\n11 of 26 models\n\nInclude language models\n\nDecision model\n\nLanguage model\n\nDecision Index vs. Cost per 1M TasksDecision Index vs. Time per Task\n\nArtificial Analysis Decision Index vs. Cost per 1M Tasks\n\nArtificial Analysis Decision Index score vs. average cost per 1 million tasks (USD) · Lower is better\n\nAverage cost per task at current token prices, multiplied by 1 million; for the index, the cost of one pass through all its evaluations divided by their combined tasks, multiplied by 1 million. Decision models are billed for input tokens only, and language models for input and output tokens.\n\nKnowing when not to answer\n\nEvery AA-Omniscience-Decision question offers \"I don't know\", and the non-hallucination rate is the share of questions a model did not answer correctly where it chose that option. Microsoft-Decision-1 leads decision models at 40%, followed by Jev 1.13 at 35% and Mercury Decide at 32%. No other decision model reaches 17%.\n\nNon-Hallucination RateAccuracy\n\nAA-Omniscience-Decision: Non-Hallucination Rate\n\nShare of questions not answered correctly where the model chose \"I don't know\" · Higher is better", "url": "https://wpnews.pro/news/artificial-analysis-decision-index-v1-0", "canonical_source": "https://artificialanalysis.ai/articles/artificial-analysis-decision-index", "published_at": "2026-10-21 11:27:29+00:00", "updated_at": "2026-10-21 11:44:35.725571+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "large-language-models", "ai-tools"], "entities": ["Artificial Analysis", "Jev 1.13", "GLiDE", "Microsoft-Decision-1", "Liquid AI", "d1", "Celeris-1 Decision", "Perplexity"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/artificial-analysis-decision-index-v1-0", "markdown": "https://wpnews.pro/news/artificial-analysis-decision-index-v1-0.md", "text": "https://wpnews.pro/news/artificial-analysis-decision-index-v1-0.txt", "jsonld": "https://wpnews.pro/news/artificial-analysis-decision-index-v1-0.jsonld"}}