Which AI models understand insurance work?

Coverage Cat's AI Insurance Benchmark ranks xAI's Grok 4.3 first in price-estimation Elo at 1,687, while OpenAI's ChatGPT 5.5 leads in quote coverage at 69.1% across 19,873 unique quote cases. The benchmark evaluates frontier models on insurance price estimation and brokerage/agent task reasoning, with separate leaderboards for each task.

Coverage Cat AI Insurance Benchmark Which AI models understand insurance work? We evaluate frontier models on two separate insurance tasks: price estimation for anonymized umbrella quote rows, and brokerage/agent task reasoning against benchmark reference answers for underwriting, eligibility, and coverage questions. Leaderboard Price-estimation performance Price-estimation results compare predicted premiums and uncertainty ranges against actual quote outcomes. These metrics are separate from the brokerage/agent task leaderboard. Grok 4.3 1,687 Elo ChatGPT 5.5 69.1% coverage 19,873 2,839 unique quote cases Elo Rating Pairwise model strength on the same insurance quoting cases. Higher scores mean a model more often beat comparable models on calibrated quote accuracy. Win Rate The share of pairwise quote battles won, with ties counted as half a win. Higher is better. Quote Coverage How often the actual annualized premium landed inside the model's predicted quote range. Higher is better. Quote MAPE Mean absolute percentage error for each model's point premium estimate. Lower is better. Winkler Loss A calibration penalty for quote ranges that miss the actual premium or are unnecessarily wide. Lower is better. Model comparison Price-estimation ranking | Rank | Model | Elo | Win rate | Coverage | Quote MAPE | Winkler loss | Record | |---|---|---|---|---|---|---|---| | 1 | Grok 4.3 xAI | 1,687 | 45.4% | 45.6% | 49.5% | 2880.00 | 7717-9290-27 | | 2 | ChatGPT 5.5 OpenAI | 1,577 | 53.9% | 69.1% | 94.0% | 3311.46 | 9162-7831-41 | | 3 | Claude Opus 4.7 Claude | 1,547 | 54.2% | 54.1% | 49.5% | 2393.84 | 9212-7765-57 | | 4 | GLM 5 Z.ai | 1,462 | 58.7% | 44.4% | 51.8% | 2685.73 | 9985-7035-14 | | 5 | Kimi K2.5 Moonshot AI | 1,451 | 44.4% | 48.9% | 62.3% | 3175.43 | 7519-9420-95 | | 6 | Mistral Large Mistral | 1,406 | 44.3% | 41.0% | 65.7% | 2942.89 | 7466-9402-166 | | 7 | DeepSeek 3.2 DeepSeek | 1,370 | 49.1% | 39.9% | 54.1% | 2779.78 | 8266-8584-184 | Eval examples Two different benchmark tasks Price-estimation rows are scored against actual quote outcomes. Brokerage/agent task rows are scored against reference answers and judged separately, so their leaderboard should be read as answer-quality performance rather than premium-estimation performance. Price-estimation examples - Estimate the annual premium and uncertainty range for an anonymized $1M California umbrella quote from a specific carrier. - Given state, carrier, coverage limit, and anonymized risk features, return calibrated P10/P50/P90 premium estimates. - Predict a quote range that contains the actual annualized premium without making the interval unnecessarily wide. Brokerage/agent task examples - A household has a listed underwriting profile. Are they likely to be eligible with a specific umbrella carrier? - In Texas, how much more does moving from $1M to $2M of umbrella coverage typically cost with a named carrier? - A customer asks about coverage requirements or eligibility constraints. What should an assistant say, using the benchmark reference answer? Methodology Domain-specific, aggregate-only benchmarking General AI benchmarks rarely measure whether a model can reason through the details that matter in insurance: liability limits, carrier constraints, premium ranges, eligibility rules, and uncertainty. This benchmark focuses on those workflows. Price scoring Quote rows compare each model's estimated annual premium and range against the actual quote outcome. Coverage rewards calibrated ranges; MAPE rewards accurate point estimates; Winkler loss penalizes ranges that miss the actual quote or are too wide. Brokerage/agent task scoring Brokerage/agent task rows compare model answers to benchmark reference answers with AI judging. The public task view reports aggregate judge scores, pairwise Elo, and win rate only. How Elo works For each shared scenario or question, every pair of model outputs is compared. Better outputs win the local battle, ties split credit, and Elo updates model strength within that benchmark section. Data protection Public results are aggregate-only. The page does not expose raw prompts, row identifiers, model responses, judge reasoning, or any operational eval artifacts. The evals use anonymized data on no-retention and no-logging platforms, so customer data is never exposed even to model providers.