Which AI models understand insurance work?

Coverage Cat AI Insurance Benchmark

We evaluate frontier models on two separate insurance tasks: price estimation for anonymized umbrella quote rows, and brokerage/agent task reasoning against benchmark reference answers for underwriting, eligibility, and coverage questions.

Leaderboard

Price-estimation performance #

Price-estimation results compare predicted premiums and uncertainty ranges against actual quote outcomes. These metrics are separate from the brokerage/agent task leaderboard.

Grok 4.3 1,687 Elo

ChatGPT 5.5 69.1% coverage

19,873 2,839 unique quote cases

Elo Rating

Pairwise model strength on the same insurance quoting cases. Higher scores mean a model more often beat comparable models on calibrated quote accuracy.

Win Rate

The share of pairwise quote battles won, with ties counted as half a win. Higher is better.

Quote Coverage

How often the actual annualized premium landed inside the model's predicted quote range. Higher is better.

Quote MAPE

Mean absolute percentage error for each model's point premium estimate. Lower is better.

Winkler Loss

A calibration penalty for quote ranges that miss the actual premium or are unnecessarily wide. Lower is better.

Model comparison

Price-estimation ranking #

Eval examples

Two different benchmark tasks #

Price-estimation rows are scored against actual quote outcomes. Brokerage/agent task rows are scored against reference answers and judged separately, so their leaderboard should be read as answer-quality performance rather than premium-estimation performance.

Price-estimation examples

• Estimate the annual premium and uncertainty range for an anonymized $1M California umbrella quote from a specific carrier.
• Given state, carrier, coverage limit, and anonymized risk features, return calibrated P10/P50/P90 premium estimates.
• Predict a quote range that contains the actual annualized premium without making the interval unnecessarily wide.

Brokerage/agent task examples

• A household has a listed underwriting profile. Are they likely to be eligible with a specific umbrella carrier?
• In Texas, how much more does moving from $1M to $2M of umbrella coverage typically cost with a named carrier?
• A customer asks about coverage requirements or eligibility constraints. What should an assistant say, using the benchmark reference answer?

Methodology

Domain-specific, aggregate-only benchmarking #

General AI benchmarks rarely measure whether a model can reason through the details that matter in insurance: liability limits, carrier constraints, premium ranges, eligibility rules, and uncertainty. This benchmark focuses on those workflows.

Price scoring

Quote rows compare each model's estimated annual premium and range against the actual quote outcome. Coverage rewards calibrated ranges; MAPE rewards accurate point estimates; Winkler loss penalizes ranges that miss the actual quote or are too wide.

Brokerage/agent task scoring

Brokerage/agent task rows compare model answers to benchmark reference answers with AI judging. The public task view reports aggregate judge scores, pairwise Elo, and win rate only.

How Elo works

For each shared scenario or question, every pair of model outputs is compared. Better outputs win the local battle, ties split credit, and Elo updates model strength within that benchmark section.

Data protection

Public results are aggregate-only. The page does not expose raw prompts, row identifiers, model responses, judge reasoning, or any operational eval artifacts. The evals use anonymized data on no-retention and no-logging platforms, so customer data is never exposed even to model providers.