Between September and October, TypeSafe, OpenAI and Cloudflare all shipped "decision model" APIs: you send a case, ask one fixed question, and get a score back. They're being pitched for fraud triage, so I wanted to know how they compare with the boring XGBoost model most fraud teams already run. I built a small open benchmark to find out. It's MIT-licensed and runs offline in about five seconds.
Same transactions, same fields, ranked by each model's score:
The metric is the review queue. Fraud analysts can only look at the top slice of cases, so the question is how much fraud lands in the first 100 reviews, or the top 1% and 5%. Results come with bootstrap ranges.
Synthetic data, 10,000 transactions, 310 frauds. Fraud found in the first 100 reviews:
| Approach | Frauds in top 100 |
|---|---|
| XGBoost | 84 |
| Rules checklist | 74 |
| General LLM | 57 |
| Decision APIs | 16–51 |
The rules look good here partly because the synthetic labels share risk factors with them, so I wouldn't read much into that row.
Real anonymised card data (ULB, 52 frauds). XGBoost caught 82.7% of fraud in its top 1%. The best AI approach caught 46.2%.
Speed and cost. The APIs are fast and cheap: 175–324 ms median latency and $0.016–$0.10 per 1,000 synthetic transactions.
XGBoost learned from labelled data. The AI approaches ran zero-shot, with no examples. That's the setup you get when a vendor says "no training needed", and it's only one way to use an LLM in fraud work. Few-shot prompts, richer context or fine-tuning could change the picture, and I'd like to see someone try.
Ties distort rankings. One API returned many identical top scores. Changing the tie-break rule moved its top-100 count from 20 to 22, so the benchmark now uses a seeded tie-break and reports it.
Make live calls opt-in. The demo runs offline on recorded outputs. Hitting real APIs needs --allow-external-requests and a --max-cost-usd estimate, so nobody runs up a bill by accident.
Use one schema for every model. Every model gets the same fields. The unavoidable differences (rounding, each provider's request format) are documented rather than hidden.
Compute cost from recorded usage. Costs come from recorded token usage and a dated price snapshot, because pricing pages change faster than benchmarks get rerun.
bash
git clone https://github.com/SmartMobileHouse/fraud-decision-lab
cd fraud-decision-lab && uv sync --frozen && make demo
It's a v1.0 snapshot on two datasets, and providers keep updating their models, so treat it as a starting point. Full write-up with charts: https://smartmobilehouse.com/research/fraud-decision-lab/
Has anyone here tried these decision APIs with few-shot examples or on real production features? I'd happily add a better configuration and publish the new numbers next to the old ones.