{"slug": "i-benchmarked-ai-decision-apis-against-xgboost-for-fraud-detection-xgboost-won-a", "title": "I benchmarked AI decision APIs against XGBoost for fraud detection. XGBoost won, by a lot.", "summary": "A developer built an MIT-licensed open benchmark, fraud-decision-lab, comparing new \"decision model\" APIs from TypeSafe, OpenAI and Cloudflare against a standard XGBoost fraud model. On real anonymised ULB card data, XGBoost caught 82.7% of fraud in its top 1% of cases versus 46.2% for the best AI approach, while on synthetic data XGBoost surfaced 84 frauds in the first 100 reviews against 16–51 for the decision APIs. The decision APIs were fast and cheap (175–324 ms median latency, $0.016–$0.10 per 1,000 synthetic transactions) but ran zero-shot, a limitation the developer notes few-shot prompting or fine-tuning could change.", "body_md": "Between September and October, TypeSafe, OpenAI and Cloudflare all shipped \"decision model\" APIs: you send a case, ask one fixed question, and get a score back. They're being pitched for fraud triage, so I wanted to know how they compare with the boring XGBoost model most fraud teams already run. I built a small open benchmark to find out. It's MIT-licensed and runs offline in about five seconds.\n\nSame transactions, same fields, ranked by each model's score:\n\nThe metric is the review queue. Fraud analysts can only look at the top slice of cases, so the question is how much fraud lands in the first 100 reviews, or the top 1% and 5%. Results come with bootstrap ranges.\n\n**Synthetic data, 10,000 transactions, 310 frauds.** Fraud found in the first 100 reviews:\n\n| Approach | Frauds in top 100 | \n|---|---|\n| XGBoost | 84 | \n| Rules checklist | 74 | \n| General LLM | 57 | \n| Decision APIs | 16–51 | \n\nThe rules look good here partly because the synthetic labels share risk factors with them, so I wouldn't read much into that row.\n\n**Real anonymised card data (ULB, 52 frauds).** XGBoost caught **82.7%** of fraud in its top 1%. The best AI approach caught **46.2%**.\n\n**Speed and cost.** The APIs are fast and cheap: 175–324 ms median latency and $0.016–$0.10 per 1,000 synthetic transactions.\n\nXGBoost learned from labelled data. The AI approaches ran zero-shot, with no examples. That's the setup you get when a vendor says \"no training needed\", and it's only one way to use an LLM in fraud work. Few-shot prompts, richer context or fine-tuning could change the picture, and I'd like to see someone try.\n\n**Ties distort rankings.** One API returned many identical top scores. Changing the tie-break rule moved its top-100 count from 20 to 22, so the benchmark now uses a seeded tie-break and reports it.\n\n**Make live calls opt-in.** The demo runs offline on recorded outputs. Hitting real APIs needs `--allow-external-requests` and a `--max-cost-usd` estimate, so nobody runs up a bill by accident.\n\n**Use one schema for every model.** Every model gets the same fields. The unavoidable differences (rounding, each provider's request format) are documented rather than hidden.\n\n**Compute cost from recorded usage.** Costs come from recorded token usage and a dated price snapshot, because pricing pages change faster than benchmarks get rerun.\n\n```\nbash\ngit clone https://github.com/SmartMobileHouse/fraud-decision-lab\ncd fraud-decision-lab && uv sync --frozen && make demo\n\nIt's a v1.0 snapshot on two datasets, and providers keep updating their models, so treat it as a starting point. Full write-up with charts: https://smartmobilehouse.com/research/fraud-decision-lab/\n\nHas anyone here tried these decision APIs with few-shot examples or on real production features? I'd happily add a better configuration and publish the new numbers next to the old ones.\n```\n\n", "url": "https://wpnews.pro/news/i-benchmarked-ai-decision-apis-against-xgboost-for-fraud-detection-xgboost-won-a", "canonical_source": "https://dev.to/jpietrus/i-benchmarked-ai-decision-apis-against-xgboost-for-fraud-detection-xgboost-won-by-a-lot-3jdh", "published_at": "2026-11-14 17:41:33+00:00", "updated_at": "2026-11-14 17:51:05.013866+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools", "ai-products", "ai-agents"], "entities": ["XGBoost", "TypeSafe", "OpenAI", "Cloudflare", "fraud-decision-lab", "SmartMobileHouse"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-ai-decision-apis-against-xgboost-for-fraud-detection-xgboost-won-a", "markdown": "https://wpnews.pro/news/i-benchmarked-ai-decision-apis-against-xgboost-for-fraud-detection-xgboost-won-a.md", "text": "https://wpnews.pro/news/i-benchmarked-ai-decision-apis-against-xgboost-for-fraud-detection-xgboost-won-a.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-ai-decision-apis-against-xgboost-for-fraud-detection-xgboost-won-a.jsonld"}}