{"slug": "predictive-database-benchmarks-vs-rf-automl-elastic-etc-up-to-10m-scale", "title": "Predictive database benchmarks vs. RF, AutoML, Elastic etc., up to 10M scale", "summary": "Aito published beta benchmarks claiming its predictive database matches or beats tuned ML baselines on high-cardinality tasks with zero training time, including 68.5% accuracy on invoice routing acceptor versus FLAML's 59.5% and 80.0% on expense categorization versus LightGBM's 76.1%. On a 10M-row linked-invoice database, Aito v2 used 597 MB of JVM heap after GC (3.8× leaner than v1), 3,258 MB on disk (2.3× smaller), 160 ms mean predict latency, 219 ms cold start (~14× faster), and 21,716 rows/s ingest throughput. Aito acknowledged it does not compete on raw query speed, stating specialized engines such as Elasticsearch will be faster for plain indexed search and filtering.", "body_md": "# Benchmarks (Beta)\n\nTwo questions decide whether Aito fits a problem: **is it accurate enough?**\nand **is it fast enough?** This section answers both with reproducible\nnumbers — accuracy against tuned ML baselines and public datasets, and\nthroughput/latency against the kind of engine you'd otherwise reach for.\n\nWe try to be honest about where Aito wins and where it doesn't.\n\n## Results at a glance\n\nEvery figure below is generated from a committed benchmark run — follow any row to the page that earns it.\n\n**Accuracy, against the best tuned baseline on the same data:**\n\n| Task | Aito | Best baseline | Training time | \n|---|---|---|---|\n| [Invoice routing — **acceptor**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (link target) | **68.5%** (v2) | FLAML 59.5% | **0 s** vs 398 s | \n| [Invoice routing — **GL code**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (32 classes) | 68.5% (v2) | FLAML 67.0% | **0 s** vs 398 s | \n| [Invoice routing — **processor**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (high-cardinality) | 19.5% (v2) | FLAML 17.0% | **0 s** vs 398 s | \n| [Expense category](https://aito.ai/docs/api/v2/benchmarks/expense-categorization#accuracy) (48 classes) | **80.0%** (v1) | LightGBM 76.1% | **0 s** vs 221 s | \n| [Invoice line → SKU matching](https://aito.ai/docs/api/v2/benchmarks/line-matching#accuracy) (live catalogue) | **90.4%** (v2) | BM25 given the same history 89.2% | **0 s** vs an index build | \n| [Bank intent](https://aito.ai/docs/api/v2/benchmarks/banking77#result) (Banking77, 77 classes) | 82.6% | fine-tuned transformers reach the low 90s | **0 s** vs a fine-tuning run | \n| [**At 1 000 training rows**](https://aito.ai/docs/api/v2/benchmarks/scaling#scale) (expense, data-efficiency) | **64%** (v2) | RF 57% · LightGBM 56% | **0 s** at every scale | \n\n**What it costs to run**, on a 10M-row linked-invoice database:\n\n|  | Aito v2 | vs the v1 engine | \n|---|---|---|\n| [JVM heap, after GC](https://aito.ai/docs/api/v2/benchmarks/footprint#table) | **597 MB** | 3.8× leaner | \n| [On disk](https://aito.ai/docs/api/v2/benchmarks/footprint#table) | 3,258 MB | 2.3× smaller | \n| [Predict latency](https://aito.ai/docs/api/v2/benchmarks/performance#predict) (GL code, mean) | 160 ms | level here; slower on link targets | \n| [Cold start](https://aito.ai/docs/api/v2/benchmarks/performance#warmup) (GL code, first query) | 219 ms | ~14× faster | \n| [Ingest throughput](https://aito.ai/docs/api/v2/benchmarks/performance#writes) | 21,716 rows/s | queryable immediately, no reindex | \n\n## Where Aito competes — and where it doesn't\n\nAito is a **predictive database**: the same `from`/` where` you filter with is\nthe evidence for `predict` / `recommend` / `relate`, with **no training step**\nand an explanation (`$why`) behind every result. That framing sets what the\nbenchmarks should — and shouldn't — measure.\n\n- **Prediction quality is the point.** Against tuned gradient-boosting and\nAutoML pipelines (Random Forest, LightGBM, FLAML) and against an LLM+RAG\nsetup, Aito is**strongest exactly on the hard, high-cardinality, cross-table\ntargets** a bag-of-features model struggles with — on the current baselines it\ntops the invoice-routing and expense suites, while purpose-built fine-tuned\nNLP models still lead on Banking77 — and it gets there with**zero training\ntime** and full explainability. That's the differentiator; that's what these\nsuites measure first.\n- **Raw query speed is not our competition area — and that's fine.** For plain\nindexed search and filtering, a specialized engine (Elasticsearch, a native\nC++ columnar store) will be faster, and we don't try to beat them at their\nown game. Aito's bar on that axis is**\"not too slow\"** : fast enough to serve\ninteractive queries while carrying the prediction layer those engines don't\nhave. The[speed](https://aito.ai/docs/api/v2/benchmarks/performance) page states this plainly, with\nnumbers.\n\nThe trade you're evaluating is: a single system that predicts, recommends, relates and searches over your live data with no model-training pipeline — versus stitching a search engine to a separately-trained, separately-served ML stack. These benchmarks are here to show that trade is a good one on quality, and acceptable on speed.\n\n## Accuracy suites\n\nEach suite is a real task with a held-out test set and published baselines, run as a booktest so the numbers are reproducible. We're starting with a focused set and will extend it with more common ML benchmarks over time.\n\n- \n**[Automated GL coding / invoice routing](https://aito.ai/docs/api/v2/benchmarks/gl-coding)** —\npredict processor, acceptor and GL code from invoice text with**both the v1\nand v2 engines** , vs Random Forest / LightGBM / FLAML AutoML. On the current\nbaseline (after the engine's scoring redesign)**Aito v2 leads top-1 on all\nthree targets** with**0 s training** — decisively on acceptor (v2\n68.5% vs FLAML\n59.5%), within\nthe n=200 noise band on GL code and processor, so read those two as parity\nwith a tuned AutoML search.**FLAML remains the strongest baseline.** The\nrank story backs it up: v2's link priors put the true acceptor at mean rank\n1.8 (v1\n2.2) and cut the processor\nrank ~3× vs v1.\n- \n**[Expense categorization](https://aito.ai/docs/api/v2/benchmarks/expense-categorization)** — predict\nan invoice's expense category (48 classes) from vendor, tax rate, amount and\nline text, vs Random Forest, LightGBM and FLAML AutoML (an LLM + RAG baseline is\na pending slot). The headline:**Aito v1 tops the top-1 table** (80.0% vs\nLightGBM's\n76.1%) — though\nthe top is a tight cluster inside a test-set CI — and Aito is**well-calibrated out of the box** , with**0 s training** and a`$why` behind\nevery prediction, where the tree defaults are over- or under-confident.\nReports accuracy,**calibration** (ECE / Brier / gap), and training time.\n- \n**[Invoice line → product matching](https://aito.ai/docs/api/v2/benchmarks/line-matching)** — match\nan incoming invoice line to a SKU in a live catalogue, as a cross-table link\nprediction, vs BM25 in three strengths. Against a catalogue-only lexical\nindex — what most matching systems ship —\n90.4% vs\n64.4%; against BM25**given the same billing history** the margin narrows to\n89.2%, which is the honest\ncomparison and the one the page leads with. The differentiator is that the\nhistory layer*is* the database — corrections are queryable on insert, with\nno reindex. Also states the open defect (popular SKUs beating thin correct\nones) and carries a partner-facing implementation section.\n- \n**[Intent classification (Banking77)](https://aito.ai/docs/api/v2/benchmarks/banking77)** — 77-way\nfine-grained intent on 13k banking queries. 82.6%\ntop-1 with**zero training** , shown in honest context against fine-tuned\ntransformer sentence-encoders (which reach the low-90s and win on raw accuracy).\n- \n**[Smart search & recommendations](https://aito.ai/docs/api/v2/benchmarks/smart-search)** — rank the\nproducts a shopper was shown so the one they bought comes first, from the query,\ntheir profile and the products' attributes, on a generated shop log with a\nknown answer key.`recommend` with`basedOn` (0.534 nDCG@10) beats\nBM25 given the same purchase history\n(0.468), mostly on\nwordings the catalogue does not contain; text search still wins on brand\nqueries. Also the For You shelf, with no query.\n\n**On the roadmap:** text classification (Reuters, BBC News) and retrieval / RAG\n(BEIR SciFact, where Aito's inverted index feeds retrieval-augmented generation).\n\n## Speed & footprint\n\n- **[Query, predict & write speed](https://aito.ai/docs/api/v2/benchmarks/performance)** — query\nlatency against Elasticsearch on identical data, the v1-vs-v2 prediction\nhead-to-head at 10M rows, and ingest throughput — with an honest read on how\nAito compares to a specialized search engine.\n- **[Memory & disk footprint](https://aito.ai/docs/api/v2/benchmarks/footprint)** — what the same 10M-row\ndatabase costs to hold:**597 MB\nof heap** , ~3.8× leaner than the v1 engine, and why the heap doesn't grow with\nthe corpus.\n- **[Queries under writes](https://aito.ai/docs/api/v2/benchmarks/writes)** — what a prediction costs on\na database that is being written to the whole time the queries run:\n853 ms at p50 under a\ncontinuous write load, and every answer byte-identical to a cold recompute.\n\n## How it scales\n\nTwo different questions get asked with the same word — *how much data do I\nneed?* and *how big can my database get?* — and [one page](https://aito.ai/docs/api/v2/benchmarks/scaling)\nnow answers both on **one corpus** (expense categorization) with **one fixed\nhold-out**, so a number on either axis is about the same data.\n\n- **Training size, 1 → 1 000 000 rows** on a corpus that grows by adding\ntenants, every method on the same rows: Aito leads from N=10 through\nN=10 000 with no training step (at N=1 000, v2 is\n64% against a field\nat 57%); the field is\nlevel at 100 000; at 1 000 000 LightGBM is ahead by a point and a half after a\n775 s\ntraining run, with Aito v1 level with Random Forest. Published as the\ncrossover it is.\n- **Database size, to 10 000 000 rows** : predict latency, the cost of coding a\nwhole invoice in one batched request, and heap / mapped / disk for both\nengines — v2 at 10M holds the database in\n1,609 MB\nof heap against v1's\n3,997 MB.\n- The finer engine-to-engine curve on the linked invoice corpus, 1k → 10M, is\nthe [capacity ladder](https://aito.ai/docs/api/v2/benchmarks/scaling-capacity) .\n\n## How the numbers are produced\n\nEvery figure comes from a run, not a slide. Accuracy suites are booktests over\nfixed test sets; performance figures come from the `InvoicePerf` /\n`FlatInvoicePerf` harnesses. The specific numbers on these pages — including the\nones cited above — are **generated at documentation-build time**: a generator\nreads each run's committed `metrics.json` and interpolates it into the page, so\nthey update themselves on re-baseline and a stale hand-typed digit can't slip\nthrough. The build fails loudly on any unresolved metric token.\n\n**Note:** Benchmarks are **beta** and use a single random seed. The expense\nsuites score 2,000 held-out rows, which puts a 95% interval of\nabout ±2 points on a cell, and every \"ahead of\" claim on those pages comes\nfrom an exact McNemar test on the rows where two methods disagree — where the\ntest does not separate them, the page says they are tied. **Invoice routing\nstill scores 200 rows** (±~5 points), so read its cells as point estimates and\nread the direction rather than the digits.", "url": "https://wpnews.pro/news/predictive-database-benchmarks-vs-rf-automl-elastic-etc-up-to-10m-scale", "canonical_source": "https://aito.ai/docs/api/v2/benchmarks/", "published_at": "2026-09-16 14:48:56+00:00", "updated_at": "2026-09-16 15:15:53.432904+00:00", "lang": "en", "topics": ["ai-products", "machine-learning", "ai-tools", "ai-infrastructure"], "entities": ["Aito", "FLAML", "LightGBM", "Random Forest", "Elasticsearch", "Banking77", "BM25"], "alternates": {"html": "https://wpnews.pro/news/predictive-database-benchmarks-vs-rf-automl-elastic-etc-up-to-10m-scale", "markdown": "https://wpnews.pro/news/predictive-database-benchmarks-vs-rf-automl-elastic-etc-up-to-10m-scale.md", "text": "https://wpnews.pro/news/predictive-database-benchmarks-vs-rf-automl-elastic-etc-up-to-10m-scale.txt", "jsonld": "https://wpnews.pro/news/predictive-database-benchmarks-vs-rf-automl-elastic-etc-up-to-10m-scale.jsonld"}}