# Predictive database benchmarks vs. RF, AutoML, Elastic etc., up to 10M scale > Source: > Published: 2026-09-16 14:48:56+00:00 # Benchmarks (Beta) Two questions decide whether Aito fits a problem: **is it accurate enough?** and **is it fast enough?** This section answers both with reproducible numbers — accuracy against tuned ML baselines and public datasets, and throughput/latency against the kind of engine you'd otherwise reach for. We try to be honest about where Aito wins and where it doesn't. ## Results at a glance Every figure below is generated from a committed benchmark run — follow any row to the page that earns it. **Accuracy, against the best tuned baseline on the same data:** | Task | Aito | Best baseline | Training time | |---|---|---|---| | [Invoice routing — **acceptor**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (link target) | **68.5%** (v2) | FLAML 59.5% | **0 s** vs 398 s | | [Invoice routing — **GL code**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (32 classes) | 68.5% (v2) | FLAML 67.0% | **0 s** vs 398 s | | [Invoice routing — **processor**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (high-cardinality) | 19.5% (v2) | FLAML 17.0% | **0 s** vs 398 s | | [Expense category](https://aito.ai/docs/api/v2/benchmarks/expense-categorization#accuracy) (48 classes) | **80.0%** (v1) | LightGBM 76.1% | **0 s** vs 221 s | | [Invoice line → SKU matching](https://aito.ai/docs/api/v2/benchmarks/line-matching#accuracy) (live catalogue) | **90.4%** (v2) | BM25 given the same history 89.2% | **0 s** vs an index build | | [Bank intent](https://aito.ai/docs/api/v2/benchmarks/banking77#result) (Banking77, 77 classes) | 82.6% | fine-tuned transformers reach the low 90s | **0 s** vs a fine-tuning run | | [**At 1 000 training rows**](https://aito.ai/docs/api/v2/benchmarks/scaling#scale) (expense, data-efficiency) | **64%** (v2) | RF 57% · LightGBM 56% | **0 s** at every scale | **What it costs to run**, on a 10M-row linked-invoice database: | | Aito v2 | vs the v1 engine | |---|---|---| | [JVM heap, after GC](https://aito.ai/docs/api/v2/benchmarks/footprint#table) | **597 MB** | 3.8× leaner | | [On disk](https://aito.ai/docs/api/v2/benchmarks/footprint#table) | 3,258 MB | 2.3× smaller | | [Predict latency](https://aito.ai/docs/api/v2/benchmarks/performance#predict) (GL code, mean) | 160 ms | level here; slower on link targets | | [Cold start](https://aito.ai/docs/api/v2/benchmarks/performance#warmup) (GL code, first query) | 219 ms | ~14× faster | | [Ingest throughput](https://aito.ai/docs/api/v2/benchmarks/performance#writes) | 21,716 rows/s | queryable immediately, no reindex | ## Where Aito competes — and where it doesn't Aito is a **predictive database**: the same `from`/` where` you filter with is the evidence for `predict` / `recommend` / `relate`, with **no training step** and an explanation (`$why`) behind every result. That framing sets what the benchmarks should — and shouldn't — measure. - **Prediction quality is the point.** Against tuned gradient-boosting and AutoML pipelines (Random Forest, LightGBM, FLAML) and against an LLM+RAG setup, Aito is**strongest exactly on the hard, high-cardinality, cross-table targets** a bag-of-features model struggles with — on the current baselines it tops the invoice-routing and expense suites, while purpose-built fine-tuned NLP models still lead on Banking77 — and it gets there with**zero training time** and full explainability. That's the differentiator; that's what these suites measure first. - **Raw query speed is not our competition area — and that's fine.** For plain indexed search and filtering, a specialized engine (Elasticsearch, a native C++ columnar store) will be faster, and we don't try to beat them at their own game. Aito's bar on that axis is**"not too slow"** : fast enough to serve interactive queries while carrying the prediction layer those engines don't have. The[speed](https://aito.ai/docs/api/v2/benchmarks/performance) page states this plainly, with numbers. The trade you're evaluating is: a single system that predicts, recommends, relates and searches over your live data with no model-training pipeline — versus stitching a search engine to a separately-trained, separately-served ML stack. These benchmarks are here to show that trade is a good one on quality, and acceptable on speed. ## Accuracy suites Each suite is a real task with a held-out test set and published baselines, run as a booktest so the numbers are reproducible. We're starting with a focused set and will extend it with more common ML benchmarks over time. - **[Automated GL coding / invoice routing](https://aito.ai/docs/api/v2/benchmarks/gl-coding)** — predict processor, acceptor and GL code from invoice text with**both the v1 and v2 engines** , vs Random Forest / LightGBM / FLAML AutoML. On the current baseline (after the engine's scoring redesign)**Aito v2 leads top-1 on all three targets** with**0 s training** — decisively on acceptor (v2 68.5% vs FLAML 59.5%), within the n=200 noise band on GL code and processor, so read those two as parity with a tuned AutoML search.**FLAML remains the strongest baseline.** The rank story backs it up: v2's link priors put the true acceptor at mean rank 1.8 (v1 2.2) and cut the processor rank ~3× vs v1. - **[Expense categorization](https://aito.ai/docs/api/v2/benchmarks/expense-categorization)** — predict an invoice's expense category (48 classes) from vendor, tax rate, amount and line text, vs Random Forest, LightGBM and FLAML AutoML (an LLM + RAG baseline is a pending slot). The headline:**Aito v1 tops the top-1 table** (80.0% vs LightGBM's 76.1%) — though the top is a tight cluster inside a test-set CI — and Aito is**well-calibrated out of the box** , with**0 s training** and a`$why` behind every prediction, where the tree defaults are over- or under-confident. Reports accuracy,**calibration** (ECE / Brier / gap), and training time. - **[Invoice line → product matching](https://aito.ai/docs/api/v2/benchmarks/line-matching)** — match an incoming invoice line to a SKU in a live catalogue, as a cross-table link prediction, vs BM25 in three strengths. Against a catalogue-only lexical index — what most matching systems ship — 90.4% vs 64.4%; against BM25**given the same billing history** the margin narrows to 89.2%, which is the honest comparison and the one the page leads with. The differentiator is that the history layer*is* the database — corrections are queryable on insert, with no reindex. Also states the open defect (popular SKUs beating thin correct ones) and carries a partner-facing implementation section. - **[Intent classification (Banking77)](https://aito.ai/docs/api/v2/benchmarks/banking77)** — 77-way fine-grained intent on 13k banking queries. 82.6% top-1 with**zero training** , shown in honest context against fine-tuned transformer sentence-encoders (which reach the low-90s and win on raw accuracy). - **[Smart search & recommendations](https://aito.ai/docs/api/v2/benchmarks/smart-search)** — rank the products a shopper was shown so the one they bought comes first, from the query, their profile and the products' attributes, on a generated shop log with a known answer key.`recommend` with`basedOn` (0.534 nDCG@10) beats BM25 given the same purchase history (0.468), mostly on wordings the catalogue does not contain; text search still wins on brand queries. Also the For You shelf, with no query. **On the roadmap:** text classification (Reuters, BBC News) and retrieval / RAG (BEIR SciFact, where Aito's inverted index feeds retrieval-augmented generation). ## Speed & footprint - **[Query, predict & write speed](https://aito.ai/docs/api/v2/benchmarks/performance)** — query latency against Elasticsearch on identical data, the v1-vs-v2 prediction head-to-head at 10M rows, and ingest throughput — with an honest read on how Aito compares to a specialized search engine. - **[Memory & disk footprint](https://aito.ai/docs/api/v2/benchmarks/footprint)** — what the same 10M-row database costs to hold:**597 MB of heap** , ~3.8× leaner than the v1 engine, and why the heap doesn't grow with the corpus. - **[Queries under writes](https://aito.ai/docs/api/v2/benchmarks/writes)** — what a prediction costs on a database that is being written to the whole time the queries run: 853 ms at p50 under a continuous write load, and every answer byte-identical to a cold recompute. ## How it scales Two different questions get asked with the same word — *how much data do I need?* and *how big can my database get?* — and [one page](https://aito.ai/docs/api/v2/benchmarks/scaling) now answers both on **one corpus** (expense categorization) with **one fixed hold-out**, so a number on either axis is about the same data. - **Training size, 1 → 1 000 000 rows** on a corpus that grows by adding tenants, every method on the same rows: Aito leads from N=10 through N=10 000 with no training step (at N=1 000, v2 is 64% against a field at 57%); the field is level at 100 000; at 1 000 000 LightGBM is ahead by a point and a half after a 775 s training run, with Aito v1 level with Random Forest. Published as the crossover it is. - **Database size, to 10 000 000 rows** : predict latency, the cost of coding a whole invoice in one batched request, and heap / mapped / disk for both engines — v2 at 10M holds the database in 1,609 MB of heap against v1's 3,997 MB. - The finer engine-to-engine curve on the linked invoice corpus, 1k → 10M, is the [capacity ladder](https://aito.ai/docs/api/v2/benchmarks/scaling-capacity) . ## How the numbers are produced Every figure comes from a run, not a slide. Accuracy suites are booktests over fixed test sets; performance figures come from the `InvoicePerf` / `FlatInvoicePerf` harnesses. The specific numbers on these pages — including the ones cited above — are **generated at documentation-build time**: a generator reads each run's committed `metrics.json` and interpolates it into the page, so they update themselves on re-baseline and a stale hand-typed digit can't slip through. The build fails loudly on any unresolved metric token. **Note:** Benchmarks are **beta** and use a single random seed. The expense suites score 2,000 held-out rows, which puts a 95% interval of about ±2 points on a cell, and every "ahead of" claim on those pages comes from an exact McNemar test on the rows where two methods disagree — where the test does not separate them, the page says they are tied. **Invoice routing still scores 200 rows** (±~5 points), so read its cells as point estimates and read the direction rather than the digits.