Predictive database benchmarks vs. RF, AutoML, Elastic etc., up to 10M scale

Aito published beta benchmarks claiming its predictive database matches or beats tuned ML baselines on high-cardinality tasks with zero training time, including 68.5% accuracy on invoice routing acceptor versus FLAML's 59.5% and 80.0% on expense categorization versus LightGBM's 76.1%. On a 10M-row linked-invoice database, Aito v2 used 597 MB of JVM heap after GC (3.8× leaner than v1), 3,258 MB on disk (2.3× smaller), 160 ms mean predict latency, 219 ms cold start (~14× faster), and 21,716 rows/s ingest throughput. Aito acknowledged it does not compete on raw query speed, stating specialized engines such as Elasticsearch will be faster for plain indexed search and filtering.

Benchmarks Beta Two questions decide whether Aito fits a problem: is it accurate enough? and is it fast enough? This section answers both with reproducible numbers — accuracy against tuned ML baselines and public datasets, and throughput/latency against the kind of engine you'd otherwise reach for. We try to be honest about where Aito wins and where it doesn't. Results at a glance Every figure below is generated from a committed benchmark run — follow any row to the page that earns it. Accuracy, against the best tuned baseline on the same data: | Task | Aito | Best baseline | Training time | |---|---|---|---| | Invoice routing — acceptor https://aito.ai/docs/api/v2/benchmarks/gl-coding accuracy link target | 68.5% v2 | FLAML 59.5% | 0 s vs 398 s | | Invoice routing — GL code https://aito.ai/docs/api/v2/benchmarks/gl-coding accuracy 32 classes | 68.5% v2 | FLAML 67.0% | 0 s vs 398 s | | Invoice routing — processor https://aito.ai/docs/api/v2/benchmarks/gl-coding accuracy high-cardinality | 19.5% v2 | FLAML 17.0% | 0 s vs 398 s | | Expense category https://aito.ai/docs/api/v2/benchmarks/expense-categorization accuracy 48 classes | 80.0% v1 | LightGBM 76.1% | 0 s vs 221 s | | Invoice line → SKU matching https://aito.ai/docs/api/v2/benchmarks/line-matching accuracy live catalogue | 90.4% v2 | BM25 given the same history 89.2% | 0 s vs an index build | | Bank intent https://aito.ai/docs/api/v2/benchmarks/banking77 result Banking77, 77 classes | 82.6% | fine-tuned transformers reach the low 90s | 0 s vs a fine-tuning run | | At 1 000 training rows https://aito.ai/docs/api/v2/benchmarks/scaling scale expense, data-efficiency | 64% v2 | RF 57% · LightGBM 56% | 0 s at every scale | What it costs to run , on a 10M-row linked-invoice database: | | Aito v2 | vs the v1 engine | |---|---|---| | JVM heap, after GC https://aito.ai/docs/api/v2/benchmarks/footprint table | 597 MB | 3.8× leaner | | On disk https://aito.ai/docs/api/v2/benchmarks/footprint table | 3,258 MB | 2.3× smaller | | Predict latency https://aito.ai/docs/api/v2/benchmarks/performance predict GL code, mean | 160 ms | level here; slower on link targets | | Cold start https://aito.ai/docs/api/v2/benchmarks/performance warmup GL code, first query | 219 ms | ~14× faster | | Ingest throughput https://aito.ai/docs/api/v2/benchmarks/performance writes | 21,716 rows/s | queryable immediately, no reindex | Where Aito competes — and where it doesn't Aito is a predictive database : the same from / where you filter with is the evidence for predict / recommend / relate , with no training step and an explanation $why behind every result. That framing sets what the benchmarks should — and shouldn't — measure. - Prediction quality is the point. Against tuned gradient-boosting and AutoML pipelines Random Forest, LightGBM, FLAML and against an LLM+RAG setup, Aito is strongest exactly on the hard, high-cardinality, cross-table targets a bag-of-features model struggles with — on the current baselines it tops the invoice-routing and expense suites, while purpose-built fine-tuned NLP models still lead on Banking77 — and it gets there with zero training time and full explainability. That's the differentiator; that's what these suites measure first. - Raw query speed is not our competition area — and that's fine. For plain indexed search and filtering, a specialized engine Elasticsearch, a native C++ columnar store will be faster, and we don't try to beat them at their own game. Aito's bar on that axis is "not too slow" : fast enough to serve interactive queries while carrying the prediction layer those engines don't have. The speed https://aito.ai/docs/api/v2/benchmarks/performance page states this plainly, with numbers. The trade you're evaluating is: a single system that predicts, recommends, relates and searches over your live data with no model-training pipeline — versus stitching a search engine to a separately-trained, separately-served ML stack. These benchmarks are here to show that trade is a good one on quality, and acceptable on speed. Accuracy suites Each suite is a real task with a held-out test set and published baselines, run as a booktest so the numbers are reproducible. We're starting with a focused set and will extend it with more common ML benchmarks over time. - Automated GL coding / invoice routing https://aito.ai/docs/api/v2/benchmarks/gl-coding — predict processor, acceptor and GL code from invoice text with both the v1 and v2 engines , vs Random Forest / LightGBM / FLAML AutoML. On the current baseline after the engine's scoring redesign Aito v2 leads top-1 on all three targets with 0 s training — decisively on acceptor v2 68.5% vs FLAML 59.5% , within the n=200 noise band on GL code and processor, so read those two as parity with a tuned AutoML search. FLAML remains the strongest baseline. The rank story backs it up: v2's link priors put the true acceptor at mean rank 1.8 v1 2.2 and cut the processor rank ~3× vs v1. - Expense categorization https://aito.ai/docs/api/v2/benchmarks/expense-categorization — predict an invoice's expense category 48 classes from vendor, tax rate, amount and line text, vs Random Forest, LightGBM and FLAML AutoML an LLM + RAG baseline is a pending slot . The headline: Aito v1 tops the top-1 table 80.0% vs LightGBM's 76.1% — though the top is a tight cluster inside a test-set CI — and Aito is well-calibrated out of the box , with 0 s training and a $why behind every prediction, where the tree defaults are over- or under-confident. Reports accuracy, calibration ECE / Brier / gap , and training time. - Invoice line → product matching https://aito.ai/docs/api/v2/benchmarks/line-matching — match an incoming invoice line to a SKU in a live catalogue, as a cross-table link prediction, vs BM25 in three strengths. Against a catalogue-only lexical index — what most matching systems ship — 90.4% vs 64.4%; against BM25 given the same billing history the margin narrows to 89.2%, which is the honest comparison and the one the page leads with. The differentiator is that the history layer is the database — corrections are queryable on insert, with no reindex. Also states the open defect popular SKUs beating thin correct ones and carries a partner-facing implementation section. - Intent classification Banking77 https://aito.ai/docs/api/v2/benchmarks/banking77 — 77-way fine-grained intent on 13k banking queries. 82.6% top-1 with zero training , shown in honest context against fine-tuned transformer sentence-encoders which reach the low-90s and win on raw accuracy . - Smart search & recommendations https://aito.ai/docs/api/v2/benchmarks/smart-search — rank the products a shopper was shown so the one they bought comes first, from the query, their profile and the products' attributes, on a generated shop log with a known answer key. recommend with basedOn 0.534 nDCG@10 beats BM25 given the same purchase history 0.468 , mostly on wordings the catalogue does not contain; text search still wins on brand queries. Also the For You shelf, with no query. On the roadmap: text classification Reuters, BBC News and retrieval / RAG BEIR SciFact, where Aito's inverted index feeds retrieval-augmented generation . Speed & footprint - Query, predict & write speed https://aito.ai/docs/api/v2/benchmarks/performance — query latency against Elasticsearch on identical data, the v1-vs-v2 prediction head-to-head at 10M rows, and ingest throughput — with an honest read on how Aito compares to a specialized search engine. - Memory & disk footprint https://aito.ai/docs/api/v2/benchmarks/footprint — what the same 10M-row database costs to hold: 597 MB of heap , ~3.8× leaner than the v1 engine, and why the heap doesn't grow with the corpus. - Queries under writes https://aito.ai/docs/api/v2/benchmarks/writes — what a prediction costs on a database that is being written to the whole time the queries run: 853 ms at p50 under a continuous write load, and every answer byte-identical to a cold recompute. How it scales Two different questions get asked with the same word — how much data do I need? and how big can my database get? — and one page https://aito.ai/docs/api/v2/benchmarks/scaling now answers both on one corpus expense categorization with one fixed hold-out , so a number on either axis is about the same data. - Training size, 1 → 1 000 000 rows on a corpus that grows by adding tenants, every method on the same rows: Aito leads from N=10 through N=10 000 with no training step at N=1 000, v2 is 64% against a field at 57% ; the field is level at 100 000; at 1 000 000 LightGBM is ahead by a point and a half after a 775 s training run, with Aito v1 level with Random Forest. Published as the crossover it is. - Database size, to 10 000 000 rows : predict latency, the cost of coding a whole invoice in one batched request, and heap / mapped / disk for both engines — v2 at 10M holds the database in 1,609 MB of heap against v1's 3,997 MB. - The finer engine-to-engine curve on the linked invoice corpus, 1k → 10M, is the capacity ladder https://aito.ai/docs/api/v2/benchmarks/scaling-capacity . How the numbers are produced Every figure comes from a run, not a slide. Accuracy suites are booktests over fixed test sets; performance figures come from the InvoicePerf / FlatInvoicePerf harnesses. The specific numbers on these pages — including the ones cited above — are generated at documentation-build time : a generator reads each run's committed metrics.json and interpolates it into the page, so they update themselves on re-baseline and a stale hand-typed digit can't slip through. The build fails loudly on any unresolved metric token. Note: Benchmarks are beta and use a single random seed. The expense suites score 2,000 held-out rows, which puts a 95% interval of about ±2 points on a cell, and every "ahead of" claim on those pages comes from an exact McNemar test on the rows where two methods disagree — where the test does not separate them, the page says they are tied. Invoice routing still scores 200 rows ±~5 points , so read its cells as point estimates and read the direction rather than the digits.