Two questions decide whether Aito fits a problem: is it accurate enough? and is it fast enough? This section answers both with reproducible numbers — accuracy against tuned ML baselines and public datasets, and throughput/latency against the kind of engine you'd otherwise reach for.
We try to be honest about where Aito wins and where it doesn't.
Results at a glance #
Every figure below is generated from a committed benchmark run — follow any row to the page that earns it.
Accuracy, against the best tuned baseline on the same data:
| Task | Aito | Best baseline | Training time |
|---|---|---|---|
| [Invoice routing — **acceptor**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (link target) | **68.5%** (v2) | FLAML 59.5% | **0 s** vs 398 s |
| [Invoice routing — **GL code**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (32 classes) | 68.5% (v2) | FLAML 67.0% | **0 s** vs 398 s |
| [Invoice routing — **processor**](https://aito.ai/docs/api/v2/benchmarks/gl-coding#accuracy) (high-cardinality) | 19.5% (v2) | FLAML 17.0% | **0 s** vs 398 s |
| [Expense category](https://aito.ai/docs/api/v2/benchmarks/expense-categorization#accuracy) (48 classes) | **80.0%** (v1) | LightGBM 76.1% | **0 s** vs 221 s |
| [Invoice line → SKU matching](https://aito.ai/docs/api/v2/benchmarks/line-matching#accuracy) (live catalogue) | **90.4%** (v2) | BM25 given the same history 89.2% | **0 s** vs an index build |
| Bank intent (Banking77, 77 classes) | 82.6% | fine-tuned transformers reach the low 90s | 0 s vs a fine-tuning run |
| At 1 000 training rows (expense, data-efficiency) | 64% (v2) | RF 57% · LightGBM 56% | 0 s at every scale | What it costs to run, on a 10M-row linked-invoice database:
| Aito v2 | vs the v1 engine | |
|---|---|---|
| JVM heap, after GC | 597 MB | 3.8× leaner |
| [On disk](https://aito.ai/docs/api/v2/benchmarks/footprint#table) | 3,258 MB | 2.3× smaller |
| [Predict latency](https://aito.ai/docs/api/v2/benchmarks/performance#predict) (GL code, mean) | 160 ms | level here; slower on link targets |
| [Cold start](https://aito.ai/docs/api/v2/benchmarks/performance#warmup) (GL code, first query) | 219 ms | ~14× faster |
| Ingest throughput | 21,716 rows/s | queryable immediately, no reindex |
Where Aito competes — and where it doesn't #
Aito is a predictive database: the same from/ where you filter with is
the evidence for predict / recommend / relate, with no training step
and an explanation ($why) behind every result. That framing sets what the
benchmarks should — and shouldn't — measure.
- Prediction quality is the point. Against tuned gradient-boosting and AutoML pipelines (Random Forest, LightGBM, FLAML) and against an LLM+RAG setup, Aito isstrongest exactly on the hard, high-cardinality, cross-table targets a bag-of-features model struggles with — on the current baselines it tops the invoice-routing and expense suites, while purpose-built fine-tuned NLP models still lead on Banking77 — and it gets there withzero training time and full explainability. That's the differentiator; that's what these suites measure first.
- Raw query speed is not our competition area — and that's fine. For plain indexed search and filtering, a specialized engine (Elasticsearch, a native C++ columnar store) will be faster, and we don't try to beat them at their own game. Aito's bar on that axis is**"not too slow"** : fast enough to serve interactive queries while carrying the prediction layer those engines don't have. Thespeed page states this plainly, with numbers.
The trade you're evaluating is: a single system that predicts, recommends, relates and searches over your live data with no model-training pipeline — versus stitching a search engine to a separately-trained, separately-served ML stack. These benchmarks are here to show that trade is a good one on quality, and acceptable on speed.
Accuracy suites #
Each suite is a real task with a held-out test set and published baselines, run as a booktest so the numbers are reproducible. We're starting with a focused set and will extend it with more common ML benchmarks over time.
Automated GL coding / invoice routing — predict processor, acceptor and GL code from invoice text withboth the v1 and v2 engines , vs Random Forest / LightGBM / FLAML AutoML. On the current baseline (after the engine's scoring redesign)Aito v2 leads top-1 on all three targets with0 s training — decisively on acceptor (v2 68.5% vs FLAML 59.5%), within the n=200 noise band on GL code and processor, so read those two as parity with a tuned AutoML search.FLAML remains the strongest baseline. The rank story backs it up: v2's link priors put the true acceptor at mean rank 1.8 (v1 2.2) and cut the processor rank ~3× vs v1. #
Expense categorization — predict
an invoice's expense category (48 classes) from vendor, tax rate, amount and
line text, vs Random Forest, LightGBM and FLAML AutoML (an LLM + RAG baseline is
a pending slot). The headline:Aito v1 tops the top-1 table (80.0% vs
LightGBM's
76.1%) — though
the top is a tight cluster inside a test-set CI — and Aito iswell-calibrated out of the box , with0 s training and a$why behind
every prediction, where the tree defaults are over- or under-confident.
Reports accuracy,calibration (ECE / Brier / gap), and training time. #
Invoice line → product matching — match an incoming invoice line to a SKU in a live catalogue, as a cross-table link prediction, vs BM25 in three strengths. Against a catalogue-only lexical index — what most matching systems ship — 90.4% vs 64.4%; against BM25given the same billing history the margin narrows to 89.2%, which is the honest comparison and the one the page leads with. The differentiator is that the history layeris the database — corrections are queryable on insert, with no reindex. Also states the open defect (popular SKUs beating thin correct ones) and carries a partner-facing implementation section. #
Intent classification (Banking77) — 77-way fine-grained intent on 13k banking queries. 82.6% top-1 withzero training , shown in honest context against fine-tuned transformer sentence-encoders (which reach the low-90s and win on raw accuracy). #
Smart search & recommendations — rank the
products a shopper was shown so the one they bought comes first, from the query,
their profile and the products' attributes, on a generated shop log with a
known answer key.recommend withbasedOn (0.534 nDCG@10) beats
BM25 given the same purchase history
(0.468), mostly on
wordings the catalogue does not contain; text search still wins on brand
queries. Also the For You shelf, with no query.
On the roadmap: text classification (Reuters, BBC News) and retrieval / RAG (BEIR SciFact, where Aito's inverted index feeds retrieval-augmented generation).
Speed & footprint #
- Query, predict & write speed — query latency against Elasticsearch on identical data, the v1-vs-v2 prediction head-to-head at 10M rows, and ingest throughput — with an honest read on how Aito compares to a specialized search engine.
- Memory & disk footprint — what the same 10M-row database costs to hold:597 MB of heap , ~3.8× leaner than the v1 engine, and why the heap doesn't grow with the corpus.
- Queries under writes — what a prediction costs on a database that is being written to the whole time the queries run: 853 ms at p50 under a continuous write load, and every answer byte-identical to a cold recompute.
How it scales #
Two different questions get asked with the same word — how much data do I need? and how big can my database get? — and one page now answers both on one corpus (expense categorization) with one fixed hold-out, so a number on either axis is about the same data.
- Training size, 1 → 1 000 000 rows on a corpus that grows by adding tenants, every method on the same rows: Aito leads from N=10 through N=10 000 with no training step (at N=1 000, v2 is 64% against a field at 57%); the field is level at 100 000; at 1 000 000 LightGBM is ahead by a point and a half after a 775 s training run, with Aito v1 level with Random Forest. Published as the crossover it is.
- Database size, to 10 000 000 rows : predict latency, the cost of coding a whole invoice in one batched request, and heap / mapped / disk for both engines — v2 at 10M holds the database in 1,609 MB of heap against v1's 3,997 MB.
- The finer engine-to-engine curve on the linked invoice corpus, 1k → 10M, is
the capacity ladder .
How the numbers are produced #
Every figure comes from a run, not a slide. Accuracy suites are booktests over
fixed test sets; performance figures come from the InvoicePerf /
FlatInvoicePerf harnesses. The specific numbers on these pages — including the
ones cited above — are generated at documentation-build time: a generator
reads each run's committed metrics.json and interpolates it into the page, so
they update themselves on re-baseline and a stale hand-typed digit can't slip
through. The build fails loudly on any unresolved metric token.
Note: Benchmarks are beta and use a single random seed. The expense suites score 2,000 held-out rows, which puts a 95% interval of about ±2 points on a cell, and every "ahead of" claim on those pages comes from an exact McNemar test on the rows where two methods disagree — where the test does not separate them, the page says they are tied. Invoice routing still scores 200 rows (±~5 points), so read its cells as point estimates and read the direction rather than the digits.