There has been a lot of discussion lately around “System One” decision models: fast, small LLM endpoints tuned to return structured routing choices, rubric scores, or risk probabilities inside application code for ~90-210 ms at $100-$345 per 1M calls.
If the input is a messy PDF, support ticket, or audio transcript, calling a small multimodal model makes complete sense.
Where I still see teams get burned is routing structured numeric/categorical feature vectors (say, 20 to 200 features inside a fraud, underwriting, or ranking hot loop) through an LLM endpoint so they can get a score plus a natural-language “reason code” in one API call. Aside from the 150 ms network RTT and token cost, even constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs, and prompt-generated reason codes are not true feature attributions.
While running an 18-model verification matrix for SMILE 7.2 (smile-serve running both on HotSpot JVM and as a GraalVM native image at ~140-164 MiB idle RSS, alongside ONNX Runtime), I measured where the latency actually goes when you pair classical tree ensembles with exact per-row TreeSHAP in-process. A few numbers stood out:
Scoring itself is almost free. Across 50 to 500 trees and 20 to 200 features (500 rps, concurrency 8, coordinated-omission corrected on an AMD EPYC 7401P), single-row GradientTreeBoost and RandomForest scoring stayed at 0.24-0.65 ms p50 and 0.47-1.57 ms p99 over loopback HTTP (and 0.21 ms p50 / 0.43 ms p99 on exported XGBoost .onnx).
Exact TreeSHAP is where naive Random Forests blow up. Scoring a tree scales linearly with depth O(T * D), but exact TreeSHAP scales with depth squared, O(T * L * D^2). On an unconstrained 300-tree, 64-feature RandomForest (max_depth=20), single-row scoring took 0.41 ms p50 / 0.92 ms p99, but adding ?explain=true pushed latency to 192.5 ms p50 / 274.4 ms p99 (worse than a cloud LLM call).
Here is the depth/leaf ablation on the 300-tree, 64-feature Random Forest (JVM, 100k measured requests after 5k warm-up):
| RF Hyperparameters (300 trees, 64 features) | .sml Size |
Cold Start | Scoring p50 / p99 | Exact TreeSHAP p50 / p99 | 500 rps p99 | Stream (rows/s) |
|---|---|---|---|---|---|---|
max_depth=20, max_nodes=0 (unbounded) |
16.8 MB | 12.50 s | 0.41 ms / 0.92 ms | 192.5 ms / 274.4 ms | 12.9 ms | 5,201 /s |
max_depth=10, max_nodes=0 |
3.40 MB | 3.91 s | 0.31 ms / 0.98 ms | 22.5 ms / 28.9 ms | 2.43 ms | 9,195 /s |
max_depth=8, max_nodes=256 |
1.56 MB | 2.49 s | 0.28 ms / 0.56 ms | 7.81 ms / 9.09 ms | 0.90 ms | 11,039 /s |
max_depth=6, max_nodes=128 |
637 KB | 2.32 s | 0.26 ms / 0.54 ms | 2.14 ms / 2.88 ms | 0.78 ms | 14,662 /s |
GradientTreeBoost (300 trees, 64 features) |
292 KB | 1.55 s | 0.24 ms / 0.47 ms | 0.69 ms / 0.91 ms | 0.64 ms | 19,241 /s |
GradientTreeBoost passed <5 ms p99 TreeSHAP across all 9 matrix cells (1.26 ms to 3.97 ms p99 even at 500 trees x 200 features), the 300-tree RandomForest on Native Image had a bimodal tail. 97.7% of requests finished in under 5 ms (p50 = 4.48 ms, p95 = 4.54 ms), zero requests landed between 9.1 ms and 33.3 ms, and 1.16% clustered at 33.8-34.3 ms. Those spikes came from per-tree path allocations inside TreeSHAP triggering GraalVM Native Image’s default Serial GC (~29 ms stop-the-world roughly once every 85 requests), whereas HotSpot’s G1GC kept p99 at 2.88 ms and p99.9 at 3.97 ms.
The takeaway for hybrid agent + ML pipelines is pretty simple: use small LLMs when you need to extract structured fields out of messy text or documents, and keep closed-set scoring, routing, and exact SHAP attributions in-process under 3 ms.
Code and benchmark harness are in the SMILE repo if anyone wants to reproduce or poke at the TreeSHAP implementation: GitHub - haifengl/smile: Statistical Machine Intelligence & Learning Engine · GitHub
For folks running real-time SHAP or feature attributions in production latency budgets (<10 ms), are you constraining tree depth at training time, using shallow boosted trees over bagged trees, or approximating explanations asynchronously off the hot path?