Benchmarking System One small LLM decision APIs vs. in-process tree ensembles and exact TreeSHAP on 64-feature tabular payloads

An 18-model verification matrix for SMILE 7.2 found that in-process tree ensembles with exact TreeSHAP outperform small LLM decision APIs on structured tabular payloads, with GradientTreeBoost scoring 0.24 ms p50 and 0.47 ms p99 and exact TreeSHAP staying under 5 ms p99 across all 9 matrix cells. On an unconstrained 300-tree, 64-feature RandomForest with max_depth=20, single-row scoring took 0.41 ms p50 but adding ?explain=true pushed exact TreeSHAP latency to 192.5 ms p50 and 274.4 ms p99, worse than a cloud LLM call. The author reports that constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs and that prompt-generated reason codes are not true feature attributions, and recommends keeping closed-set scoring, routing and exact SHAP attributions in-process under 3 ms while reserving small LLMs for extracting structured fields from messy text or documents.

There has been a lot of discussion lately around “System One” decision models: fast, small LLM endpoints tuned to return structured routing choices, rubric scores, or risk probabilities inside application code for ~90-210 ms at $100-$345 per 1M calls. If the input is a messy PDF, support ticket, or audio transcript, calling a small multimodal model makes complete sense. Where I still see teams get burned is routing structured numeric/categorical feature vectors say, 20 to 200 features inside a fraud, underwriting, or ranking hot loop through an LLM endpoint so they can get a score plus a natural-language “reason code” in one API call. Aside from the 150 ms network RTT and token cost, even constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs, and prompt-generated reason codes are not true feature attributions. While running an 18-model verification matrix for SMILE 7.2 smile-serve running both on HotSpot JVM and as a GraalVM native image at ~140-164 MiB idle RSS, alongside ONNX Runtime , I measured where the latency actually goes when you pair classical tree ensembles with exact per-row TreeSHAP in-process. A few numbers stood out: Scoring itself is almost free. Across 50 to 500 trees and 20 to 200 features 500 rps, concurrency 8, coordinated-omission corrected on an AMD EPYC 7401P , single-row GradientTreeBoost and RandomForest scoring stayed at 0.24-0.65 ms p50 and 0.47-1.57 ms p99 over loopback HTTP and 0.21 ms p50 / 0.43 ms p99 on exported XGBoost .onnx . Exact TreeSHAP is where naive Random Forests blow up. Scoring a tree scales linearly with depth O T D , but exact TreeSHAP scales with depth squared, O T L D^2 . On an unconstrained 300-tree, 64-feature RandomForest max depth=20 , single-row scoring took 0.41 ms p50 / 0.92 ms p99, but adding ?explain=true pushed latency to 192.5 ms p50 / 274.4 ms p99 worse than a cloud LLM call . Here is the depth/leaf ablation on the 300-tree, 64-feature Random Forest JVM, 100k measured requests after 5k warm-up : | RF Hyperparameters 300 trees, 64 features | .sml Size | Cold Start | Scoring p50 / p99 | Exact TreeSHAP p50 / p99 | 500 rps p99 | Stream rows/s | |---|---|---|---|---|---|---| | max depth=20, max nodes=0 unbounded | 16.8 MB | 12.50 s | 0.41 ms / 0.92 ms | 192.5 ms / 274.4 ms | 12.9 ms | 5,201 /s | | max depth=10, max nodes=0 | 3.40 MB | 3.91 s | 0.31 ms / 0.98 ms | 22.5 ms / 28.9 ms | 2.43 ms | 9,195 /s | | max depth=8, max nodes=256 | 1.56 MB | 2.49 s | 0.28 ms / 0.56 ms | 7.81 ms / 9.09 ms | 0.90 ms | 11,039 /s | | max depth=6, max nodes=128 | 637 KB | 2.32 s | 0.26 ms / 0.54 ms | 2.14 ms / 2.88 ms | 0.78 ms | 14,662 /s | | GradientTreeBoost 300 trees, 64 features | 292 KB | 1.55 s | 0.24 ms / 0.47 ms | 0.69 ms / 0.91 ms | 0.64 ms | 19,241 /s | GradientTreeBoost passed <5 ms p99 TreeSHAP across all 9 matrix cells 1.26 ms to 3.97 ms p99 even at 500 trees x 200 features , the 300-tree RandomForest on Native Image had a bimodal tail. 97.7% of requests finished in under 5 ms p50 = 4.48 ms , p95 = 4.54 ms , zero requests landed between 9.1 ms and 33.3 ms, and 1.16% clustered at 33.8-34.3 ms. Those spikes came from per-tree path allocations inside TreeSHAP triggering GraalVM Native Image’s default Serial GC ~29 ms stop-the-world pause roughly once every 85 requests , whereas HotSpot’s G1GC kept p99 at 2.88 ms and p99.9 at 3.97 ms. The takeaway for hybrid agent + ML pipelines is pretty simple: use small LLMs when you need to extract structured fields out of messy text or documents, and keep closed-set scoring, routing, and exact SHAP attributions in-process under 3 ms. Code and benchmark harness are in the SMILE repo if anyone wants to reproduce or poke at the TreeSHAP implementation: GitHub - haifengl/smile: Statistical Machine Intelligence & Learning Engine · GitHub https://github.com/haifengl/smile For folks running real-time SHAP or feature attributions in production latency budgets <10 ms , are you constraining tree depth at training time, using shallow boosted trees over bagged trees, or approximating explanations asynchronously off the hot path?