{"slug": "benchmarking-system-one-small-llm-decision-apis-vs-in-process-tree-ensembles-and", "title": "Benchmarking System One small LLM decision APIs vs. in-process tree ensembles and exact TreeSHAP on 64-feature tabular payloads", "summary": "An 18-model verification matrix for SMILE 7.2 found that in-process tree ensembles with exact TreeSHAP outperform small LLM decision APIs on structured tabular payloads, with GradientTreeBoost scoring 0.24 ms p50 and 0.47 ms p99 and exact TreeSHAP staying under 5 ms p99 across all 9 matrix cells. On an unconstrained 300-tree, 64-feature RandomForest with max_depth=20, single-row scoring took 0.41 ms p50 but adding ?explain=true pushed exact TreeSHAP latency to 192.5 ms p50 and 274.4 ms p99, worse than a cloud LLM call. The author reports that constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs and that prompt-generated reason codes are not true feature attributions, and recommends keeping closed-set scoring, routing and exact SHAP attributions in-process under 3 ms while reserving small LLMs for extracting structured fields from messy text or documents.", "body_md": "There has been a lot of discussion lately around “System One” decision models: fast, small LLM endpoints tuned to return structured routing choices, rubric scores, or risk probabilities inside application code for ~90-210 ms at $100-$345 per 1M calls.\n\nIf the input is a messy PDF, support ticket, or audio transcript, calling a small multimodal model makes complete sense.\n\nWhere I still see teams get burned is routing structured numeric/categorical feature vectors (say, 20 to 200 features inside a fraud, underwriting, or ranking hot loop) through an LLM endpoint so they can get a score plus a natural-language “reason code” in one API call. Aside from the 150 ms network RTT and token cost, even constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs, and prompt-generated reason codes are not true feature attributions.\n\nWhile running an 18-model verification matrix for SMILE 7.2 (`smile-serve` running both on HotSpot JVM and as a GraalVM native image at ~140-164 MiB idle RSS, alongside ONNX Runtime), I measured where the latency actually goes when you pair classical tree ensembles with exact per-row `TreeSHAP` in-process. A few numbers stood out:\n\nScoring itself is almost free. Across 50 to 500 trees and 20 to 200 features (500 rps, concurrency 8, coordinated-omission corrected on an AMD EPYC 7401P), single-row `GradientTreeBoost` and `RandomForest` scoring stayed at 0.24-0.65 ms p50 and 0.47-1.57 ms p99 over loopback HTTP (and 0.21 ms p50 / 0.43 ms p99 on exported XGBoost `.onnx`).\n\nExact `TreeSHAP` is where naive Random Forests blow up. Scoring a tree scales linearly with depth `O(T * D)`, but exact `TreeSHAP` scales with depth squared, `O(T * L * D^2)`. On an unconstrained 300-tree, 64-feature `RandomForest` (`max_depth=20`), single-row scoring took 0.41 ms p50 / 0.92 ms p99, but adding `?explain=true` pushed latency to 192.5 ms p50 / 274.4 ms p99 (worse than a cloud LLM call).\n\nHere is the depth/leaf ablation on the 300-tree, 64-feature Random Forest (JVM, 100k measured requests after 5k warm-up):\n\n| RF Hyperparameters (300 trees, 64 features) | `.sml` Size | Cold Start | Scoring p50 / p99 | Exact TreeSHAP p50 / p99 | 500 rps p99 | Stream (rows/s) | \n|---|---|---|---|---|---|---|\n| `max_depth=20, max_nodes=0` (unbounded) | 16.8 MB | 12.50 s | 0.41 ms / 0.92 ms | 192.5 ms / 274.4 ms | 12.9 ms | 5,201 /s | \n| `max_depth=10, max_nodes=0` | 3.40 MB | 3.91 s | 0.31 ms / 0.98 ms | 22.5 ms / 28.9 ms | 2.43 ms | 9,195 /s | \n| `max_depth=8, max_nodes=256` | 1.56 MB | 2.49 s | 0.28 ms / 0.56 ms | 7.81 ms / 9.09 ms | 0.90 ms | 11,039 /s | \n| `max_depth=6, max_nodes=128` | 637 KB | 2.32 s | 0.26 ms / 0.54 ms | 2.14 ms / 2.88 ms | 0.78 ms | 14,662 /s | \n| `GradientTreeBoost` (300 trees, 64 features) | 292 KB | 1.55 s | 0.24 ms / 0.47 ms | 0.69 ms / 0.91 ms | 0.64 ms | 19,241 /s | \n\n`GradientTreeBoost` passed `<5 ms` p99 TreeSHAP across all 9 matrix cells (1.26 ms to 3.97 ms p99 even at 500 trees x 200 features), the 300-tree `RandomForest` on Native Image had a bimodal tail. 97.7% of requests finished in under 5 ms (`p50 = 4.48 ms`, `p95 = 4.54 ms`), zero requests landed between 9.1 ms and 33.3 ms, and 1.16% clustered at 33.8-34.3 ms. Those spikes came from per-tree path allocations inside `TreeSHAP` triggering GraalVM Native Image’s default Serial GC (~29 ms stop-the-world pause roughly once every 85 requests), whereas HotSpot’s G1GC kept p99 at 2.88 ms and p99.9 at 3.97 ms.\nThe takeaway for hybrid agent + ML pipelines is pretty simple: use small LLMs when you need to extract structured fields out of messy text or documents, and keep closed-set scoring, routing, and exact SHAP attributions in-process under 3 ms.\n\nCode and benchmark harness are in the SMILE repo if anyone wants to reproduce or poke at the TreeSHAP implementation: [GitHub - haifengl/smile: Statistical Machine Intelligence & Learning Engine · GitHub](https://github.com/haifengl/smile)\n\nFor folks running real-time SHAP or feature attributions in production latency budgets (<10 ms), are you constraining tree depth at training time, using shallow boosted trees over bagged trees, or approximating explanations asynchronously off the hot path?", "url": "https://wpnews.pro/news/benchmarking-system-one-small-llm-decision-apis-vs-in-process-tree-ensembles-and", "canonical_source": "https://discuss.huggingface.co/t/benchmarking-system-one-small-llm-decision-apis-vs-in-process-tree-ensembles-and-exact-treeshap-on-64-feature-tabular-payloads/191865#post_1", "published_at": "2026-11-14 19:41:59+00:00", "updated_at": "2026-11-14 20:52:58.837567+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools", "ai-infrastructure", "large-language-models"], "entities": ["SMILE", "smile-serve", "GraalVM Native Image", "HotSpot JVM", "ONNX Runtime", "XGBoost", "AMD EPYC 7401P", "haifengl"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/benchmarking-system-one-small-llm-decision-apis-vs-in-process-tree-ensembles-and", "markdown": "https://wpnews.pro/news/benchmarking-system-one-small-llm-decision-apis-vs-in-process-tree-ensembles-and.md", "text": "https://wpnews.pro/news/benchmarking-system-one-small-llm-decision-apis-vs-in-process-tree-ensembles-and.txt", "jsonld": "https://wpnews.pro/news/benchmarking-system-one-small-llm-decision-apis-vs-in-process-tree-ensembles-and.jsonld"}}