{"slug": "compiling-autoregressive-llm-control-into-policy-exact-parallel-inference-for", "title": "Compiling Autoregressive LLM Control into Policy-Exact Parallel Inference for Hierarchical RAG", "summary": "A new arXiv paper (2610.30664v1) reports that compiling autoregressive LLM control into parallel inference for hierarchical RAG matches serial grammar-constrained decoding on 84–89% of questions across four document-disjoint splits and two scales on the primary MetaX device, with every instrumented divergence attributed to floating-point near-ties. The compiled approach uses one root prefill plus one tree-causal probe to evaluate every prefix-conditioned action distribution, giving a two-forward critical path and zero output tokens, and cuts control latency by 35–47% versus token-wise serial decoding and 6–7% versus an idealized chunked engine. An H800 repetition preserved the ordering, and a further probe pass certifies the exact order-marginalized MAP set.", "body_md": "arXiv:2610.30664v1 Announce Type: new \nAbstract: Large language models increasingly act as controllers in structured decision pipelines, yet their decisions are still executed by autoregressive text decoding. Hierarchical RAG is one such setting: at each frontier of a document tree, an LLM commits to $b$ of $k$ sections under an input-dependent action grammar, executed as either failure-prone free-form JSON or reliable but strictly serial grammar-constrained generation. We show that this serialization is unnecessary for bounded-horizon control: the reachable future forms a shallow action-prefix tree, which we compile --- one root prefill plus one tree-causal probe evaluate every prefix-conditioned action distribution with serial decoding's tokens, positions, and causal ancestry. A prefix-state lemma makes compiled control policy-exact in exact arithmetic: the same policy with a two-forward critical path and zero output tokens. One further probe pass certifies the exact order-marginalized MAP set. Across four document-disjoint splits and two scales on the primary MetaX device, compiled control matches the serial policy on $84$--$89\\%$ of questions --- every instrumented divergence a floating-point near-tie --- and cuts control latency by $35$--$47\\%$ versus token-wise serial decoding and $6$--$7\\%$ versus an idealized chunked engine; an H800 repetition preserves the ordering.", "url": "https://wpnews.pro/news/compiling-autoregressive-llm-control-into-policy-exact-parallel-inference-for", "canonical_source": "https://arxiv.org/abs/2610.30664", "published_at": "2026-10-23 04:00:00+00:00", "updated_at": "2026-10-23 04:13:10.596327+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["arXiv", "MetaX", "H800"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/compiling-autoregressive-llm-control-into-policy-exact-parallel-inference-for", "markdown": "https://wpnews.pro/news/compiling-autoregressive-llm-control-into-policy-exact-parallel-inference-for.md", "text": "https://wpnews.pro/news/compiling-autoregressive-llm-control-into-policy-exact-parallel-inference-for.txt", "jsonld": "https://wpnews.pro/news/compiling-autoregressive-llm-control-into-policy-exact-parallel-inference-for.jsonld"}}