{"slug": "faro-function-space-adaptive-return-to-risk-optimization-via-cross-block-equal", "title": "FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation", "summary": "Researchers released FARO, a function-space adaptive optimizer that assigns each parameter block its own update geometry and reallocates a frozen per-block budget by cross-block equal risk contribution (ERC), the risk-parity principle from portfolio theory. On a GPT-style model trained on FineWeb, FARO-Adaptive reached 3.2780 nats at the full schedule, matching the hand-designed NorMuonAndAdam recipe at 3.2794, while FARO-ERC lowered clean-data loss on all 6 seeds by 0.0136 nats on average with no added time or memory. The source code is available at https://github.com/wxj630/FARO.", "body_md": "arXiv:2610.40858v1 Announce Type: new \nAbstract: Training a large language model is compute-bound, and current recipes divide the work between optimizers by hand: orthogonalized momentum for attention and MLP matrices, Adam-style updates for the rest. This division rests on human priors about which layer type deserves which geometry, and it fixes a single global update budget that every parameter block shares. We introduce FARO, a function-space adaptive optimizer that makes the partition of parameters into blocks the unit of decision. FARO gives each block a geometry by its realized counterfactual return, then freezes the routes and reallocates the frozen per-block budget by cross-block equal risk contribution (ERC), the risk-parity principle from portfolio theory. On a GPT-style model trained on FineWeb, the framework gives two results. First, its FARO-Adaptive arm reaches 3.2780 nats at the full schedule, matching NorMuonAndAdam (3.2794), the best hand-designed recipe: it recovers the human prior without hand-setting per-layer geometry. Second, FARO-ERC improves on that arm: it lowers the clean-data loss at every one of 6 seeds, by 0.0136 nats on average, a modest but robust gain. Its sign test, paired test, and confidence interval all agree, and it exceeds the measured run-to-run noise at no added time or memory. It works by trimming the riskiest blocks rather than expanding the safest, and it holds up under a corrupted stream, still lowering the mean loss and winning the seed majority. The source code is available at https://github.com/wxj630/FARO.", "url": "https://wpnews.pro/news/faro-function-space-adaptive-return-to-risk-optimization-via-cross-block-equal", "canonical_source": "https://arxiv.org/abs/2610.40858", "published_at": "2026-11-02 05:00:00+00:00", "updated_at": "2026-11-02 05:21:03.152919+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["FARO", "FARO-Adaptive", "FARO-ERC", "NorMuonAndAdam", "FineWeb", "GPT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/faro-function-space-adaptive-return-to-risk-optimization-via-cross-block-equal", "markdown": "https://wpnews.pro/news/faro-function-space-adaptive-return-to-risk-optimization-via-cross-block-equal.md", "text": "https://wpnews.pro/news/faro-function-space-adaptive-return-to-risk-optimization-via-cross-block-equal.txt", "jsonld": "https://wpnews.pro/news/faro-function-space-adaptive-return-to-risk-optimization-via-cross-block-equal.jsonld"}}