{"slug": "frontieror-challenge-puts-agentic-algorithm-design-to-a-hidden-instance-test", "title": "FrontierOR Challenge Puts Agentic Algorithm Design to a Hidden-Instance Test", "summary": "The FrontierOR Challenge will require teams to design and iteratively improve agentic algorithms that solve optimization problems on private, hidden instances, with registration closing Dec. 27, 2026, at 23:59 EST and Stage 1 beginning Jan. 4, 2027. Stage 1 covers 60 problems drawn from a 180-problem audited benchmark across six application categories and five formulation types over four weekly rounds of 15 problems, and Stage 2 runs Feb. 8–14, 2027 on problems outside the published benchmark. The challenge white paper cites a 39% result from the prior one-shot benchmark, where the strongest model returned a feasible solution with an objective gap no greater than 1% against Gurobi on 39% of large instances — a figure the paper says should not be read as a general agent success rate or a forecast for the Main track, which allows only open-source dependencies in a sandbox limited to 2 vCPU, 4 GB of memory and a 4 MB archive.", "body_md": "The FrontierOR Challenge represents a fundamental shift in how we evaluate the efficacy of [agentic workflows](https://forkast.news/learn/what-is-agentic-ai/). Rather than assessing a model’s ability to provide a static, correct answer, the competition requires teams to design, test, and iteratively improve algorithms capable of solving complex optimization problems. For builders, the challenge is to engineer a system that can navigate unseen instances under predefined operating rules.\n\nThe competition structure, detailed in the [challenge white paper](https://arxiv.org/html/2611.26171v1), is designed to test scalability and generalization. Stage 1 consists of 60 problems drawn from a 180-problem audited benchmark, covering six application categories and five formulation types, including combinatorial, mixed-integer, quadratic, stochastic, and robust or bilevel models. The stage unfolds over four weekly rounds of 15 problems, commencing Jan. 4, 2027. Prospective participants must finalize registration and team formation via the [Getting Started page](https://frontieror-challenge.com/getting-started) by Dec. 27, 2026, at 23:59 EST. Public instances are provided for development and local testing, but official scoring relies exclusively on private, hidden instances. Stage 2, scheduled for Feb. 8–14, 2027, introduces problems outside the published benchmark, testing systems on entirely unseen instances.\n\nBuilders must navigate two distinct tracks, each imposing different engineering constraints. The Main track requires algorithm code for a network-disabled standardized sandbox, limited to 2 vCPU, 4 GB of memory, and a 4 MB archive; only open-source dependencies are allowed, and each problem has a strict time limit. The Self-hosted track permits any software, including commercial solvers, and accepts solutions computed on participants’ own infrastructure within a timed window. Because the tracks utilize separate private instance sets and operate under different resource environments, they are ranked separately. Consequently, scores are not direct comparisons across the two environments.\n\nThe challenge’s scoring makes the distinction between a runnable submission and a successful algorithm concrete. Every returned solution is first checked for feasibility. If the solution is feasible, its objective value is compared against a frozen reference on hidden instances. Teams are ranked by average problem scores, with normalized solving time used as a tie-breaker; infeasible or missing instances receive a score of 0. Each problem carries equal weight in the stage average, so raw instance totals alone do not define the ranking. Main entrants submit code for the organizers’ sandbox, while Self-hosted entrants upload solutions produced on their own machines. A team choosing the Main track must ensure their code functions within the published resource and dependency limits. Conversely, Self-hosted teams can utilize broader software but must solve and upload each instance within its specific window. Both tracks evaluate performance on unseen instances, but the artifacts differ: algorithm code in the Main track and computed solutions in the Self-hosted track.\n\nThe Challenge white paper cites a 39% result, noting that the strongest model met the stated condition on 39% of “large instances”: returning a feasible solution with an objective gap no greater than 1% against Gurobi and running no slower than the reference. The [original benchmark paper](https://arxiv.org/html/2605.25246) describes the scope as “Full-set task instances.” These labels differ and should not be treated as identical. This statistic belongs to the prior one-shot benchmark, not the current Challenge. The earlier benchmark also allowed generated algorithms to call Gurobi, unlike the Challenge Main track’s open-source-only rule.\n\nThis 39% figure is benchmark-specific and should not be treated as a general agent success rate. It serves as historical context rather than a forecast for Main-track participants. It cannot predict how new workflows will perform across the Challenge’s rounds or on the new Stage 2 problems. No Challenge outcomes are available yet, and the schedule and rulebook alone cannot show whether teams will meet its feasibility and quality requirements.\n\nFor builders, the immediate decision is which track matches the system they can actually submit. Preparing against public problems can exercise a system, but only private instances determine scored performance. The challenge’s eventual results will provide evidence about these particular workflows and rules; until then, past benchmark performance cannot substitute for competition results.", "url": "https://wpnews.pro/news/frontieror-challenge-puts-agentic-algorithm-design-to-a-hidden-instance-test", "canonical_source": "https://forkast.news/frontieror-challenge-puts-agentic-algorithm-design-to-a-hidden-instance-test/", "published_at": "2026-11-22 01:23:18+00:00", "updated_at": "2026-11-22 01:43:17.670361+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "artificial-intelligence"], "entities": ["FrontierOR Challenge", "Gurobi"], "also_reported_by": [], "takeaway": "The strongest model met the feasible-within-1%-of-Gurobi and no-slower-than-reference condition on 39% of large instances in the prior one-shot benchmark, which allowed generated algorithms to call Gurobi — a setup the FrontierOR Main track forbids by permitting only open-source dependencies in a 2 ", "recommendation": {"advice": "Evaluate agentic optimization systems on private hidden instances with feasibility checks and a 1% objective-gap threshold against Gurobi, since public instances and one-shot benchmark scores do not predict hidden-instance performance.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/frontieror-challenge-puts-agentic-algorithm-design-to-a-hidden-instance-test", "markdown": "https://wpnews.pro/news/frontieror-challenge-puts-agentic-algorithm-design-to-a-hidden-instance-test.md", "text": "https://wpnews.pro/news/frontieror-challenge-puts-agentic-algorithm-design-to-a-hidden-instance-test.txt", "jsonld": "https://wpnews.pro/news/frontieror-challenge-puts-agentic-algorithm-design-to-a-hidden-instance-test.jsonld"}}