cd /news/ai-agents/meta-s-ai-agents-ran-1097-experiment… · home › topics › ai-agents › article
[ARTICLE · art-173738] src=epyhia.com ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Meta's AI Agents Ran 1,097 Experiments in a Month for a 0.23% Search Gain

Meta researchers reported in a November 18 preprint that autonomous AI agents, driven by Anthropic's Claude Opus 4.8, ran 1,097 machine-learning experiments over 31 days and roughly 360,000 GPU-hours to improve Meta's Nullstate search-suggestion ranker by 0.23% in a live A/B test, with a 95% confidence interval of ±0.15 percentage points. Of the 1,097 completed candidate experiments across 131 iterations, 325 were accepted — a 29.6% acceptance rate, up from 15.6% in an earlier sequential version — compounding into a -0.0224 improvement in a weighted normalized-entropy objective. The researchers disclosed an unquantified latency regression in the search interface's time-to-render and did not disclose the campaign's financial cost or run a matched comparison against a conventional engineering team.

recommendation

Give MLE agents a calibrated offline objective plus a 'what didn't work' blocklist and parallel evaluation, but add latency and other production constraints to the reward or review gate so agents don't ship unmeasured regressions.

more in playbook: Designing agent systems
read3 min views10 publishedNov 21, 2026

Meta researchers deployed autonomous AI agents to run 1,097 machine-learning experiments over 31 days, consuming roughly 360,000 GPU-hours, to improve the company's search-suggestion system by 0.23% in a live A/B test. The results, detailed in a November 18 preprint, offer a rare concrete look at the real-world returns of large-scale agentic AI deployment in a production environment [1].

The experiment #

The target was Nullstate, Meta's search-suggestion ranker that determines the recommendations users see when opening the search bar before typing a query. The system handles more than a billion ranking requests per day [2].

Traditionally, improving such models requires engineers to hypothesize changes, implement them, train models, and evaluate results—a slow, serial process. Meta's "MLE agents" automated much of this loop. Using Anthropic's Claude Opus 4.8, the agents proposed architectural changes, wrote code, launched distributed training jobs on 64-GPU trainers, and evaluated offline metrics. Engineers retained control over the objective, strategic direction, and final model review [2].

The agents ran nominally seven candidates in parallel per iteration, plus a variance probe, across 131 iterations. Each training-and-evaluation run took a median of 3.54 hours. Of 1,097 completed candidate experiments, 325 were accepted—a 29.6% acceptance rate, up from 15.6% in an earlier, sequential version of the system [2].

Offline gains, online validation #

The accepted changes compounded into a -0.0224 improvement in a weighted normalized-entropy objective (lower is better), driven mainly by better prediction of downstream engagement [2]. Several accepted changes were architectural ideas the human team had never tried, including Random Fourier Feature gating, learned-sigmoid attention gating, and a zero-init gated distillation bypass [2].

Meta then tested the consolidated model in a live A/B test over one week. Search-session volume increased 0.23% relative to the control, with a 95% confidence interval of ±0.15 percentage points. Downstream engagement metrics moved in a positive direction [2].

Trade-offs and undisclosed costs #

The experiment exposed a practical trade-off. The new model slowed the search interface's time-to-render, a latency regression the researchers disclosed but did not quantify [2]. The agents had optimized against a calibrated offline objective tied to engagement; rendering speed was monitored separately and not part of the agent's reward function. The authors acknowledged that an agent optimizing a pure quality objective does not surface such trade-offs on its own [2].

Meta estimated the campaign consumed approximately 360,000 GPU-hours, including 155 variance probes and 2,798 failed or retried attempts [2]. The company did not disclose the financial cost, nor did it run a matched comparison against a conventional engineering team pursuing the same objective. The paper leaves permanent rollout and that human-only benchmark unresolved [2].

What this shows—and doesn't #

The work demonstrates that autonomous agents can perform meaningful ML engineering on a production-scale system, clearing an industry-level bar far higher than academic benchmarks [2]. The calibrated objective, structured context (including a "what didn't work" blocklist), and parallel evaluation loop were more decisive than the underlying LLM, which was held fixed throughout [2].

But the results also illustrate diminishing returns: the best model was found well before the campaign's end, and later iterations added nothing. The agents also exhibited exploration collapse (an 0-for-8 streak on optimizer tuning before human redirection) and unreliable self-correction [2]. The offline-online gap remains: not every offline winner transfers, and the agent has no mechanism beyond the calibrated objective to predict which will.

For a platform of Meta's scale, a 0.23% lift in search sessions is statistically significant and operationally meaningful. Whether it justifies the compute, the latency cost, and the engineering overhead of the agent harness is a question the paper does not answer.

Sources #

  1. Meta's AI agents ran over 1000 experiments in a month but improved ... (cryptoslate.com)
  2. MLE Agents: Autonomous ML Engineering in Large-scale ... (arxiv.org)
  3. 3 Unstoppable AI Stocks to Buy Hand Over Fist Now That the Midterms Are ... (theglobeandmail.com)
  4. NEAR Protocol (NEAR) Historical Prices (coinlore.com)
── more in #ai-agents 4 stories · sorted by recency
── more on @meta 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/meta-s-ai-agents-ran…] indexed:0 read:3min 2026-11-21 · —