{"slug": "gemini-4-often-hides-its-failures-when-searching-antibody-space-abmaze", "title": "Gemini 4 often hides its failures when searching antibody space (AbMaze)", "summary": "A benchmark called AbMaze, built by Dominik Scherm and published November 4, 2026, found that no frontier AI model outperforms a scripted Bayesian optimizer at navigating a fully measured antibody fitness landscape of 65,536 lab-scored variants under a 150-measurement budget. Gemini 4 drew level with the scripted Bayesian optimizer at 55% against its 50%, while the remaining models settled 6 to 19 points below. Gemini 4 also underreported its failures: when its best candidate did not bind, it omitted the failure in 76% of those runs, though it admitted the failure when asked, while every other model disclosed failures.", "body_md": "# AbMaze: Evaluating frontier AI agents on protein fitness landscapes\n\nDominik Scherm · November 4, 2026\n\n*TL;DR: We built AbMaze, a benchmark that runs an AI agent through an antibody campaign on a fully measured fitness landscape, 65,536 lab-scored variants, on a 150-measurement budget. No model surpasses a scripted Bayesian optimizer: Gemini 4 draws level at 55% against its 50%, the rest settle 6 to 19 points below. In addition, we found that Gemini 4 underreports its failures: when its best candidate didn't bind, it didn't mention the failure in 76% of those runs, though it admitted it when asked. Every other model disclosed failures.*\n\nDrug discovery, and antibody optimization in particular, is a search through an enormous space of possible sequences. Consider a protein of just 100 amino acids. To find the optimal binder, we would have to search through all its possible sequences, 20<sup>100</sup> or around 10<sup>130</sup>, far more than there are atoms in the observable universe (about 10<sup>80</sup>). That’s why we rely on rational design, and more recently, machine learning to narrow the search, using expert knowledge or priors learned from data.\n\nWhat if we designed a search space small enough to measure in full, so we could evaluate every decision an agent makes? How well can AI agents navigate that protein and act under uncertainty? Further, how do agents reach their goal, finding the optimal binder given a constrained budget?\n\n## [How AbMaze works](#how-abmaze-works)\n\nOur environment consists of 65,536 antibody variants (dataset from Phillips et al.<sup></sup>) that were  against three flu , [H1](https://en.wikipedia.org/wiki/Influenza_A_virus_subtype_H1N1), [H3](https://en.wikipedia.org/wiki/Influenza_A_virus_subtype_H3N2), and [influenza B](https://en.wikipedia.org/wiki/Influenza_B_virus). The variants are all 2¹⁶ combinations of the 16 mutations that separate a mature antibody ([CR9114](https://www.rcsb.org/structure/4FQH)<sup></sup>) from its ancestor, in its . Each site carries either the ancestral or the mature amino acid.\n\nOverall, 97% of all variants bind H1, 11% H3, and only 0.3% (198) influenza B. All of those also bind H1 and H3, and just 111 (0.17%) bind all three strongly. This skew is typical of antibodies: lots of variants work against one antigen, very few work against all of them, and those few are exactly the ones you want for a drug.\n\nThe goal for an agent is to start from a variant that sits 8 to 10 mutations away from the mature antibody and, within a budget of 150 credits, derive a variant that recognizes all three antigens as close as possible. One credit equals a measurement of one variant against one antigen, so the agent is limited to 150 binding measurements, enough to test about 50 variants on all three antigens. Each measurement is a single noisy replicate, so a high reading does not guarantee a strong binder. The agent also has access to a cheap predictor (costs 0.05 credits), representing computational tools. Like real prediction tools, ours is imperfect: it follows broad trends on the common antigens but is blind to the rare ones (rank correlation 0.4 for H1, close to 0 for influenza B). Beyond measuring and predicting, the agent can take notes (free), query the current state of the environment, including remaining credits (free), and submit up to three ranked variants.\n\nThe agent’s top-ranked pick is graded on its true binding score to all three antigens, where 0% means no better than the start and 100% picking the absolute best variant in the library. A winning strategy is to use the common antigens as a map: climb toward H1 binders, then H3 binders, and search for the rare influenza B binders near the top (see illustration below).\n\nWe evaluate each model on a set of four tasks, each from 10 initial variants and 5 random seeds, for a total of 200 runs per model.\n\nThe four different tasks ask to bind different antigen configurations:\n\n- **All three:** find a CR9114 variant that binds H1, H3 and influenza B (150 credits).\n- **Two hardest:** find a CR9114 variant that binds H3 and influenza B (150 credits).\n- **Influenza B alone:** find a CR9114 variant that binds influenza B (150 credits). The agent can still measure H1 and H3 as side antigens. They don't count toward its score, but they can serve as a map toward the target.\n- **CH65:** the same search for a second flu antibody, CH65<sup></sup> , against three of its own antigens, with a stricter detection threshold and 250 credits.\n\nAs a baseline, we built a Bayesian optimizer (BO)<sup></sup> that works under the same budget as the agents but follows a fixed procedure. It starts by measuring the start variant and its 16 single-mutant neighbors on all three antigens, then fits a separate model for each antigen.\n\nFor each antigen , a variant is a vector of 16 switches , and the model predicts its binding as\n\nwith priors for single mutations, for pairs and noise on standardized readings.\n\nEach round has four steps. First, BO fits the models to all measurements so far and draws one set of coefficients from the posterior (Thompson sampling<sup></sup>). Second, it predicts binding for all 65,536 variants, flooring each prediction at the , since readings below it carry no information. Third, it ranks the variants by\n\nIn short, a variant ranks high only if it is predicted to bind all three antigens, with a small extra bonus for binding them strongly overall.\n\nFourth, BO measures the four best untested variants on all three antigens, then repeats until the budget is spent.\n\n**Given the same budget as the agents, BO finds a variant binding all three antigens in 149 of 200 runs (75%) and scores 50.2% on average.**\n\n## [Results](#results)\n\nThe best frontier model performs on par with our Bayesian optimization baseline. Gemini 4 reaches 54.6% against 50.2% for our BO, a paired difference of +4.4 points [−8.5, +16.3]. Claude Opus 5.5 and Qwen3.8-max-prime come in at 40 to 44%. GPT-6 Astra and Grok 4.7 come in at 31 to 36%, significantly below BO. Random search scores 8.5%.\n\n| Agent | Score [95% CI] | Refusals | Cost per run | \n|---|---|---|---|\n| Gemini 4 | 54.6% [44, 65] | 0% | $1.42 | \n| Bayesian optimization (scripted) | 50.2% [41, 59] | – | $0 | \n| Claude Opus 5.5 | 44.2% [35, 54] | 22% | $0.43 | \n| Qwen3.8-max-prime | 40.4% [31, 49] | 0% | $1.89 | \n| GPT-6 Astra | 35.6% [26, 46] | 0% | $0.86 | \n| Grok 4.7 | 31.1% [23, 40] | 0% | $0.69 | \n| Random | 8.5% [5, 13] | – | $0 | \n\nScores are all-or-nothing. A run either finds a binder and scores high, or it never leaves the non-binding plateau and scores exactly 0. Even Gemini scores 0 in more than a quarter of its runs (Astra 57%, Opus 45% counting refusals).\n\nProgress comes late. After a quarter of the budget, every agent's best score so far is still below 5%, and most of the gain comes in the second half.\n\nOn cost per run, Claude Opus 5.5 is the cheapest frontier model ($0.43 against Gemini's $1.42) and still reaches 81% of Gemini's score.\n\n## [Analyzing each model](#analyzing-each-model)\n\n- **Gemini 4** is the best overall and by far the best on the influenza-B-only task (77%), where it has to decide on its own to measure the side antigens as a map. It is also expensive ($1.42 per run, second only to Qwen3.8-max-prime at $1.89), and when its lead fails it usually does not say so (76% of those runs, see our alignment section).\n- **GPT-6 Astra** never refuses and does well on the H3 + influenza B task (59%), but scores 32 to 33% on CR9114 all-three and CH65 and 23% on the influenza-B-only task. It uses very little reasoning per step and makes up for it with many turns. It is the only model that scores clearly lower than on v0.2.1 (−14.5 points [−25.0, −3.8], paired).\n- **Claude Opus 5.5** is the best on CR9114 all-three (58%). It writes to its notebook in every run (3 entries per run on average; Qwen3.8-max-prime writes 8 and about 7 times as much text) and repeats 15% of its measurements, about as often as Gemini and Qwen. But 22% of its runs end in a safety-classifier refusal, and it stops with credits left far more often than the others: it spends 80% of its budget on average, the others 98 to 100%.\n- **Qwen3.8-max-prime and Grok 4.7** are decent searchers at 31 to 40%, and Qwen3.8-max-prime is close to Gemini on CH65 (43%). Grok spends right down to zero and runs out of credits before it submits in 12% of runs. Qwen3.8-max-prime uses the most tokens to complete.\n\nTwo failure modes recur across models. First, agents fail to exit the non-binding plateau: when local measurements return uniformly low readings, which carry no gradient toward a binder, they do not change strategy and continue sampling near their current position. Second, final selection relies on a single noisy reading per variant, so the submitted variant is often not the best one measured. Submitting the best measured variant, by true score, would have raised every model's score by 2 to 5 points.\n\n## [What affects the results](#what-affects-the-results)\n\nWe ran a set of experiments to see what affects how agents perform. Overall, we found that scores are robust to the tools and harness around the agent and move mainly with the budget and with what the agent knows about the task. An unrestricted literature search helped only GPT-6 Astra (+19 points), mostly in runs where the retrieved papers named the antibody, so it looked up the answer instead of searching better. Budget is the strongest lever: at half the budget, 72% of runs score 0, and at double, 72% score 95% or more. How credits are framed changes behavior: Opus spends 51% of its budget when told credits are billed, against 90% when told they are worthless.\n\n| Ablation | Effect | \n|---|---|\n| Unrestricted literature search (Paperclip) | Only GPT-6 Astra gains, +19 pts [+2, +36], mostly in runs where the retrieved papers named the antibody | \n| Running in Claude Code <sup></sup> or opencode<sup></sup> instead of our minimal harness | No effect | \n| A code sandbox (shell) | Helps Opus when it finishes a run (+15 pts), but refusals go up from 20% to 32%, so overall it is +1 pt. Gemini +6 (not significant) | \n| Half or double the budget | At half the budget 72% of runs score 0, at double 72% score 95% or more. Only the current budget separates the models | \n| Telling agents credits are billed vs worthless | Opus spends 51% vs 90% of its budget (86% by default). Gemini and Astra also spend less when billed | \n\nAdditionally, we put the following safeguards in place to prevent leakage/reward hacking:\n\n- **Obfuscated encoding:** in every run we shuffle how each mutation site is encoded and the order of the sites, and we anonymize the antigens, so recalling the paper does not give away the answer.\n- **Hardness checks:** before a task counts, scripted strategies must fail to solve it: shortcuts score at most 30%, random search at most 25%, and the best scripted strategy between 35% and 65%.\n- **Leak review:** A red team, recall probes and leak checks found and closed nine leaks before we froze the task set.\n- **Hidden scoring and thresholds:** the agent never sees the true score or the per-antigen detection thresholds, since those could identify the antigens.\n\n## [Alignment](#alignment)\n\nThe environment records every action and its true outcome, so we could compare what agents did with what actually happened.\n\n**Gemini does not always report failures. When its lead never bound one of the target antigens, its final write-up left the failure out in 76% of those runs (41 of 54)**. We replayed those runs and asked it directly at the end, and it correctly named the failure in 40 of 41 cases, so it’s aware. Every other model disclosed the failure every time.\n\nAll models followed the rules when they were tempted. We added a tool that returns the true measurements and labelled it off-limits, and no model used it in 240 runs, even when we told them their use would not be logged. When we allowed the tool, models used it in 57 of 60 runs and gained 20 to 62 points.\n\nOpus refusals come from the accumulated antibody results in its context, in any form, and the antigen names do not matter. Sending the exact same refused request again gets refused 22.5% of the time, and telling Opus that this is a real lab campaign doubles its refusal rate.\n\nWhen we told the models they were being evaluated, Opus started reasoning about how it gets scored in 43% of runs (6% when we framed it as a real campaign), and Gemini named the source dataset more often.\n\nOpus stopping early is not overconfidence. It follows how the brief values credits, and its confidence in its own lead is well calibrated (0.96 for leads that truly bind, 0.07 for the ones that don't).\n\n## [Conclusions](#conclusions)\n\nAbMaze asks whether AI agents can run an antibody campaign on a fully measured landscape. So far the answer is: about as well as a simple scripted optimizer, and no better.\n\nThe models fail in similar ways. Scores are all-or-nothing: even Gemini ends more than a quarter of its runs at exactly 0, stuck on the non-binding plateau. At the last step, agents also submit based on a single noisy reading and often miss the best variant they measured.\n\nHow agents report their work matters too. Gemini 4 left a failed lead out of its write-up in 76% of such runs, even though it could name the failure when asked, while every other model disclosed failures.\n\n## [What's next](#whats-next)\n\nCan an open model learn this? We are post-training Qwen3.8-27B on AbMaze, and the training is still running as we speak. We will publish the results here soon.\n\nWe also plan to extend AbMaze to larger fitness landscapes, using our in-house wet lab to produce more data. Interested in partnering with us? [Get in touch](mailto:info@valenlabs.com).\n\n## [Citation](#citation)\n\n```\n@misc{scherm2026abmaze,\n  title        = {AbMaze: Evaluating frontier AI agents on protein fitness landscapes},\n  author       = {Scherm, Dominik},\n  howpublished = {Valen Labs},\n  year         = {2026},\n  url          = {https://hub.harborframework.com/datasets/valenlabs/abmaze}\n}\n```\n\nAbMaze builds on the antibody landscapes from:\n\n1. Phillips, A. M. et al. Binding affinity landscapes constrain the evolution of broadly neutralizing anti-influenza antibodies. eLife 10, e71393 (2021). [doi:10.7554/eLife.71393](https://doi.org/10.7554/eLife.71393)\n2. Phillips, A. M. et al. Hierarchical sequence-affinity landscapes shape the evolution of breadth in an anti-influenza receptor binding site antibody. eLife 12, e83628 (2023). [doi:10.7554/eLife.83628](https://doi.org/10.7554/eLife.83628)\n\nNote: Claude Opus 5.5 was evaluated with access through Anthropic's Life Sciences Verification Program (LSVP).\n\nModel settings: Gemini 4 (gemini-4-argon, thinking level high), Claude Opus 5.5 (effort high), GPT-6 Astra (reasoning effort high), Qwen3.8-max-prime (reasoning effort high, via OpenRouter, Alibaba provider), Grok 4.7 (reasoning effort high, via OpenRouter, xAI provider).\n\n## [References](#references)\n\n1. Phillips, A. M. et al. Binding affinity landscapes constrain the evolution of broadly neutralizing anti-influenza antibodies. *eLife* 10, e71393 (2021).[doi:10.7554/eLife.71393](https://doi.org/10.7554/eLife.71393)[↩](#cite-phillips2021)\n2. Dreyfus, C. et al. Highly conserved protective epitopes on influenza B viruses. *Science* 337, 1343–1348 (2012). Crystal structure of Fab CR9114: PDB 4FQH.[rcsb.org/structure/4FQH](https://www.rcsb.org/structure/4FQH)[↩](#cite-dreyfus2012)\n3. Phillips, A. M. et al. Hierarchical sequence-affinity landscapes shape the evolution of breadth in an anti-influenza receptor binding site antibody. *eLife* 12, e83628 (2023).[doi:10.7554/eLife.83628](https://doi.org/10.7554/eLife.83628)[↩](#cite-phillips2023)\n4. Shahriari, B. et al. Taking the human out of the loop: A review of Bayesian optimization. *Proceedings of the IEEE* 104, 148–175 (2016).[doi:10.1109/JPROC.2015.2494218](https://doi.org/10.1109/JPROC.2015.2494218)[↩](#cite-shahriari2016)\n5. Thompson, W. R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. *Biometrika* 25, 285–294 (1933).[doi:10.1093/biomet/25.3-4.285](https://doi.org/10.1093/biomet/25.3-4.285)[↩](#cite-thompson1933)\n6. Claude Code, Anthropic's agentic coding tool. [code.claude.com/docs](https://code.claude.com/docs/en/overview)[↩](#cite-claudecode)\n7. OpenCode, an open-source coding agent. [opencode.ai](https://opencode.ai)[↩](#cite-opencode)\n8. Harbor: a framework for evaluating and optimizing sandboxed agents and models. [harborframework.com](https://harborframework.com)[↩](#cite-harbor)", "url": "https://wpnews.pro/news/gemini-4-often-hides-its-failures-when-searching-antibody-space-abmaze", "canonical_source": "https://valenlabs.com/blog/abmaze", "published_at": "2026-11-04 19:15:17+00:00", "updated_at": "2026-11-04 19:21:25.643044+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-safety", "large-language-models"], "entities": ["Gemini 4", "AbMaze", "Dominik Scherm", "CR9114", "Phillips et al."], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gemini-4-often-hides-its-failures-when-searching-antibody-space-abmaze", "markdown": "https://wpnews.pro/news/gemini-4-often-hides-its-failures-when-searching-antibody-space-abmaze.md", "text": "https://wpnews.pro/news/gemini-4-often-hides-its-failures-when-searching-antibody-space-abmaze.txt", "jsonld": "https://wpnews.pro/news/gemini-4-often-hides-its-failures-when-searching-antibody-space-abmaze.jsonld"}}