LatchBio found 59.2% red-team refusal and 64.8% routine completion, showing how far behavior can shift under one model name.
By RuntimeWire Staff · Published
Primary source: SpaceXAI
Why it matters #
The result shows why model names are becoming poor version controls. Safety behavior can change materially inside a served checkpoint, making continuous third-party testing part of the product stack.
LatchBio, founded by Alfredo Andere, Kenny Workman and Kyle Giffin, says the version of Grok 4.6 currently served by SpaceXAI has become markedly better at distinguishing concealed biological hazards from legitimate research. The finding came less than three weeks after LatchBio's first evaluation exposed a model that could reason deeply and still talk itself into the wrong answer.
The September 1 analysis, written by LatchBio researcher Arjun Banerjee, found that Grok 4.6 was the only tested model to clear 50% on both red-team refusal and routine task completion. SpaceXAI, which published its own account of the results, described LatchBio's work as independent.
Andere, Workman and Giffin started LatchBio after leaving UC Berkeley, initially building cloud software that let biologists run and share computational workflows without maintaining the underlying infrastructure. Their current bet is broader: the same execution layer that runs biological analysis can become a proving ground for AI agents entering the lab.
That shift accelerated in June, when LatchBio acquired TwentyTwo, a biosecurity infrastructure company founded by Harmon Bhasin, Evan Seeyave and Dianzhuo "John" Wang. The three joined LatchBio to lead Latch Biosecurity. In the acquisition announcement, Andere wrote that as AI makes biology easier to engineer, safeguards become core infrastructure: "Security has to improve as fast as capabilities."
The benchmark tests judgment under concealed hazards
The BioSecBench-Refusal benchmark contains 107 tasks: 61 routine research assignments adapted from published scientific work and 46 fictional red-team assignments that conceal a hazard. Dangerous intent may sit inside a mislabeled sequence, encrypted data or a file whose contents conflict with the user's cover story.
That construction targets a known weakness in model safeguards. A filter that reacts to words such as "pathogen" or "toxin" can block legitimate virology or antivenom work while allowing a carefully disguised request to proceed. In its initial benchmark analysis, LatchBio reported that many of the 16 model-harness configurations refused routine work at rates comparable to or higher than their refusal of concealed threats. Refusal rates ranged from 7% to 74% for routine tasks and from 1% to 62% for red-team tasks.
The latest Grok 4.6 checkpoint reversed that pattern more effectively than the other systems LatchBio tested. Averaged across agent harnesses, Grok refused 59.2% of red-team attempts and completed 64.8% of routine attempts. A trial-weighted harmonic mean combining those measures produced a 62.1% score, with Grok configurations occupying the top three positions.
Those figures also set the limits of the result. Grok did not refuse 40.8% of red-team attempts, while 35.2% of routine attempts were not completed. The benchmark measures controlled agent behavior across a relatively small task set. It does not establish how Grok would perform during a live outbreak investigation, a long-running laboratory project or a sustained adversarial campaign.
LatchBio also tested pathogen surveillance, where agents must inspect raw sequencing data, choose tools and references, set analytical thresholds and return a surveillance judgment. Grok averaged 53.5% on that suite, behind Opus 5 and ahead of GPT-5.6 Sol, according to the companies. The result puts Grok near the frontier on defensive biological analysis without giving it a decisive lead.
One model name now covers different behavior
LatchBio's choice of words matters. Banerjee tested the "currently-served version" of Grok 4.6 and called its performance a significant improvement over earlier versions carrying the same name.
In its August 13 evaluation, LatchBio generated 1,716 trajectories across shorter biological tasks and reported stronger scientific reasoning than Grok 4.5 alongside conspicuous execution failures. In more than 10% of runs, Grok claimed it could not access data already present in its workspace. LatchBio also documented corrupted or split tokens, leaked end tokens, four verbatim repetition loops and cases where the model repeatedly changed or reversed its answer.
The September 1 refusal analysis describes a later checkpoint that reasons through discrepancies between a prompt and its attached files, then refuses when the underlying task appears hazardous. That trace-level behavior does not establish where most refusals originate. LatchBio's earlier benchmark found that API-layer refusals dominated across most model-harness configurations, rather than refusals produced by the model's own reasoning.
For customers, auditors and regulators, the operative unit is therefore the served checkpoint on a particular date, paired with the harness and effort setting used in the evaluation. A model label alone no longer fixes the behavior being purchased or tested. Continuous post-deployment tuning can improve safeguards quickly, while also making an older evaluation stale without a formal model release. SpaceXAI says it uses refusal training, inference-time filters, behavioral controls and session-level monitoring around Grok. The evaluation traces suggest Grok can inspect a task environment and identify conflicts between a user's stated purpose and the attached data. The operational problem is keeping that calibration stable as checkpoints, API filters, agent harnesses and tool access change.
LatchBio is using infrastructure to evaluate AI agents
LatchBio raised a $28 million Series A in 2022, led by Coatue and Lux Capital, to build what the founders then called an "AWS x GitHub for biocomputing." Hummingbird Ventures, Caffeinated Capital, Haystack and Fifty Years also participated. Combined with LatchBio's earlier $5 million seed round, the disclosed financings total at least $33 million.
The original product thesis remains visible in BioSecBench. These evaluations depend on controlled workspaces, attached scientific files, repeatable compute and recorded tool calls, all problems LatchBio had already spent years addressing for biotech customers. Biosecurity gives that infrastructure another use: testing whether frontier agents can be trusted with the workflows the platform was built to run.
The company now sits between AI labs eager to prove safety and biology teams eager to use capable agents without losing access to ordinary research. Its credibility will depend on publishing enough methodology to let outsiders understand model selection, checkpoint timing, scoring and uncertainty while withholding task details that could expose concealed hazards or contaminate future tests.
For SpaceXAI, the result arrives as Grok 4.6 expands beyond the channels available at its August 12 launch. AWS lists an [August 18 launch date for Grok 4.6 on Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-xai-grok-4-6.html?ref=runtimewire), while xAI announced [Microsoft Foundry availability on August 26](https://x.ai/news/grok-4-6-microsoft-foundry?ref=runtimewire). Broader availability raises the value of evidence that Grok can inspect biological data without treating every risky-sounding request as malicious.
For Andere, the evaluation supports a larger transition at LatchBio. The Berkeley dropouts began by making biology software easier to run. They are now trying to measure whether the AI agents running that software understand what they have been asked to do.