ai and ml
Sandboxes, permissions, and VMs aren't enough to keep frontier models at bay. "Data diodes" might do the job
To prevent frontier AI models breaking out of test environments and collaborating to hack other companies, we may have to rethink the network architectures used for model training.
Eli-Shaoul Khedouri, CEO of Intuition Machines, argues that past work in the defense and intelligence communities shows the way forward.
Rogue AI models rise from the level of developer regret to mass threat when they gain access to the internet, something that defenses erected by OpenAI and its partners tried but failed to prevent.
Pointing to a post published by his company's hCaptcha service, Khedouri argues that technology like data diodes – hardware that enforces a one-way flow of information on a network – can be deployed to prevent security incidents like OpenAI's hack of Hugging Face.
The hCaptcha team points to the use of data diodes as a data transit mechanism at a sensitive compartmented information facility (SCIF), an environment implemented in classified settings.
"They allow files, logs, or telemetry to enter or exit the SCIF's classified network to an unclassified network and provide a way to prove e.g. that logs from a training run can only flow one way," the hCaptcha team explains, adding that such technology fits with the Bell-LaPadula security architecture designed for the US Defense Department.
A basic implementation would involve two machines connected via network cards linked by one-way optical fiber – and without a data path back to the model.
Training and reinforcement learning could run in an isolated zone with no internet access and an optical ingress diode would grant access only to vetted artifacts. The hCaptcha researchers suggest a second diode to send telemetry to a sel4 receiver and scrubber, while a separate out-of-band network manages the cluster.
This sort of scheme would require immutable snapshots of software registries like PyPI, GitHub, npm, and might also need mocked versions of various web services and APIs .
This would come at a cost, and would require implementation time that frontier AI labs may not be prepared to spend at the moment.
"The systems described are widely deployed in high assurance domains, and the components are commercially available," Khedouri told The Register. "However, they have not been adopted by frontier labs to date."
He pointed to NIST and DOD guidelines [PDF] that suggest an overhead of 10 to 20 percent for applying formal specifications and system architecture.
"We estimate total cost overhead for high assurance training clusters at less than five percent per gigawatt, but in practice the speed at which frontier labs are moving is a greater impediment than cost," Khedouri said.
The current commercial environment, he said, would make it difficult for any one AI lab to delay its training to build and test effective safeguards, particularly if their competitors might not do the same.
"In practice, the cost of high assurance on the systems side would be a small fraction of what OpenAI is now spending on the new monitoring, chain-of-thought oversight, and safety safeguards they introduced after their models hacked HuggingFace," he said.
While Khedouri is focused primarily on convincing frontier labs to implement better training defenses, he said other organizations may want to consider similar network architecture. "This architecture is designed for entities training models with frontier cyber capabilities," he said. "Recently that has only included two companies. However, this is rapidly becoming relevant to smaller organizations and individuals who train models.
"'Abliterated' open weight models with safeguards removed are readily available and now starting to approach the frontier in cyber abilities, and (reinforcement learning) RL post-training has become much more approachable in the past year."
The goal of high assurance system design in the context of model training is to make unwanted action physically impossible, Khedouri said.
"Attempting to monitor the behavior of an untrustworthy agent is an AGI-hard problem, as models have poor interpretability and this appears to be getting worse as their capabilities increase and they become more evaluation-aware," he said.
"Hardware can provide limited guarantees like the direction in which data can be sent, but cannot solve the problem of untrustworthy models with the ability to reach the internet and find new exploits in their environment.
"This is why combining physical one-way data flows (data diodes) and formal verification (to prove properties of the receiver) is much more effective than running a software sandbox on a host connected to the internet."
hCaptcha is focused on fraud and abuse work and has no plan to offer a model-resistant training stack. Khedouri said he shared this advice in the hope it helps AI firms that don't have experience with high assurance system design. ®