Rust framework for deploying attested, immutable workloads in trusted and confidential computing environments on Kubernetes with support for Microsoft Azure, Google Cloud, and AMD SEV-SNP bare-metal, alongside NVIDIA H100 GPUs in confidential mode
Introduction #
Artificial Intelligence is a bigger part of everyday life than ever before. The range of tasks people bring to AI has grown remarkably broad. People have daily interactions with AI systems, for example through chatbots, where they ask all sorts of questions ranging from asking medical questions to drafting client emails, debugging code, or summarizing reports.
This shift is more than a convenience; it represents a fundamental change in how we seek information, make decisions, and even communicate.
As AI becomes more integrated with our lives and more capable, it raises important questions such as:
- Can you trust the answers returned by the model?
- Are your interactions with the AI system private?
- Is your private information used to train and improve the AI systems?
Given these concerns, companies started banning the use of tools such as ChatGPT to prevent employees from leaking corporate data. A prime example is Samsung, which in 2023 blocked access to AI chatbots after an employee accidentally leaked confidential source code (source).
Employees frequently find workarounds, whether by switching to newer, unblocked AI services or using their mobile phones to photograph and process screen content, effectively bypassing IT safeguards altogether.
These persistent workarounds suggest that banning AI to employees is not the way to go and would impact their productivity. Instead of fighting usage, some organizations are looking toward a structural solution: deploying AI models on-premise. By hosting models on internal servers, companies can ensure that data never leaves their internal network and have full control over them.
However, this level of control comes at a high price, as the infrastructure and expertise required to maintain high-performance clusters are expensive, and it’s a cost that not every organization can afford.
Other companies would prefer to deploy to open cloud platforms such as Microsoft Azure or Google Cloud Platform, but the question remains: how can you be sure that the cloud provider is not acting maliciously?
Because the Cloud Provider owns the infrastructure and the hypervisor where your VM is running, there is nothing technically stopping them from exfiltrating the contents of the RAM of your VM.
By hosting infrastructure on the cloud, you grant the provider physical and administrative control. The Cloud Service Provider (CSP) controls the hypervisor, which is the piece of software that controls the lifecycle and manages the memory of your virtual machine, and it’s the key component responsible for isolating your virtual machine from the other tenants.
Despite standard security measures, two critical issues persist:
- In this model, data can be protected in transit (for example with TLS) and at rest (thanks to storage encryption), but it becomes vulnerable as “data in use” because it has to be decrypted to be used by the program running on the virtual machine. Given that the cloud provider controls the hypervisor, which manages the memory of your virtual machine, it has access to your data unencrypted while it is processed. For this reason, workflows involving sensitive data are often pulled back to on-premises private clouds.
- As a user, you have no way of knowing whether you can trust the application you are interacting with, and you can’t know if the application has been backdoored. The padlock icon in the top-left corner of the browser only guarantees a secure “pipe”—that is, your communication to the server is encrypted and private in transit—but it tells you nothing about the security of the destination, specifically the integrity of the application running on the server you’re connected to.
So cloud solutions are good as long as you decide to trust the cloud provider, which is a risk not everyone is willing to take; however, confidential computing technology is here to eliminate this need for blind trust.
Introduction to Confidential Computing #
To try to address these problems, Cloud Service Providers (CSPs) have begun offering confidential virtual machines that rely on a set of technologies designed to execute sensitive workloads in shared environments.
Confidential Computing provides two key solutions to eliminate the need for blind trust:
- Protection of Data in Use (Confidentiality) : Computation is isolated in a segregated environment called an enclave. This TEE (Trusted Execution Environment), implemented by technologies like AMD SEV-SNP (AMD Secure Encrypted Virtualization (SEV) ), guarantees complete data privacy from the host (including the hypervisor) because memory is at all times encrypted. Data is only decrypted inside the secure boundary of the CPU during processing.
- Verification of Application Integrity (Trust) : This is achieved through remote attestation (Remote attestation - Edgeless Systems wiki ), which provides the user with the ability to verify the authenticity and integrity of the services they run. The user receives a report from the enclave, including cryptographic measurements of its state, that can be checked to verify that the enclave can be trusted. A user starts interacting with the enclave only after having verified its attestation.
In this scenario, the Cloud Provider still owns the infrastructure and runs the hypervisor, but the memory is cryptographically encrypted, so it’s unreadable from the perspective of the Cloud Provider.
Confidential Computing promises to provide the technical guarantees needed to run remote workloads with the same level of confidentiality and integrity as an on-premise deployment, even for sensitive data and workloads from untrustworthy CSPs and malicious tenants sharing the same cloud infrastructure.
Towards confidential and transparent AI #
Confidential computing gives us the building blocks necessary to run sensitive workloads on cloud infrastructure. In 2023, Mithril Security released BlindLlama, a project showcasing a confidential AI chatbot running on a cloud-hosted virtual machine. By retrieving a cryptographic attestation report linked directly to the secure hardware, end-users could mathematically verify the integrity and trustworthiness of the application they were interacting with.
By validating this attestation, users achieved full transparency: they could verify not only the isolation of the virtual machine itself, but also the exact version of the OS, the specific application running, and the AI model being served. However, a major architectural limitation of BlindLlama was its restriction to a single virtual machine; it did not support attesting a distributed cluster of VMs. Given that a single virtual machine can be attached to at most 8 GPUs, it means that the most powerful models can’t be used.
For this reason, we started the development of Fluorite.
Fluorite #
Fluorite is an open-source, transparent Kubernetes distribution designed for deploying attested, unalterable workloads in trusted computing and confidential computing environments.
Built on top of K3s, a lightweight Kubernetes distribution, Fluorite provides a framework for running distributed enclave applications across a cluster.
The key features of Fluorite:
- Cluster-wide attestation : users can request and verify the cryptographic attestation of the entire cluster.
- Zero privileged access post-bootstrap : Once provisioning of the cluster is complete, the operator (Ops and dev team) retains no privileged access to the cluster and can’t modify its configuration.
- Build transparency : All artifacts are built with verifiable provenance using the SLSA framework to guarantee supply chain security and the transparency of the build artifacts we provide. This includes the application running on the attested cluster.
- Secure inter-node communication : Nodes communicate securely via WireGuard encrypted tunnels.
- Multiple cloud support : Deploy on Microsoft Azure (Trusted Launch VMs and Confidential VMs), Google Cloud Platform (Shielded VMs), or bare-metal using AMD SEV-SNP hardware.
- GPU support : Out-of-the-box support for NVIDIA GPUs, including H100s with confidential computing mode. This enables Fluorite to be used for AI workloads. For example, our reference architecture features a fully confidential AI chat web application.
By eliminating privileged access after deployment, Fluorite enables the creation of unalterable secure services. These services guarantee the integrity and confidentiality of both computation and user data, and provide protection against insider threats by allowing their functionality to be remotely verified.
The multi-node cluster. The nodes communicate via a Wireguard encrypted channel. Each node owns a set of Nvidia GPUs running in confidential mode (CC).
Architecturally, your application is deployed across the cluster, and K3s handles distributing the workload across the nodes. Each worker node operates as an isolated, confidential virtual machine with attached NVIDIA GPUs running in secure mode. The end user only ever interacts with the master control plane node, which acts as the secure ingress point for the entire cluster.
Fluorite does not add any additional performance overhead compared to other solutions using AMD SEV-SNP and NVIDIA GPUs in confidential mode. In fact, research has shown that this hardware-level confidential computing configuration introduces less than a 7% overhead compared to standard, non-confidential environments (see: [Zhu et al., 2024] Confidential Computing on nVIDIA H100 GPU: A Performance Benchmark Study).
Furthermore, verifying the attestation report is a one-time operation performed when the interaction with the system begins; it never sits inline with your live data processing or model inference.
Learn more about how Fluorite works and how you can deploy it yourself by visiting our documentation at https://mithril-security.notion.site.
The code is open source and available at the following link: https://github.com/mithril-security/fluorite.
Demo #
We have provisioned a bare-metal instance running with AMD SEV-SNP and an Nvidia H100 running in confidential mode (CC mode). On this machine, we deployed an instance of LibreChat, where you can interact directly with an open-weight LLM (Qwen/Qwen2.5-7B-Instruct).
The demo is located at https://chat.demo.mithrilsecurity.io/. Once you register, you will be able to chat with the LLM. Registration does not require a valid email address. We do not send confirmation links, and we will never have access to your personal information. The application is deployed with Ray, a framework that supports cross-node tensor parallelism (TP) and pipeline parallelism (PP), which distribute model inference across multiple GPUs and nodes. This way, even larger models can be hosted across nodes while staying confidential.
We configured Fluorite to run the entire operating system and all server applications entirely in RAM. This stateless, ephemeral architecture means there is no persistent disk attached to the machine where user conversations could ever be logged or recovered. Furthermore, the RAM itself is entirely encrypted at the hardware level thanks to AMD SEV-SNP.
On the demo page, you will find a link to our verification tool, which allows you to independently audit the attestation of the machine running the demo. Under the hood, this tool pulls the cluster’s cryptographic attestation report directly from the live demo and verifies its measurements against expected reference values. Combining a standard TLS handshake with a Remote Attestation report is what we at Mithril Security refer to Attested TLS. Technical information on where the attestation of the demo is stored, how it is retrieved and checked, is available in the documentation.
This mathematically proves that the OS and application running the demo match exactly what we built on our CI pipeline, and that the underlying hardware configuration is a genuine virtual machine running on bare-metal with AMD SEV-SNP enabled.
Conclusions #
With Fluorite, we showcased how it is possible to deploy an attested, multi-node cluster of virtual machines across diverse cloud providers and bare-metal environments, all while leveraging Nvidia GPUs running in confidential mode. Fluorite represents the evolution of the foundations laid by the BlindLlama project, bringing multi-node scalability and broader hardware configurations.
Fluorite was developed by the Mithril Security Contributors. This work was commissioned and funded by the nonprofit organization Future of Life Institute.
Fluorite is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt).
[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)by the author.