{"slug": "hardware-rooted-ai-security-that-wont-slow-you-down", "title": "Hardware-Rooted AI Security That Won’t Slow You Down", "summary": "NVIDIA announced that its Confidential Computing technology for Blackwell GPUs achieves up to 98% of the inference performance of non-secure solutions, enabling hardware-rooted AI security without significant slowdown. The technology protects enterprise data and proprietary models during inference through hardware trust, attestation, and optimized software frameworks.", "body_md": "AI has transformed how organizations operate, driving unprecedented levels of productivity and innovation. However, AI adoption can be impeded by concerns surrounding data privacy, sovereignty and how to secure data while it is in use, or during inference and engagement with AI models. NVIDIA Confidential Computing (CC) was engineered to be a secure and performant solution for the era of agentic AI to scale any model securely.\n\nCC enables the protection of enterprise data and proprietary model weights and the model itself during active inference. In this post, we will provide an overview of CC and demonstrate benchmarks that show its inference performance is nearly identical (up to 98%) to solutions that don’t enable CC security.\n\n**Data, code, and model integrity**\n\nCC provides a security layer that spans silicon, interconnect, and system software. Here’s how it works:\n\n### Hardware root of trust\n\nNVIDIA Blackwell GPUs, including the NVIDIA RTX PRO 6000, HGX B200, and HGX B300, are engineered with CC embedded in the hardware. The HGX B200 and HGX B300 GPUs support confidential computing across multiple GPUs (up to 8) with NVIDIA NVLink encryption. At the silicon level, the GPU maintains a private signing key that is fused at the time of manufacturing and never exposed to software, firmware, or the host system. This key is the foundation of the attestation chain.\n\n### Attestation: Verification before execution\n\nBefore a confidential workload receives any secrets, it undergoes remote attestation. The NVIDIA Remote Attestation Service (NRAS) verifies a signed evidence bundle—the GPU’s hardware report combined with CPU TEE measurements (AMD SEV-SNP or Intel TDX)—against a known-good reference integrity manifest (RIM).\n\nOnce the Confidential VM (CVM) is in a verified, unmodified state, secrets such as model decryption keys can be deployed into the CVM. The attestation handshake is typically a one-time startup event. Once the workload is running, attestation does not add latency to individual inference requests.\n\n*Figure 2. Attestation services remotely validate the identity, configuration, and integrity of Trusted Execution Environments and issue cryptographic proof*\n\n## Optimizing AI inference performance in Confidential Computing\n\nCC changes to AI inference performance on Blackwell GPUs can come from two areas:\n\n**Secure work submission latency:** For inference, secure work submission latency is often the larger factor and due to the added overhead from encryption and kernel launches, smaller units of work are more affected. Increasing the amount of work performed per GPU work launch reduces the impact of the secure launch overhead.**Reduced host-to-device CPU-to-GPU bandwidth:** If a workload depends heavily on transferring inputs to the GPU, performance will depend on whether the required bandwidth to keep the GPU fully utilized exceeds the encrypted transfer bandwidth available in CC mode.\n\nSeveral innovations optimize inference performance with CC including:\n\n**CC-safe autotuner timing:** FlashInfer replaces event timers in CC mode with the GPU global timer register, allowing autotuners to accurately compare kernel candidates and select the fastest implementation for each shape.**Async D2H copy worker:** SGLang moves per-step token readback off the scheduler’s critical path. This helps restore compute/copy overlap because CC can otherwise make many host-to-device and device-to-host copies effectively synchronous during cudaMemcpyAsync.**Piecewise CUDA graph support:** SGLang adds CUDA graph replay for prefill and mixed batches, reducing kernel launch overhead that is amplified in CC mode.\n\nNVIDIA continues to work with upstream communities for inference frameworks to ensure these frameworks are optimized for performance.\n\nWe measured the inference performance of CC across different key metrics. Below are the details on the test setup and measurements.\n\n## Benchmark results\n\n*Across all workload configurations tested, enabling CC mode produced minimal throughput and time per output token overhead during steady-state inference.*\n\nThe following table summarizes CC throughput, TTFT, TPOT overhead on Blackwell Ultra (HGX B300) for model Qwen/Qwen3.5-397B-A17B-FP8\n\n### Relative Performance of Confidential Computing\n\nConcurrency | ISL/OSL = 1024 / 1024 | ISL/OSL = 8192 / 1024 | ||\nThroughput/GPU (tok/s) | Median TPOT (ms) | Throughput/GPU (tok/s) | Median TPOT (ms) | |\nΔ% vs OFF | Δ% vs OFF | Δ% vs OFF | Δ% vs OFF | |\n4 | -2.0% | -1.6% | -3.5% | -3.6% |\n8 | -2.6% | -2.4% | -2.8% | -2.9% |\n16 | -5.3% | -4.9% | -2.8% | -3.0% |\n32 | -6.3% | -7.8% | -1.0% | -0.9% |\n64 | -6.2% | -6.8% | -2.3% | -2.4% |\n128 | -7.5% | -8.1% | -3.5% | -3.5% |\n256 | -4.6% | -4.1% | -3.6% | -3.7% |\n\n*Table 1. Relative performance impact of enabling NVIDIA Confidential Computing*\n\n## Test Setup\n\n**Benchmark:** Qwen 3.5 397B-A17B model at FP8 precision**Environment:** Virtual Machine with GPU passthrough**Baseline:** Confidential Computing Off**Experiment:** Confidential Computing On\n\nAll other variables held constant.\n\n**Hardware Configurations**\n\nHGX B300 with Blackwell Ultra.\n\n**Software Stack**\n\nComponent | Version / Detail |\nPlatform | Intel TDX |\nHost OS | Ubuntu 25.10 |\nHost Kernel | 6.17.0-20-generic |\nGuest OS | Ubuntu 24.04.4 LTS |\nGuest Kernel | 6.8.0-124-generic |\nGuest vCPUs | 256 |\nGuest NUMA | 2 nodes |\nNVIDIA Driver | 595.71.05 |\nVBIOS | FW 1.4.x [97.10.64.00.0C] |\nGPU Power Limit | 1100.00 |\nCUDA | 13.2 |\nSGlang |\n|\n\n**NCCL****OpenSSL****Orchestration*** Table 2. Software configuration for test setup*\n\nNote: Please follow the CPU power and vCPU pinning configuration described in this [document.](https://docs.nvidia.com/cc-deployment-guide-tdx.pdf)\n\n**Workload Parameters**\n\nEach configuration was tested across a range of conditions representative of real enterprise inference workloads:\n\n**Input/output token lengths: **8192/1024, 1024/1024** Batch sizes:** 4, 8, 16, 32, 64, 128 and 256 concurrent requests. **Inference framework (Mode):** SGLang (Server)**Baseline:** Without –enable-symm-mem\n\n**Metrics Collected**\n\n**Output Throughput per GPU (tokens/sec/gpu)****Median Time to First Token (TTFT)** — latency from request submission to first token generated, in ms**Median Time Per Output Token (TPOT)** — per-token generation latency in steady-state streaming, in ms\n\n**Path forward**\n\nHardware-level security with CC protects sensitive AI workloads while preserving the performance needed for production AI workloads.\n\nCC provides a stronger security foundation for production inference workloads with minimal performance overheads. In our evaluation using Qwen 3.5 on SGLang, we observed this across a sweep of concurrency levels, input sequence lengths, and output sequence lengths, proving that organizations can secure their AI workloads and data, and stay compliant to regulation without compromising on performance.\n\nJoin NVIDIA and our partners to secure your AI workloads with CC on Blackwell by accessing the resources below.\n\n**Resources**\n\n[NVIDIA Confidential Computing Documentation](http://developer.nvidia.com/confidential-computing)[NVIDIA Blackwell Architecture Whitepaper](http://resources.nvidia.com/en-us-blackwell-architecture)[NVIDIA GPU Operator and Container Toolkit](http://docs.nvidia.com/datacenter/cloud-native/)[NVIDIA Remote Attestation Service (NRAS)](http://developer.nvidia.com/docs/security/attestation-sdk)[NIST SP 800-207 Zero Trust Architecture](http://csrc.nist.gov/publications/detail/sp/800-207/final)[HIPAA Security Rule (HHS)](http://hhs.gov/hipaa/for-professionals/security)[GDPR Article 32 — Security of Processing](http://gdpr.eu/article-32-security-of-processing)", "url": "https://wpnews.pro/news/hardware-rooted-ai-security-that-wont-slow-you-down", "canonical_source": "https://developer.nvidia.com/blog/hardware-rooted-ai-security-that-wont-slow-you-down/", "published_at": "2026-07-02 21:25:42+00:00", "updated_at": "2026-07-07 00:55:20.990488+00:00", "lang": "en", "topics": ["ai-safety", "ai-infrastructure", "ai-chips", "ai-research"], "entities": ["NVIDIA", "Blackwell", "RTX PRO 6000", "HGX B200", "HGX B300", "NVLink", "FlashInfer", "SGLang"], "alternates": {"html": "https://wpnews.pro/news/hardware-rooted-ai-security-that-wont-slow-you-down", "markdown": "https://wpnews.pro/news/hardware-rooted-ai-security-that-wont-slow-you-down.md", "text": "https://wpnews.pro/news/hardware-rooted-ai-security-that-wont-slow-you-down.txt", "jsonld": "https://wpnews.pro/news/hardware-rooted-ai-security-that-wont-slow-you-down.jsonld"}}