Key Takeaways
- NVIDIA’s DGX B200 benchmark shows confidential computing retaining 96.1-98.2% of output-token throughput at concurrency levels 1 through 16, with per-token latency within 1.2-4.3% of the unprotected baseline.
- Keeping Blackwell fast under CC required CC-aware fixes: pageable memory instead of pinned buffers for host-to-device transfers, GPU %globaltimer for autotuner stability, and dropping NVLS multicast for topology-matched collectives.
- Because CPU-only TEEs like Intel TDX and AMD SEV-SNP cannot close GPU-memory and PCIe exposure for model weights, accelerator-level enclaves are now the practical path to compliant, high-performance private inference.
Table of Contents
The 96% Threshold: Confidential Inference Stops Being a Performance Tax
Confidential computing has long carried a reputation as the secure option that slows everything down.
New benchmark data from NVIDIA flips that assumption for production-grade large language model inference.
On an eight-GPU DGX B200 system, enabling NVIDIA Confidential Computing retained between 96.1 and 98.2 percent of output-token throughput across concurrency levels from 1 to 16.
Mean time per output token stayed within 1.2 to 4.3 percent of the unprotected baseline.
The test workload was not a toy model.
It paired TensorRT-LLM with the DeepSeek-R1-0528-NVFP4 model under a 32K input and 1K output sequence configuration.
The results, published by NVIDIA’s developer blog on September 22, come from a controlled CC-on versus CC-off comparison.
The implication is direct: private inference no longer requires a double-digit performance penalty on Blackwell GPUs.
Inside the CC-Aware Engineering That Keeps Blackwell Fast
Secure execution changes the assumptions behind memory movement, timing, scheduling, and multi-GPU communication.
NVIDIA’s engineering team selected a workload that would expose those changes instead of hiding them.
Long context stresses prefill data movement, extended output generation amplifies per-token overhead, and low concurrency limits the system’s ability to mask costs.
The controlled setup used the following workload.
- Model: nvidia/DeepSeek-R1-0528-NVFP4
- Inference framework: TensorRT-LLM with PyTorch backend
- I/O sequence length: 32K input and 1K output
- Concurrent requests: 1, 2, 4, 8, and 16
- Parallelism: tensor parallel 8, expert parallel 1, pipeline parallel 1
- KV cache: FP8
Engineers held the model, hardware, framework version, sequence lengths, parallelism, and concurrency constant while toggling only the confidential computing state.
That isolation turns the comparison into a precise overhead measurement rather than a vendor guess.
At each concurrency level, output throughput retention was calculated as one hundred times CC-on tokens per second divided by CC-off tokens per second.
Latency overhead followed a similar formula for time per output token.
The hardware and software stack remained identical across both modes.
- Hardware: one NVIDIA DGX B200 system with eight NVIDIA B200 GPUs
- Platform: Intel TDX
- Host OS: Ubuntu 25.10 with kernel 6.17.0-20-generic
- Guest OS: Ubuntu 24.04.4 LTS with kernel 6.8.0-124-generic
- Guest vCPUs: 256 across two NUMA nodes
- NVIDIA driver: 595.71.05
- CUDA: 13.2
- TensorRT-LLM container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22
CC-Aware Memory Selection Changes Host-to-Device Movement
On B200 confidential GPUs, host-to-device transfers pass through a software-encrypted bounce buffer because the GPU cannot directly access protected CVM memory.
That removes the usual asynchronous advantage of pinned memory and can block the calling thread.
TensorRT-LLM compensates by selecting pageable memory for affected paths instead of unconditionally relying on pinned buffers.
For device-to-host reads, repeated token and sampling data moves to an asynchronous worker so the scheduler does not stall during decode.
The relevant TensorRT-LLM change is documented in pull request 11573.
Autotuner Stability Moves to GPU Globaltimer
Kernel autotuning normally uses CUDA events to compare candidate tactics.
Under the tested confidential configuration, CUDA-event timestamps became unstable and could push the autotuner toward a slower kernel.
TensorRT-LLM switches to the GPU %globaltimer for tactic measurements inside confidential mode while preserving CUDA events outside it.
The change appears in TensorRT-LLM pull request 11657.
Multi-GPU Communication Drops NVLS and Chooses a Different Path
NVLS multicast is not available in B200 confidential configurations.
That means NCCL_SYMMETRIC cannot deliver its intended multicast benefit but can still pay memory registration and cross-rank synchronization costs before falling back to a non-multicast collective.
Frameworks operating in this environment must detect NVLS availability and pick communication algorithms matched to message size, topology, and workload characteristics.
Why Model-Weight Protection and TEE Economics Are Reshaping Production AI
The benchmark matters because inference workloads now regularly process proprietary model weights, enterprise context, and regulated personal data simultaneously.
Storage and network encryption do not close the ‘data in use’ gap; model weights still sit in plaintext inside GPU memory during inference.
Private AI Review notes that confidential inference servers use hardware enclaves to close exactly that exposure.
RAND Corporation’s May 2024 report catalogued 38 distinct attack vectors across nine categories for model-weight theft.
The same analysis estimated that an amateur attacker has under a 20 percent chance of exploiting a given ML-stack vulnerability, while a well-resourced nation-state clears an 80 percent chance.
The sharpest competitive tension sits between CPU-only TEEs and accelerator-level protection.
Intel TDX and AMD SEV-SNP can secure the virtual machine, but they cannot close the PCIe and GPU-memory exposure for LLM weights.
NVIDIA’s approach extends isolation into the GPU itself through a Compute Protected Region walled off from the host OS and privileged administrators.
Hopper introduced GPU confidential computing in mid-2023 and reached general availability in 2024.
Blackwell then extends the protection envelope to inter-GPU traffic, covering an attack surface Hopper left partly open.
Independent benchmarks on H100 hardware placed typical LLM query overhead below 7 percent in a 2024 arXiv study.
That same study found the overhead concentrated in time to first token rather than inter-token latency.
NVIDIA’s vendor-reported figures for H100 and H200 confidential computing already ranged from 95 to 99 percent of native throughput.
The new B200 figures come from NVIDIA’s own benchmark team, so they are best read as vendor-published results with a clearly stated methodology.
Alternatives such as homomorphic encryption and multi-party computation remain impractical for interactive LLM serving.
A 2025 analysis concluded that trusted execution environments are the only viable method for protecting large-scale inference at interactive latency today.
Homomorphic encryption can run two to three orders of magnitude slower than plaintext for complex operations, while multi-party computation cannot meet low-latency inference demands.
Even private operation or a provider promise not to train on inputs does not make an AI service confidential.
Production deployments require hardware-backed attestation, memory isolation, and controlled key release, not just an encrypted connection.
Trusted execution environments do not eliminate every risk.
Hardware bugs, software flaws, side-channel leakage, social engineering, supply-chain compromise, and physical access all remain outside the enclave boundary.
The Deployment Mandate for Private Inference
The 96 to 98 percent throughput window changes the calculus for teams serving proprietary models under regulatory pressure.
Security configuration and inference optimization must now be treated as a single full-stack engineering problem, not competing priorities.
For teams building AI inference performance and security content at scale, Andres SEO Expert’s programmatic SEO AI automation turns technical benchmarks into compounding organic authority — contact Andres SEO Expert.
Frequently Asked Questions
What is NVIDIA Confidential Computing for AI inference?
NVIDIA Confidential Computing is a hardware-backed approach that isolates AI workloads inside a protected GPU environment. It helps keep model weights, enterprise context, and regulated data encrypted and inaccessible to the host OS, privileged administrators, and other tenants during inference.
How much performance overhead does confidential inference add on NVIDIA B200 GPUs?
On an eight-GPU DGX B200 system, NVIDIA Confidential Computing retained 96.1 to 98.2 percent of output-token throughput across concurrency levels from 1 to 16. Mean time per output token stayed within 1.2 to 4.3 percent of the unprotected baseline.
What is the 96 percent threshold for confidential inference?
The 96 percent threshold refers to benchmark results showing confidential inference can preserve about 96 percent or more of native LLM throughput. It signals that private inference no longer requires a double-digit performance penalty on Blackwell GPUs for production-grade workloads.
Why is confidential computing important for protecting model weights?
Storage and network encryption do not protect data in use. During inference, proprietary model weights can sit in plaintext inside GPU memory. Confidential computing uses hardware enclaves to close that exposure, which matters as RAND documented 38 model-weight theft attack vectors across nine categories.
How does TensorRT-LLM optimize confidential inference on Blackwell?
TensorRT-LLM uses CC-aware memory selection for host-to-device transfers, switches autotuner timing to the GPU globaltimer in confidential mode, and adapts multi-GPU communication when NVLS multicast is unavailable. These changes reduce overhead while preserving throughput.
What are the limitations of trusted execution environments for private AI inference?
TEEs protect data in use but do not eliminate every risk. Hardware bugs, software flaws, side-channel leakage, social engineering, supply-chain compromise, and physical access remain outside the enclave boundary. They also require hardware-backed attestation, memory isolation, and controlled key release.
How do CPU-only TEEs compare with GPU confidential computing?
Intel TDX and AMD SEV-SNP can secure the virtual machine, but they do not close PCIe and GPU-memory exposure for LLM weights. NVIDIA extends isolation into the GPU through a Compute Protected Region, and Blackwell adds protection for inter-GPU traffic that Hopper left partly open.
