PyTorch Gets Native FP8 Training for AMD Instinct GPUs

AMD’s FP8 Training Pipeline Becomes a Native PyTorch Citizen

AMD, Meta, and PyTorch engineers have upstreamed a full set of FP8 training optimizations so that AMD Instinct GPUs now deliver competitive out-of-the-box performance in TorchTitan and TorchAO. The technical breakdown, published this week by PyTorch, closes a silent numerical correctness gap tied to AMD’s FP8 variant while unlocking double-digit throughput gains on dense and MoE architectures.

First shown at PyTorch Conference 2025 with linear scaling beyond 1,000 GPUs using the Primus-Turbo library, the work is now merged directly into mainline pytorch/ao and pytorch/torchtitan. Teams with AMD Instinct GPUs can access the gains by upgrading both projects with no AMD-specific installation required.

The effort rests on three pillars: native support for AMD’s e4m3fnuz FP8 format, grouped GEMM enablement for Mixture-of-Experts routing on ROCm, and a Triton kernel fusion pipeline that reduces data movement at multiple levels. The sharpest result landed on DeepSeek-V3 671B shapes, where a single fused forward-pass kernel erased 89 percent of the FP8 quantization penalty.

Inside the Kernel Fusion Work That Recovered 89 Percent of MoE Quantization Overhead

As detailed in the PyTorch technical breakdown, FP8 training quantizes the forward pass, gradient input, and gradient weight update for every linear layer. The bottleneck for MoE models is not 8-bit arithmetic but the memory traffic and kernel launches that surround it.

Key performance results from the upstreamed contributions include:

• Llama3-8B dense: 13.4 percent higher throughput than BF16 with rowwise FP8.
• DeepSeek-MoE-16B backward pass: 4.2x throughput gain from transpose removal and Triton fusion.
• DeepSeek-V3 671B MoE layer: 6.2x colwise scales speedup, from 7,290 to 1,170 microseconds.
• DeepSeek-V3 671B forward pass: 17 percent end-to-end throughput improvement, from 5,996 to 7,027 tokens per second.

Correctness First: AMD’s FNUZ FP8 Format

AMD Instinct GPUs implement the e4m3fnuz FP8 variant, which caps representable values at 240 and removes NaN and Inf encodings entirely. TorchAO initially assumed a different format, leading to values scaled beyond the hardware-valid range without any error signal.

The correction introduced hardware auto-detection so TorchAO selects the proper FP8 dtype and maximum value on AMD platforms instead of hardcoding NVIDIA e4m3fn behavior. Additional fixes corrected MI300X peak FLOPS reporting and added platform-specific loss baselines for FNUZ numerics.

Grouped GEMM for Mixture-of-Experts

MoE architectures like DeepSeek V3 and Llama 4 route tokens to variable-size expert batches. Grouped GEMM therefore needs per-row activation scales, per-expert-column weight scales, and offset-based routing to match each token to the right expert.

The ROCm path was enabled by adapting TorchAO’s quantization pipeline to use the correct dtype and dispatch through the Composable Kernel backend. That removed a major architectural blocker for FP8 training on sparse expert models.

Three-Level Kernel Fusion Attack

In the backward pass, removing a redundant transpose-contiguous-transpose pattern and fusing the scale-and-cast chain into single Triton kernels produced the 4.2x DeepSeek-MoE-16B gain. On the forward path, a five-kernel eager chain repeated across experts added roughly 90 ms of overhead per step.

A single fused Triton kernel that parallelizes across experts and output-dimension blocks replaced the entire forward quantization chain. That change drove the 17 percent end-to-end improvement on DeepSeek-V3 671B with 8x MI325X GPUs.

At the memory-access level, the colwise scales kernel had non-coalesced writes that cost 7,290 microseconds per MoE layer. Transposing the output tile through Local Data Share and adding a fused single-pass variant cut that figure to 1,170 microseconds.

Triton atomic operations default to acquire-release memory ordering, which inserts unnecessary fences around commutative reductions on AMD GPUs. Switching to relaxed ordering behind a torch.version.hip guard removed those fences while leaving NVIDIA behavior untouched.

One attempted optimization did not survive. Expanding the Triton autotune search space from one to up to 16 candidate configurations delivered no measurable improvement on Llama 4 shapes and increased first-iteration compile time, so the change was reverted.

What Independent FP8 Research Signals for AI Training Economics

Independent validation is arriving just as the open-source training stack matures. A research paper published August 2 by Hong Kong Polytechnic University and InfiX.ai presents an end-to-end FP8 training recipe for reasoning-enhanced language models, keeping master weights, weight gradients, and optimizer states in FP32 while quantizing weights block-wise and activations token-wise.

Across 160B-token pre-training runs on Qwen2.5 models, that independent work reports up to a 22 percent training time reduction, a 14 percent peak memory decrease, and a 19 percent throughput improvement. Backward pass computation time dropped by up to 32 percent, and training loss curves stayed nearly identical to BF16.

The paper does not discuss AMD GPUs, TorchTitan, or TorchAO, which leaves the AMD and Meta collaboration as the clearest evidence for this specific integration. Still, the independent results confirm that FP8 training economics are broad enough to matter across model families and frameworks.

On the deployment side, AMD’s official Day-0 support guidance for Alibaba’s Qwen 3.8 MoE family, released this week, shows how quickly FP8 on AMD is moving into production. Qwen 3.8’s 512-expert sparse architecture runs on MI300X, MI325X, and MI355X GPUs with FP8 and MXFP4 precision through ROCm, SGLang, and vLLM.

AMD reports that its SGLang FP8 deployment reaches 97.4886 percent GSM8K accuracy at 112.693 tokens per second of output throughput, while MXFP4 hits the same accuracy at 1,060.45 tokens per second. Those are vendor-published deployment numbers, but they underline that FP8 and MXFP4 are no longer experimental on AMD silicon.

The Road to MI355X and MXFP8 Is Already Paved

The team is already building MXFP8 grouped GEMM and quantization kernels for forward and backward passes on MI355X GPUs, making FP8 efficiency a default for the next generation of AMD AI hardware. For teams producing technical SEO and content systems around AI infrastructure shifts like FP8 training, programmatic SEO and AI automation is how Andres SEO Expert turns technical momentum into discoverable authority — start the conversation.