MXFP8 FlashAttention-4 Hits 2.85 Petaflops in Meta’s Production Training

FP8 Attention Moves Into Production on Blackwell

On September 16, 2026, PyTorch published a technical breakdown of a production-grade extension to FlashAttention-4 that runs end-to-end MXFP8 block-scaled attention on NVIDIA Blackwell GPUs.

The implementation reaches 2.85 petaflops per second in forward pass on LLM-shaped workloads and about 2 petaflops per second in backward.

Meta’s engineering teams have already deployed the kernel in GEM ad recommendation model training, making it one of the first state-of-the-art MXFP8 FlashAttention-4 forward and backward implementations used inside a live production workload.

The code is open-sourced inside Meta’s Ads Model Kernel Library, as documented in PyTorch’s technical breakdown.

Inside the TMEM Bargain and Zero-Gather Jagged Pipeline

Blackwell’s tensor cores introduce block-scaled matrix multiply instructions that operate natively on microscaling formats, delivering two to four times the throughput of BF16 matrix multiply.

That raw speed only materializes when scale factors live in tensor memory, when every operand is quantized along the correct contraction dimension, and when precision conversion overhead stays hidden behind full-speed math.

• TMEM allocation: Scale factors reuse accumulator regions under existing or added barriers.
• Online quantization: Post-softmax activations and dS are converted in registers with warp-wide reductions.
• Fused producers: RMSNorm and projection GEMM epilogues emit FP8 without separate passes.
• Jagged handling: Only scale-factor tensors are padded and permuted; FP8 data stays compact.

The forward kernel computes scores, softmax, and output from query, key, and value tensors.

Because tensor memory has only 512 columns and is already fully occupied, the engineering team overlaps scale-factor regions with accumulator lifetimes.

For example, score scales reuse the output region before the output GEMM starts, while output scales share the score region under an existing barrier.

One new barrier protects score scales that overlap the alternate score accumulator.

The team also improved an unroll-KV optimization that interleaves the last PV GEMM of one tile with the first QK GEMM of the next, hiding tile-boundary latency.

That change previously regressed BF16 by delaying a correction warp; moving the barrier wait after row-sum computation fixed it.

Online conversion of post-softmax activations from FP32 to MXFP8 uses 32-element blocks, three-instruction max operations, and bit extraction instead of logarithms or division.

The softmax row maximum is reused for scale computation, avoiding extra max passes across most elements.

For jagged sequence lengths, the design uses split addressing: compact FP8 data stays unpadded while only the much smaller scale-factor tensors are scattered and aligned for tensor memory access.

This avoids a full tensor gather, which would have inflated memory traffic.

Backward requires five GEMMs and a transpose-invariant quantization scheme.

Query, key, and output gradient operands use square 32-by-32 blocks, so the same E4M3 payload serves both transposed and non-transposed matrix multiplies.

The dS intermediate is quantized once per square block with a warp-wide absolute maximum reduction.

The dQ reduction step moves to FP16 with static scaling after ablation studies showed a global memory bandwidth bottleneck.

Finally, quantization is fused into RMSNorm and projection GEMM epilogues, eliminating separate global memory passes.

On B200 module shapes, fusing quantization into the GEMM epilogue raises throughput from roughly 0.20 petaflops per second to 0.91 to 0.98 petaflops per second, a 4.4 to 4.7 times speedup.

Fusing RMSNorm with quantization lifts effective bandwidth from 0.75 to 0.87 terabytes per second up to 2.6 to 4.0 terabytes per second.

What the Benchmarks Show About Low-Precision Training

On common LLM shapes, the MXFP8 forward kernel reaches 2.85 petaflops per second, nearly identical to cuDNN 9.24 MXFP8 at 2.82 petaflops per second and 1.43 times faster than the internal BF16 baseline at 2.00 petaflops per second.

Backward throughput lands at 1.82 petaflops per second with FP32 dQ accumulation and 1.98 petaflops per second with FP16 accumulation, representing 1.21 times and 1.32 times gains over the cuDNN BF16 reference.

On Meta’s internal recommendation shapes with jagged keys at maximum length 16,384, the kernel reaches 2.54 petaflops per second, a 1.59 times gain over the internal BF16 kernel and within 4 percent of cuDNN MXFP8.

With uniform keys, the figure is 2.59 petaflops per second, 1.51 times faster than the BF16 counterpart.

The backward pass reaches 1.42 petaflops per second with FP32 dQ and 1.58 petaflops per second with FP16 dQ, up to 1.52 times faster than the internal BF16 baseline.

One important caveat appears in the cuDNN MXFP8 backward comparison.

Nonuniform E8M0 value scales did not match the reference for dQ and dK, and the discrepancy disappeared only after switching to uniform value scales.

The published 1.76 petaflops per second cuDNN backward result is therefore labeled performance-only rather than accuracy-qualified.

Two competitive signals emerge from the benchmark set.

First, cuDNN 9.24 stays within striking distance on internal forward shapes, trailing or leading by around four percent.

Second, the cuDNN backward result is not accuracy-qualified under nonuniform V scales, while the FA4 fork exposes quantized gradients instead of BF16 gradients.

That shifts downstream memory and precision decisions for teams building on this path.

At the module level, speedups scale with token count.

For 4,096 KV tokens, the median latency is essentially unchanged at 1.00 times.

At 8,192 tokens, MXFP8 delivers 1.18 times lower latency; at 16,384 tokens, the gain reaches 1.30 times.

That trajectory matters because smaller recommendation shapes remain bandwidth-bound and need fused kernels to avoid memory traffic, not just faster math.

In FP32 dQ configuration, SQNR values ranged from 18.84 dB to 28.02 dB, while the strict summary recorded 23 passing tensors, 1 borderline, and 0 failures.

Productionization for GEM training required numerical stabilization techniques inspired by SageAttention3 and earlier low-precision attention analyses.

The Kernel Stack That Changes the Next Training Cycle

The MXFP8 FlashAttention-4 implementation proves that block-scaled attention on Blackwell is a systems engineering problem, not a one-line datatype swap.

It also gives teams a reusable open-source starting point for LLM and recommendation training pipelines.

For teams working where AI infrastructure meets search visibility, programmatic SEO AI automation is how Andres SEO Expert turns complex technical shifts into ranking authority — contact us.