Key Takeaways
- NVFP4 compression cuts Nemotron 3.5 Lightning from 66 GB to 22 GB, boosting throughput up to 4x.
- QAD (quantization-aware distillation) restores accuracy lost by aggressive 4-bit compression, approaching BF16 levels.
- Benchmark gains are concentrated in agentic and coding tasks, making 4-bit inference viable for production.
Table of Contents
NVFP4 Compression Cuts Nemotron 3.5 Lightning From 66 GB to 22 GB
A technical deep dive published yesterday on NVIDIA’s developer blog details how the Nemotron 3.5 Lightning checkpoint drops from 66 GB to 22 GB while unlocking up to 4x faster throughput.
The compressed NVFP4 model preserves accuracy close to a BF16 baseline across the published benchmark suite.
The method is quantization-aware distillation, which pairs a full-precision teacher with a quantized student to recover lost accuracy.
This is not a routine post-training quantization result.
Aggressive W4A16 compression pushes far enough that conventional PTQ leaves measurable gaps across reasoning and coding benchmarks.
A second distillation stage closes most of that gap without changing the model’s footprint.
Inside the Two-Stage Pipeline That Keeps Accuracy Alive
Quantization-aware distillation starts with a frozen BF16 teacher model.
Stage one runs post-training quantization to produce a low-precision student checkpoint with weights compressed to W4A16 NVFP4.
Stage two trains that student against the teacher using a KL divergence loss on logits.
The student’s forward pass runs through simulated quantization, so it learns to operate in the same noisy regime it will face at inference.
This dual signal forces the student to reproduce the teacher’s full behavior rather than simply predicting the next token.
Two scale strategies define how quantization parameters behave during training.
- Dynamic scale QAD: starts from a max-calibrated PTQ checkpoint and recomputes weight and activation scales on the fly.
- Frozen scale QAD: starts from an MSE-based PTQ checkpoint and freezes calibrated scales while only weights update.
The choice follows directly from the PTQ recipe.
Max-calibrated recipes feed dynamic scale training because their scales are designed to adapt.
MSE-based recipes such as ‘four_over_six’ lock scales during distillation because recomputing them at every step would be too expensive.
Aggressive quantization targets Mamba linear layers at W4A16, a choice that would be unsafe under PTQ alone.
That decision is deliberate: the pipeline banks size and latency gains first, then lets QAD close the accuracy gap.
Training ablations show sequence length is critical for long-context benchmarks.
The final QAD run scaled to 522K tokens after initial experiments at 256K.
Constant learning rate, no warmup, disabled dropout, and gradient clipping at 1.0 define the distillation loop.
The recipe runs across two nodes with eight GPUs, using tensor parallel 2 and expert parallel 4.
The complete recipe ships inside NVIDIA Model Optimizer as a single end-to-end launcher.
Megatron-Bridge offers a parallel PyTorch-native path for teams already working inside the NeMo Framework.
Benchmark Reality Check: Where QAD Wins and Where It Does Not
The most revealing data comes from intermediate checkpoints pushed to aggressive quantization.
Checkpoint A saw post-training quantization land at 96.33% median score recovery.
QAD lifted that figure to 99.72% using the same 21.19 GB footprint.
AIME 2025 recovered from a 3.70-point drop to within 0.57 points of BF16.
Checkpoint B repeated the pattern, moving from 95.84% to 98.53% median recovery.
On Checkpoint B, the AA v4.1 Index rose 3.45 points from 20.03 to 23.48 against a BF16 baseline of 24.81.
The final shipping checkpoint tells a more nuanced story.
Because its quantization was deliberately conservative, PTQ already sits at 99.24% median recovery.
QAD lands at 98.97%, slightly lower on aggregate.
Its gains are concentrated in agentic and coding benchmarks instead.
Terminal-Bench v2.1 rises by 3.79 points under QAD, and SWE-Bench Multilingual adds 1.07 points.
HLE, BrowseComp, SWE-Bench Verified, and PinchBench all show smaller gains.
This is not a blanket victory for QAD across every metric.
The technique earns its place where aggressive compression would otherwise be unacceptable, or where long-running agent performance matters more than median score.
What NVFP4 Means for Blackwell-Scale AI Deployment
NVFP4 is not merely a smaller numeric container.
The format uses a shared exponent and compact mantissa, with fine-grain block scaling: one FP8 scale per 16 values plus a second FP32 whole-tensor scale.
This design is documented in NVIDIA’s official Model Optimizer guidance for DGX Spark and DGX Station deployments.
NVIDIA Build positions NVFP4 as reducing memory by roughly 3.5x versus FP16 and 1.8x versus FP8, with accuracy losses usually under 1 percent against FP8.
That accuracy figure remains vendor-reported and has not yet been independently benchmarked across broad model families at production scale.
A separate official demonstration from NVIDIA’s Developer channel shows the format scaling far beyond Nemotron 3.5 Lightning, as shown in the following demonstration:
In that walkthrough, Nemotron 3 Ultra shrinks from 1 TB to 350 GB using a ’46 scaling config’ across two B300 nodes.
Calibration runs through 512 passes and completes in about 25 minutes, but not all layers use NVFP4.
TensorRT-LLM documentation confirms NVFP4 support is currently limited to Blackwell sm120 and sm100/103 GPUs.
Hopper, Ada Lovelace, and Ampere hardware do not support the format.
The same documentation lists NVFP4 model support for DeepSeek-R1, Llama, Mixtral, and Qwen families, but does not yet mention Nemotron 3.5 Lightning explicitly.
That gap means production teams will need to verify the QAD-exported checkpoint against their own inference stack before assuming drop-in compatibility.
The wider NVFP4 story is moving faster than any single model card.
NVIDIA’s own deployment examples span DeepSeek-R1 distillations, while the Nemotron 3.5 Lightning recipe is published separately through Model Optimizer and Megatron-Bridge.
Adopting teams must therefore distinguish between format-level support and recipe-level support.
The Precision Calculus Has Shifted for Inference Teams
The Nemotron 3.5 Lightning QAD recipe proves that 4-bit quantization does not automatically mean accepting visible quality loss. For teams building AI automation pipelines that must account for model efficiency and deployment constraints, Andres SEO Expert’s programmatic SEO and AI automation service is how the firm approaches it — contact Andres SEO Expert to map the workflow.
Frequently Asked Questions
What is NVFP4 compression and why does it reduce Nemotron 3.5 Lightning from 66 GB to 22 GB?
NVFP4 is a 4-bit floating-point format using a shared exponent, compact mantissa, and fine-grain block scaling. Applying NVFP4 to the Nemotron 3.5 Lightning checkpoint cuts its storage from 66 GB to 22 GB while enabling up to 4x faster throughput. The compression is paired with quantization-aware distillation so the smaller model stays close to the BF16 accuracy baseline.
How does quantization-aware distillation differ from standard post-training quantization?
Standard post-training quantization directly converts a trained model to lower precision, which can leave measurable accuracy gaps under aggressive W4A16 compression. Quantization-aware distillation adds a second stage: a frozen BF16 teacher supervises the quantized student using KL divergence loss on logits, and the student’s forward pass runs through simulated quantization. This dual signal trains the student to reproduce the teacher’s behavior in the noisy low-precision regime it will encounter at inference.
What is the difference between dynamic scale QAD and frozen scale QAD?
Dynamic scale QAD starts from a max-calibrated PTQ checkpoint and recomputes weight and activation scales on the fly during distillation. Frozen scale QAD starts from an MSE-based PTQ checkpoint and keeps the calibrated scales fixed while only weights update. The choice depends on the PTQ recipe: max-calibrated scales are designed to adapt, while MSE-based scales such as four_over_six are frozen because recomputing them at every step is too expensive.
Which GPUs and model families support NVFP4?
TensorRT-LLM documentation confirms NVFP4 support is limited to Blackwell sm120 and sm100/103 GPUs; Hopper, Ada Lovelace, and Ampere hardware do not support it. Documented model support currently includes DeepSeek-R1, Llama, Mixtral, and Qwen families, but Nemotron 3.5 Lightning is not explicitly listed yet, so production teams must verify the exported checkpoint with their own inference stack before deployment.
Does QAD improve benchmark scores across every metric?
No, QAD is not a blanket victory on every benchmark. For aggressively quantized checkpoints, it can lift median recovery from roughly 96 percent to 99 percent, with strong recoveries on AIME 2025 and the AA v4.1 Index. On the final shipping checkpoint, PTQ already reaches 99.24 percent median recovery and QAD lands at 98.97 percent on aggregate, but QAD still produces concentrated gains on agentic and coding benchmarks such as Terminal-Bench v2.1 and SWE-Bench Multilingual. The technique earns its place where aggressive compression would otherwise be unacceptable or where long-running agent performance matters more than median score.
What training recipe was used for the Nemotron 3.5 Lightning QAD run?
The final QAD run scaled to 522K token sequence length after initial experiments at 256K, using a constant learning rate, no warmup, disabled dropout, and gradient clipping at 1.0. The recipe ran across two nodes with eight GPUs using tensor parallel 2 and expert parallel 4. It is available as an end-to-end launcher in NVIDIA Model Optimizer, with a PyTorch-native Megatron-Bridge path for teams in the NeMo Framework.
How much memory reduction does NVFP4 provide versus FP16 and FP8?
NVIDIA positions NVFP4 as reducing memory by roughly 3.5x versus FP16 and 1.8x versus FP8, with accuracy losses usually under 1 percent against FP8. The accuracy figure is vendor-reported and has not yet been independently benchmarked across broad model families at production scale. NVIDIA’s demonstration also shows Nemotron 3 Ultra shrinking from 1 TB to 350 GB using a 46 scaling config across two B300 nodes.
