Cloudflare Cracks the Memory Wall for Giant Open-Source AI Models

Cloudflare Cracks the Memory Wall for Giant Open-Source AI Models

Cloudflare’s engineering team has released a deep technical account of how it serves some of the world’s most demanding open-source large language models — Moonshot’s Kimi K-series and Z.ai’s GLM — at global scale on Workers AI.

The breakthrough lies not in new hardware but in three software-layer optimizations: quantizing the KV cache from 16-bit to 8-bit floating point, compressing model weights from 8-bit to 4-bit integers for decoding, and deploying a low-overhead integrity layer that prevents cache corruption when hundreds of concurrent requests share GPU memory.

The upshot: Cloudflare can now run these mixture-of-experts long-context models with up to 1.37 million tokens of context on a single cluster, delivering 41% higher aggregate throughput while cutting cost-per-token by roughly 30%, all without any measurable accuracy loss.

Quantizing the KV Cache and Shrinking Weights — How the Stack Works

The memory bottleneck for long-context models is rarely the model’s own weights.

It is the KV cache, a structure that holds attention keys and values for every token already processed, that balloons with conversation length and fills GPU memory first.

By default, that cache is stored at 16-bit precision. Cloudflare moves it to 8-bit floating point, slicing its footprint in half.

On Kimi K2.6, this lifts the maximum contextual capacity from around 686,000 tokens to 1.37 million tokens without changing the underlying hardware.

A direct performance comparison on a disaggregated H200 deployment shows that at any single concurrency level, BF16 is a few percent faster per token.

But BF16 runs out of cache at 32 concurrent requests. FP8 scales to 64 concurrent requests and reaches 2,192 tokens per second — roughly 41% higher aggregate throughput than BF16’s peak.

Because Cloudflare runs prefill and decode as separate pools, the precision choice can be applied surgically: prefill remains compute-bound, so it keeps the slightly higher throughput of BF16, while decode adopts the memory-efficiency of FP8.

Benchmark checks across GSM8K, ARC, MMLU, MMLU-Pro and internal evaluations confirm that FP8 and BF16 caches are indistinguishable, with accuracy differences falling within fractions of a percentage point.

Weight compression adds a second memory-saving lever.

For the GLM 5.2 model, Cloudflare shrinks the weights from 8-bit floating point down to 4-bit integers, reducing the checkpoint size from 705 GB to 421 GB — a 40% cut.

Across an 8-way tensor-parallel setup, per-GPU memory consumption drops from roughly 88 GB to 52 GB, freeing room for about 1.18 million tokens of KV cache on the same device.

Again, accuracy remains virtually unchanged: benchmark results for GSM8K, ARC, MMLU, and MMLU-Pro stay within 0.8 points of the FP8 baseline, and the model passes 62 out of 63 internal exam questions in both configurations.

Smaller weights make the decode phase faster because generating each token requires streaming model weights out of GPU memory, an operation limited by memory bandwidth.

At low concurrency — where per-request latency matters most — the INT4 decode throughput for GLM jumps by 55%, from 60 to 92 tokens per second.

As with the KV cache, the disaggregated architecture lets Cloudflare use INT4 for the memory-bound decode stage and FP8 for the compute-bound prefill stage, extracting the best of each precision.

With memory pressure reduced and many more requests sharing GPU resources, the risk of cache-corruption bugs rises.

To guard against it, Cloudflare built a KV cache integrity checking system that tags every physical cache page and validates those tags before each decode operation reads from the cache.

If a mismatch is detected, the affected request is aborted rather than allowed to return data from the wrong page.

On a production-model deployment handling 8,192-token inputs and 1,000-token outputs, the integrity layer adds less than 1% overhead on both throughput and p95 latency.

The check is executed as a separate batch validation rather than fused into the attention kernel, avoiding race conditions and keeping the cost negligible.

Beyond Quantization: Lossless Compression and the Future of GPU-Efficient Inference

Cloudflare’s published optimizations focus on quantization, but the company’s broader inference strategy leans on another layer: lossless compression of model tensors.

The ACM SIGARCH community recently surveyed techniques that exploit the sparsity of floating-point exponent bits to reduce data volume without any accuracy trade-off, achieving roughly 30% size reduction on average.

Cloudflare’s own contribution, the Unweight lossless decompressor for NVIDIA H100 GPUs, directly applies these ideas, as noted in the SIGARCH analysis.

That places Cloudflare’s serving pipeline at the intersection of two complementary paths — aggressive quantization for the memory-bound decoding phase and lossless compression across the data path — while the broader research ecosystem pushes similar techniques toward network, NVLink, and memory interconnects.

The SIGARCH survey highlights methods like DFloat11, which uses offline Huffman coding to cut weight sizes by about 30% and accelerate weight offloading, and Invariant Bit Packing, which generalizes to runtime-generated KV caches.

Systems like ZipServ and ZipNN have demonstrated double-digit throughput gains inside GPU memory, confirming that lossless compression is becoming a practical, not just theoretical, lever for serving infrastructure.

Cloudflare’s roadmap points toward expanding FP8 KV caches across more of its fleet, validating NVFP4 weights on NVIDIA’s upcoming Blackwell architecture, and driving the integrity-checking overhead toward a permanent, near-zero cost across all deployments.

That forward-looking posture signals that serving frontier open-source models efficiently will remain a moving target, and that the balance between memory capacity, latency, and safety will define competitive advantage for any platform scaling AI inference.

From Data Center to Edge: What Efficient Model Serving Unlocks for AI’s Next Decade

What makes Cloudflare’s engineering account notable is not just the raw throughput numbers but the architectural discipline: every byte of GPU memory is treated as a resource to be stretched across more customers without degrading the model’s intelligence.

That same philosophy — ruthless memory optimization, precision-switching at the right stage, and low-cost safety layers — will likely become the benchmark for any cloud provider serious about running large open-source models at scale.

For digital businesses, the underlying lesson is that infrastructure efficiency directly unlocks better user experiences and lower costs, whether you are serving a billion-token conversation or a high-traffic website.

Andres SEO Expert brings the same performance-obsessed engineering mindset to WordPress speed optimization, turning memory-like constraints into measured gains in Core Web Vitals and user-perceived load times through deep technical tuning and custom caching strategies.

Where the infrastructure edge meets cloud-native hosting, the same principles of workload disaggregation and resource orchestration apply, and our managed WordPress cloud environments are built to deliver that level of reliability and throughput under real-world traffic loads.

To see how that technical rigor translates to digital performance for your own projects, connect with Andres and learn more about our approach.