Long-Context Interactivity Reset: Groq 3 LPX Hits 3,431 Tokens/s

A Third-Party Benchmark Resets the Long-Context Interactivity Bar

NVIDIA reports that Groq 3 LPX has reached a median speed of 3,431 output tokens per second on the Artificial Analysis 100K context benchmark using the Gemma 4 31B model.

The measurement, published August 24, 2026, is the first third-party benchmark of the accelerator on the Vera Rubin NVL72 platform.

That speed at 100,000 tokens of context changes the calculus for multi-turn agentic sessions, where context balloons across hundreds of turns.

As reported by NVIDIA’s developer blog, the same system was also measured at 10K context, producing a median of 3,382 output tokens per second.

The narrow gap between 10K and 100K scores signals that long context no longer forces a steep interactivity penalty.

Inside the Deterministic Compiler Architecture That Cuts First-Bit Latency

Serving a model above 3,000 tokens per second while maintaining a 100K-token KV cache creates a coordination problem that bandwidth alone cannot solve.

At the tiny batch sizes required for high interactivity, tensor parallelism can introduce more overhead than it removes.

Groq 3 LPX attacks that overhead through a deterministic execution model.

The compiler sees every one of the 256 LP30 local processing units, the 128 GB of collective SRAM, and the 96 chip-to-chip links per chip running at 112 Gbps.

It uses that visibility to plan every transfer down to the clock cycle before the workload begins.

That pre-scheduling eliminates real-time arbitration, so data can leave on the exact cycle it is ready and be consumed on the exact cycle it arrives.

Each LPU can also act as a router, allowing point-to-point traffic to pass through other LPUs when needed.

The result is a sharp reduction in first-bit latency, the fixed cost of selecting a link, synchronizing endpoints, and resolving conflicts.

In small-batch inference, first-bit latency looms far larger than the actual transfer time.

By minimizing that fixed cost, the architecture makes tensor parallelism viable in the exact low-batch regime where it normally collapses.

Fine-grained overlap compounds the advantage.

Because matrix multiplication can be expressed as a series of dot products, the compiler schedules compute and communication on 320-byte vector boundaries.

An LPU can transmit a 320-column output slice immediately after computing it, rather than waiting for the full matrix operation to finish.

This increases overlap between computation and communication and cuts the communication tail that plagues small tensors.

Why 100K Context Changes Agentic Coding and Multi-Agent Economics

Agentic workloads are inherently multiturn. Every generated response is appended to the context, so the next turn must process more history than the one before.

After hundreds of turns, the model may be replaying a substantial portion of the entire session before producing each new token.

Limited context forces agents to operate with only a fraction of prior context, regardless of the underlying model’s speed or intelligence.

The benchmark data shows that Groq 3 LPX holds its interactivity nearly flat from 10K to 100K context.

At 10K, the median was 3,382 output tokens per second; at 100K, it was 3,431.

That stability matters for coding agents that read hundreds of files and routinely exceed 100K context tokens while generating thousands of reasoning and output tokens.

On the open-source SPEED-Bench coding benchmark, the same Gemma 4 model on Groq 3 LPX produced a median of 4,767 output tokens per second and a P80 of 5,520.

More than 20 percent of tasks finished at over 5,500 tokens per second.

At 3,431 tokens per second, decoding 5,000 tokens takes roughly 1.5 seconds.

The same generation at 100 tokens per second takes about 50 seconds, and many widely used agentic tools currently sit closer to 60 tokens per second.

That difference determines how many coding iterations a developer can afford before attention and productivity collapse.

Co-Execution Playbooks: Vera Rubin NVL72 and LPX Disaggregation

Groq 3 LPX does not replace Vera Rubin NVL72; it gives the platform a low-latency partner for specific serving tiers.

The technical breakdown defines three co-execution configurations.

• Standard prefill-decode disaggregation: Vera Rubin NVL72 handles prefill and hands off the KV cache once per turn, while Groq 3 LPX executes the decode step from weights held in SRAM.
• Attention-FFN disaggregation: Vera Rubin NVL72 computes attention and stores the KV cache in DRAM, while Groq 3 LPX runs the feed-forward layers; intermediate tokens cross between racks once per full-attention layer.
• External-drafter speculative decoding: Groq 3 LPX runs a small draft model ahead of a large target model on Vera Rubin NVL72, which verifies and commits tokens and returns rejected positions for the next chunk; each rack keeps its own KV cache and only draft tokens cross the link.

These modes let each rack focus on the portion of the workload it executes best.

The same co-design scales toward a projected GPT-OSS model at 2 trillion parameters, pushing the Vera Rubin platform into high-interactivity agentic territory for very large models.

A New Inference Tier Emerges for AI Factories

A benchmark at 100K context now signals a new inference tier where long context and high interactivity no longer have to be traded against each other. For teams building long-context agentic inference pipelines that need to scale, programmatic SEO and AI automation is how Andres SEO Expert approaches it — talk to the team.