Key Takeaways
- EPD disaggregation delivers up to 5x faster time-to-first-token and 7x faster end-to-end response on image-heavy, short-output multimodal workloads.
- EPD gains are workload-dependent: longer outputs and larger models (e.g., 27B) can erase the advantage, while quantized MoE models amplify it.
- The pattern is now built into vLLM, SGLang, and Dynamo; mixed text-image traffic sees up to 42% lower TTFT.
Table of Contents
Five Times Faster First Tokens Hinge on One Architectural Split
NVIDIA’s Dynamo team has published a new benchmark that reframes multimodal serving as a scheduling problem, not a model problem.
On September 9, 2026, the team documented how encode-prefill-decode disaggregation can deliver up to five times faster time to first token and seven times faster end-to-end response on the open-source Dynamo inference framework.
The pattern works best for image-heavy prompts, short-to-medium output lengths, and quantized mixture-of-experts models.
The mechanism is not a new model architecture.
It is an operational split that moves vision encoding off the worker that handles prefill and decode.
Where Disaggregation Changes the Serving Math
Encode-prefill-decode disaggregation splits a multimodal request into three stages: media goes through a vision transformer to produce embeddings, prefill consumes those embeddings, and decode generates tokens.
In a conventional aggregated worker, those stages share a single scheduling domain on one GPU.
That design works while media processing is light, but heavy image or video inputs can stall prefill and decode work behind vision encoding.
Dynamo supports three placement choices: aggregated, colocated encoder workers, and disaggregated encoder workers on a lower-cost GPU tier.
The colocated pattern keeps encoder workers and a prefill-decode worker on the same GPU while giving them independent queues.
The disaggregated pattern uses a separate encoder tier and transfers embeddings to the primary GPUs over NIXL.
- Input media load: More images or higher resolution raises visual-token volume, creating the encoder bottleneck EPD removes.
- Output sequence length: Longer outputs move latency toward decode, shrinking end-to-end gains even when TTFT stays strong.
- Model size and precision: Smaller, mixture-of-experts, and quantized LLMs give the vision encoder a larger share of total compute, which makes EPD more valuable.
- Traffic mix: Mixed text and multimodal workloads avoid head-of-line blocking because text requests can bypass encoder work.
In an image-heavy benchmark with ten images per request and output length 1,024 tokens, the colocated configuration cut mean TTFT by 58 percent.
The heterogeneous setup reduced TTFT by 50 percent and served 70 percent more traffic under the same 100-millisecond inter-token latency SLO.
Those TTFT gains held across a much wider media-load sweep.
With five to fifty images per request, aggregated serving latency climbed steeply as visual token counts rose, while both EPD topologies stayed nearly flat.
Output length is the boundary that many teams miss.
When output length increased from 128 to 2,048 tokens with five images per request, heterogeneous EPD’s end-to-end advantage narrowed from 20.3 percent to 5.2 percent.
Colocated EPD moved from an 11.8 percent gain to a 2.5 percent regression because decode pressure began competing with encoder work on the same GPUs.
Model size produces a similar boundary.
Across Qwen3.5 configurations, colocated EPD delivered 2.62x goodput for a 4B model, 1.50x for 9B, and 0.65x for 27B.
The 27B result fell below break-even because the LLM’s prefill and decode work left too little isolated encoder compute to justify the coordination overhead.
Quantization shifts that threshold in EPD’s favor.
When both the LLM and vision encoder ran in BF16, colocated EPD reached 1.78x the goodput of aggregated serving.
Quantizing only the active LLM weights to NVFP4 raised that figure to 2.64x because the vision encoder remained in BF16 while prefill and decode accelerated.
Mixed traffic showed the cleanest operational benefit.
Under a 50:50 text-image mix at output length 128, EPD reduced mean TTFT for text requests by 42.2 percent and for image requests by 30.8 percent.
Text requests improved more because they could enter LLM prefill without waiting for vision encoding to complete.
Three adjacent levers widen the optimization window in Dynamo.
Parallel media decoding moves download and decode work into the frontend, cutting mean encoder request latency by 26 percent in an encoder-only benchmark.
Embedding cache reuses embeddings for repeated media content, and multimodal KV routing sends requests that share media to workers already holding relevant KV blocks.
From a Single Benchmark to a Cross-Stack Production Pattern
EPD has moved from a framework-specific optimization to production infrastructure across competing serving stacks.
The surrounding ecosystem is now building the scheduling, routing, and failure-handling layers that make disaggregation viable at scale.
Z.ai’s GLM-5.3-Flash announcement describes the model as the first natively multimodal release in the GLM-5 series, with 320B total parameters and 18B active parameters.
Its production serving architecture on SGLang separates multimodal encoding, prompt prefill, and token-by-token decoding into independently scalable worker pools.
The vendor reports a 3x end-to-end serving improvement on Chinese AI chip hardware, a figure that is compelling but remains tied to that specific hardware baseline.
The Dynamo figures are more aggressive on image-heavy TTFT and end-to-end responses, but the two claims are not directly comparable across different baselines, chips, and traffic patterns.
Research from arXiv adds another layer: disaggregation needs smarter scheduling, not just separate workers.
An SLO-aware scheduler called Kairos improved TTFT SLO attainment by up to 23.9 percent and time-per-output-token attainment by up to 27.1 percent over a DistServe baseline.
The study framed EPD as one part of a larger disaggregated inference problem in which prefill and decode require different scheduling policies.
The vLLM Ascend release notes show the pattern crossing into Chinese AI hardware ecosystems, with EPD documentation and a load-balance proxy example added in release v0.17.0.
Later release notes carry the operational warnings: asymmetric DCP settings for sparse-attention models can create accuracy issues, and high-concurrency prefill-decode deployments may hang under heavy load.
Both vLLM and SGLang have roadmaps for further EPD development, which reinforces that the pattern is not a single-vendor effect.
It is becoming a default assumption in inference platform design.
What the Next Inference Build Should Assume
Multimodal serving is no longer a question of whether to separate vision work from language work, but where to draw the split based on media load, output length, model size, and precision. For teams building AI inference or automation pipelines that need to scale with the same systems-level discipline, programmatic SEO AI automation is how Andres SEO Expert applies this efficiency logic — start the conversation here.
Frequently Asked Questions
What is encode-prefill-decode (EPD) disaggregation in multimodal serving?
Encode-prefill-decode disaggregation is an operational split that separates multimodal requests into three stages: vision encoding, prefill, and decode. Rather than sharing a single scheduling domain on one GPU, the vision transformer runs as a separate worker or tier, and embeddings are transferred to prefill-decode GPUs over NIXL. This prevents heavy image or video inputs from stalling prefill and decode behind vision encoding.
How much faster is NVIDIA Dynamo with EPD for image-heavy workloads?
On an image-heavy benchmark with ten images and 1,024 output tokens, colocated EPD reduced mean time-to-first-token by 58 percent and heterogeneous EPD by 50 percent while serving 70 percent more traffic under a 100-ms inter-token latency SLO. Dynamo reports up to five times faster TTFT and seven times faster end-to-end response for image-heavy prompts and short to medium output lengths.
When does EPD provide the largest advantage?
EPD has the strongest impact when media load is high, outputs are short-to-medium, and the language model is small, mixture-of-experts, or quantized. High visual-token counts make encoding the bottleneck, while longer output sequences shift latency toward decode and shrink the gain. Quantizing the LLM to NVFP4 while keeping the vision encoder at BF16 improved goodput from 1.78x to 2.64x compared to an aggregated baseline.
What happens to EPD gains as model size or output length grows?
With five images per request, moving output length from 128 to 2,048 tokens reduced heterogeneous EPD’s end-to-end advantage from 20.3 percent to 5.2 percent; colocated EPD went from an 11.8 percent gain to a 2.5 percent regression. Across Qwen3.5 configurations, colocated EPD delivered 2.62x goodput for the 4B model, 1.50x for 9B, and 0.65x for 27B, so the 27B model fell below break-even.
What complementary techniques can be combined with EPD?
Parallel media decoding cuts mean encoder request latency by 26 percent by moving download and decode work to the frontend. Embedding cache reuses embeddings for repeated content, and multimodal KV routing routes requests that share media to workers already holding relevant KV blocks. These levers widen the optimization window in Dynamo and can be used with either EPD placement.
Is EPD limited to NVIDIA’s Dynamo framework?
No. Z.ai’s GLM-5.3-Flash separates encoding, prefill, and decoding into independently scalable worker pools on SGLang, reporting a 3x serving improvement on Chinese AI chips. vLLM Ascend added EPD documentation in release v0.17.0, and research like Kairos improves SLO attainment beyond a DistServe baseline. EPD is becoming a cross-platform production pattern.
What should the next inference build assume?
Inference platforms should assume that multimodal serving requires an explicit decision about where to split vision encoding from language work, tuned by media load, output length, model size, and precision. EPD with colocated or heterogeneous workers must be paired with smart scheduling, routing, and failure-handling layers to meet goodput and SLO targets rather than treated as a rigid topology.
