Key Takeaways
- Together AI’s rollout engine shifts live traffic between model deployments in staged steps guarded by health checks and metric-based regression gates.
- A live upgrade tripped the latency gate at 10% traffic when p95 hit 1,740 ms versus 734 ms on the source, a 137% regression against a 25% budget.
- Canary, blue-green, and rolling strategies share one step engine, backed by guarantees for capacity, readiness, share safety, and autoscaler respect.
Table of Contents
The Gate That Caught a 137% p95 Regression at 10% Traffic
Together AI has added canary rollouts to its dedicated model inference endpoints, letting teams move live traffic from a source model deployment to a target deployment in controlled stages.
The September 2026 release pairs health checks with metric-based regression gates, automatic system-pauses, and reverse-rollout recovery to prevent production downtime.
In a live platform run, an upgrade from Qwen2.5-7B-Instruct to Qwen3.5-9B tripped the latency gate after only 10% of traffic moved to the new model.
According to Together AI, the target reached a p95 router latency of 1,740 milliseconds against 734 milliseconds on the source, a 137% regression that exceeded the 25% budget.
The rollout froze at the 10% canary share, was canceled, and was then reversed to return 100% of traffic to the original model.
Across the entire run, the endpoint served 6,800 requests with zero non-200 responses.
Inside the Rollout Engine: Steps, Gates, and Recovery Paths
A rollout migrates traffic between two deployments on the same endpoint: a source that is serving today and a target that should serve tomorrow.
Operators choose one of three strategies, each using the same step engine and health gates but differing in traffic movement and capacity cost.
- Canary: traffic moves through staged percentages such as 10%, 50%, and 100%, with wait periods and optional metric checks between steps.
- Blue-green: one gated cutover from 0% to 100% once the target is healthy, useful for fast switches when double capacity is affordable.
- Rolling: replica-by-replica replacement that preserves total capacity, best for same-model engine or configuration changes.
Before each canary step, the target scales up and passes a health gate.
Only after routing caches converge does the source drain its matching share.
The wait period and metric gate run before the step is recorded as passed, so a regressed step is never marked complete.
Rollouts are created in a pending state and do nothing until an explicit start command.
The CLI ships as ‘tg’ in the Together Python package version 2.34.0 or newer.
Metric gates can use router error rate, router latency, or concurrent inflight requests, all measured identically for source and target.
A regression check compares the target against the source with a percentage budget, which self-calibrates for latency without requiring a known absolute p95.
A threshold check enforces an absolute SLO such as an error rate below 0.01.
Timing windows matter because the gate must wait for its full lookback period plus roughly 90 seconds of ingestion lag.
With a 300-second window, the platform automatically grows the wait period to 390 seconds.
When a gate trips, the rollout enters a system-paused state rather than aborting automatically.
The platform re-queries several times over about 90 seconds to filter transient blips, then waits for a human decision.
Operators can resume, promote, or cancel; the cancel action freezes the current traffic split and leaves both deployments serving.
There is no rollback verb; reversing traffic means creating a new rollout with source and target swapped.
The platform guarantees several properties across all rollout types.
- Capacity: target replicas round up and source drains round down, so a same-size swap never runs below starting capacity.
- Readiness: traffic never lands on capacity that has not passed health checks.
- Share safety: each side keeps the replicas its traffic share requires, or the rollout holds and pauses as under-served.
- Autoscaler respect: rollouts raise floors and lift target maximums without overriding operator policy.
- Gate integrity: gates read only the current step’s traffic and require enough samples.
Why Canary Traffic Splitting Is a Dividing Line in AI Infrastructure
Model deployment has long borrowed from software release engineering, but canary tooling for generative AI remains uneven across platforms.
Cloudera’s developer blog outlines canary workflows for predictive models on OIP v2 and KServe v2 serving engines, with traffic sliders for an 80/20 split and one-click rollback to a baseline version.
That same source states the limitation plainly: generative AI endpoints powered by single-engine runtimes such as NVIDIA NIM do not support multi-version traffic splitting and require 100% traffic allocation per container endpoint.
Together AI’s dedicated inference rollout engine takes the opposite path by making canary, blue-green, and rolling strategies native to generative model deployments.
DevRev’s guidance on canary deployments for AI agents adds a sharper warning: agents fail quietly compared with conventional software.
Declining accuracy, tone drift, or wrong skill selection may not show up in error rates alone, so live side-by-side comparison against a proven version becomes the key signal.
DevRev points to task success, groundedness, escalation rate, latency, and cost as the metrics that should control rollout pace.
Infrastructure guidance for LLM traffic also warns that a canary usually needs a full extra replica, not a fractional GPU share, because weights and KV cache do not shrink with a small traffic percentage.
Routing should stay centralized in one gateway, and sessions should be hashed on a stable key to avoid mid-thread model switches.
A couple of hundred requests will not reveal rare safety failures, so pre-registering minimum sample sizes and maximum soak times is critical.
Rollback should be a gateway weight change to zero, not a node drain or image rebuild.
The New Operational Bar for Production Model Upgrades
Canary rollouts are moving from a scripted workaround to a platform-native control plane for production AI.
The Together AI demo shows that a regression can be caught, paused, reversed, and survived without a single failed request.
For teams building AI deployment pipelines that need to scale, programmatic SEO AI automation is how Andres SEO Expert approaches it — contact Andres SEO Expert.
Frequently Asked Questions
What are canary rollouts for AI model inference?
Canary rollouts move a small, controlled share of live inference traffic from a source model deployment to a target deployment before full promotion. They use health checks, metric gates, wait periods, and recovery paths to catch regressions without downtime.
How did Together AI catch a 137% p95 latency regression at 10% traffic?
During a live upgrade from Qwen2.5-7B-Instruct to Qwen3.5-9B, the target hit a p95 router latency of 1,740 milliseconds versus 734 milliseconds on the source. That 137% regression exceeded the 25% latency budget, so the rollout froze at 10% canary traffic, was canceled, and was reversed to return 100% of traffic to the original model.
What metrics can trigger a canary rollout gate?
Metric gates can use router error rate, router latency, or concurrent inflight requests, measured identically for source and target. A regression check compares target against source with a percentage budget, while a threshold check enforces an absolute SLO such as an error rate below 0.01.
What happens when a canary metric gate trips?
The rollout enters a system-paused state instead of aborting automatically. The platform re-queries several times over about 90 seconds to filter transient blips, then waits for a human decision. Operators can resume, promote, or cancel; cancel freezes the current traffic split and leaves both deployments serving.
What is the difference between canary, blue-green, and rolling deployments?
Canary moves traffic through staged percentages such as 10%, 50%, and 100% with wait periods and optional metric checks. Blue-green performs one gated cutover from 0% to 100% once the target is healthy. Rolling replaces replicas one by one while preserving total capacity, which is best for same-model engine or configuration changes.
Why is canary traffic splitting harder for generative AI than predictive models?
Generative AI endpoints often run on single-engine runtimes such as NVIDIA NIM, which may not support multi-version traffic splitting and require 100% traffic allocation per container endpoint. AI agents can also fail quietly through accuracy decline, tone drift, or wrong skill selection, so live side-by-side comparison and metrics like task success, groundedness, escalation rate, latency, and cost become critical.
How much extra capacity does a canary deployment need?
A canary usually needs a full extra replica, not a fractional GPU share, because model weights and KV cache do not shrink with a small traffic percentage. Routing should stay centralized in one gateway, sessions should be hashed on a stable key, and teams should pre-register minimum sample sizes and maximum soak times to catch rare failures.
