Topograph Gives Kubernetes and Slurm a Live Map of GPU Topology

The Hidden Cost of Blind Scheduling

On September 22, 2026, NVIDIA’s developer blog published a technical walkthrough that puts a long-standing AI infrastructure problem squarely in the open. The post details Topograph, an open source toolkit that discovers how cluster hardware is physically connected and converts that map into inputs Kubernetes and Slurm schedulers can consume.

For AI factories, this is not a niche concern. Poor GPU placement fragments topology domains and forces traffic across shared links, which raises job costs while GPUs continue consuming power without making progress.

AI factories are power-limited systems that deliver maximum value when fully optimized.

Topograph’s answer is to make the physical network a first-class scheduling input rather than a manually maintained afterthought.

Inside Topograph’s Provider-to-Scheduler Pipeline

As detailed in NVIDIA’s developer blog, Topograph operates on two concepts. Providers handle discovery from cloud APIs or on-premises fabric systems and normalize that data into a canonical model. Engines output that model in formats each workload manager expects.

Cloud providers with working integrations include Google Cloud, Lambda, Nebius, Nscale, Oracle Cloud Infrastructure, and Crusoe. On-premises deployments can use an InfiniBand provider with ibnetdiscover or NetQ for Spectrum-X and Multi-Node NVLink domains. The provider interface is open, so operators can add support for their own environments.

From Fabric Discovery to Canonical Topology

Five components keep the topology view current. The API server validates and dispatches discovery requests. A node observer watches Kubernetes node or pod changes. A node data broker collects per-node attributes. The provider converts cloud or fabric data. The engine writes scheduler-ready output.

• API server: validates requests, aggregates duplicates, and dispatches discovery jobs.
• Node observer: watches Kubernetes node or pod changes and requests regeneration with retries.
• Node data broker: stores per-node attributes as annotations.
• Provider: converts cloud or fabric data into the canonical representation.
• Engine: writes that representation in scheduler-specific formats.

Topograph regenerates its view on request and when watched cluster changes occur. The API exposes endpoints for generation, result retrieval, health, and Prometheus metrics. Repeated identical requests reset a trailing timer and are processed once.

For testing without production hardware, simulation models describe node and switch hierarchies. The kwok-nodes utility and Kind/KWOK helpers turn those models into virtual Kubernetes nodes.

Kubernetes Node Labels and Gang Scheduling

For Kubernetes, the default scheduler does not discover physical interconnect hierarchy. Topograph publishes node labels such as fabric.topograph.run/tier-0 and fabric.topograph.run/tier-1, where tier-0 is the leaf switch closest to the compute node.

These labels can be used as topologyKey values in preferred pod affinity. KAI Scheduler, a CNCF Sandbox project donated by NVIDIA, can use the same node labels for topology-aware gang scheduling. Kubernetes 1.36 also introduced alpha topology-aware workload scheduling through KEP-5732.

Topograph also supports Node Feature Discovery through an NFD engine. That output is intended for consumers that already rely on NodeFeatureGroup objects, not as a replacement for native pod affinity labels.

Slurm, Slinky, and On-Premises Modes

On Slurm clusters, Topograph installs through Debian or RPM packages. It can generate cluster-wide tree format, block format, or per-partition YAML configurations. The optional reconfigure parameter runs scontrol reconfigure after a file is written.

Slinky, the Slurm-on-Kubernetes platform developed by SchedMD and now part of NVIDIA after its December 2025 acquisition, uses a dedicated engine. Topograph maps Kubernetes nodes to slurmd pods and writes Slurm topology data to a ConfigMap.

For Multi-Node NVLink systems, a narrower DRA provider reads existing nvidia.com/gpu.clique labels when regenerating topology configuration. Dynamic node modes annotate selected Kubernetes nodes with the current Slurm topology specification.

The Topology Arms Race Reaches Production Kubernetes

Topograph is arriving at a moment when the Kubernetes scheduling layer is rapidly evolving. DRA reached general availability in Kubernetes 1.34. Native gang scheduling landed in 1.35 and was iterated again in 1.36. Device taints and tolerations are set to reach GA in 1.37.

Those features solve part of the problem, but topology visibility has remained a blind spot. The default Kubernetes scheduler treats nvidia.com/gpu as an indivisible integer count, which hides NUMA, PCIe switch, and RDMA locality. Independent analysis has documented an H100 cluster where distributed training ran 30 to 40 percent slower because the scheduler placed the GPU and its RDMA NIC on different NUMA nodes.

This is not just a hardware problem. One author-reported case study describes a research team running 100-GPU training jobs that cut GPU waste from roughly 23 percent to under 4 percent after enabling gang scheduling with Kueue. The figures come without primary data citations, but they illustrate the operational stakes.

NVIDIA’s enterprise orchestration layer is moving in the same direction. The 2.3.1 release notes for NVIDIA Mission Control confirm that Run:ai has added topology-aware scheduling, DRA support, and Kubernetes Gateway API support. Topology placement visibility now reports the actual name, level, value, and constraint status of each workload.

That visibility closes the loop. Topograph supplies the raw topology map for open source schedulers. Mission Control and Run:ai then expose that placement detail to operations teams at the platform level.

When Placement Stops Being a Blind Spot

Topograph’s real contribution is not a new scheduling algorithm but a current, provider-reported topology layer that turns physical locality into a first-class input for Kubernetes and Slurm. For AI factory operators running power-constrained GPU estates, that removes one of the last silent performance taxes. For teams building technical authority around AI infrastructure as quickly as the scheduler landscape is moving, programmatic SEO AI automation is how Andres SEO Expert approaches it — reach out here.