Key Takeaways
- NVIDIA FLARE 2.9 separates long-running federation services from short-lived job workers, keeping GPUs busy and coordination persistent.
- A portable ‘resource_spec’ block lets one job declare GPUs, CPU units, and memory, which Docker, Kubernetes, and Slurm launchers translate into native runtime settings.
- Studies act as multi-tenant boundaries, mapping datasets, secrets, approved images, and scheduler policies locally so hospitals, universities, and cloud tenants share one federation.
Table of Contents
Federated Learning’s Real Bottleneck Just Moved From Models to Machines
Federated learning’s real bottleneck has shifted from model development to infrastructure coordination.
A technical brief published today on NVIDIA Developer Blog maps that shift directly.
NVIDIA FLARE 2.9 introduces a two-layer architecture that separates persistent federation services from dynamically launched job workers.
Each site in the same federation can now execute jobs through Docker, Kubernetes, or Slurm while retaining local control over datasets, secrets, images, and scheduling policies.
A hospital on a Docker host, a university on a Slurm cluster, and a cloud tenant on Kubernetes can participate in one study without first standardizing on a single platform.
The Two-Layer Execution Model That Decouples Coordination From Compute
As detailed on NVIDIA Developer Blog, Docker and Kubernetes deployment support first appeared in NVIDIA FLARE 2.8, while Slurm joins in version 2.9.
NVIDIA FLARE deployments now operate as two distinct planes.
Long-running server and client parent processes maintain the federation, authenticate connections, and coordinate submitted work.
Separate job workers then perform the actual training or analysis and exit when their tasks complete.
This separation keeps federation services available without holding GPUs idle between jobs.
When a data scientist submits a job, each parent launches a worker through the site’s configured execution platform.
The job itself describes resource intent through a portable ‘resource_spec’ block in its ‘meta.json’.
Portable keys include ‘num_of_gpus’, ‘num_of_cpus’, and ‘memory’.
An ‘@default’ profile applies to every targeted site, while named client or server entries override only the values that differ.
Each launcher resolves those portable values into native runtime settings.
- Docker converts CPU units to ‘nano_cpus’ and memory to byte-level limits.
- Kubernetes sets matching CPU and memory requests and limits, then adds the GPU request.
- Slurm emits ‘–cpus-per-task’, ‘–mem’, and GPU ‘–gres’ flags.
Docker Workstations Without Idle GPUs
Docker targets single-host environments such as workstations, lab servers, and edge systems.
FLARE runs a persistent parent container from a parent image, then launches a separate job container for each submission.
The parent image holds the FLARE runtime and launch components, while the job image can contain the training framework, model code, and application dependencies.
Sites can expose GPUs through NVIDIA Container Toolkit and apply Docker settings for shared memory, mounts, and networking.
Infrastructure owners can keep the parent environment stable while researchers update their training image independently.
Kubernetes Pods and Admission Control
On Kubernetes, a Helm chart installs each persistent FLARE server or client as a parent pod.
Every submitted job creates a separate job pod, and the Kubernetes scheduler places it according to CPU, memory, GPU, storage, and placement requirements.
FLARE connects federated jobs to namespaces, service accounts, Secrets, persistent volumes, node selectors, tolerations, and admission policies.
A study can use a pod template to target one node pool while another study uses a different pool.
Cluster operators retain RBAC, registry credentials, network policy, and GPU enablement.
Slurm Allocations for HPC Clusters
For shared GPU clusters, the FLARE Slurm launcher submits each server or client job worker as a batch job.
Slurm selects the compute nodes and enforces GPU, CPU, memory, partition, account, QoS, and time limits.
The launcher supports bare execution, Pyxis and Enroot containers, and Apptainer containers.
Slurm remains the resource authority, preserving the operational model cluster administrators already use.
Study Boundaries Without Mixing Tenants
Sharing infrastructure raises a second question: how do multiple teams run studies without mixing users, jobs, or data?
FLARE studies provide a logical multi-tenant boundary within one deployment.
Each study defines participating client sites and the admin users allowed to open a session.
Site operators use a local ‘study_runtime.yaml’ file to map each study to local dataset mounts, environment variables, secret-backed references, approved images, and scheduling policies.
- Dataset mounts tied to the study and site.
- Secret-backed environment variables resolved from the site’s secret store.
- A site-approved default job image for that study.
- Scheduler policies such as partitions, accounts, and QoS for Slurm.
In Kubernetes, a pathology study can map to a site-owned data volume and database credentials without storing raw secret values.
The launcher resolves the study entry, mounts the data volume read-only, and injects ‘secretKeyRef’ references from the site’s secret store.
Why Slurm, Kubernetes, and Docker Are Becoming One Operational Plane
Industry scheduler comparisons have long drawn a hard line between Slurm and Kubernetes.
Slurm natively provides gang scheduling, topology-aware physical GPU placement, and MPI integration.
Kubernetes schedules pods individually and generally treats GPUs as generic resources, requiring plugins such as NVIDIA GPU Feature Discovery and Topology Manager to approximate that behavior.
Those differences are now becoming an operational plane rather than a barrier.
Google Cloud’s official documentation now includes a supported Helm-based Slurm-on-GKE deployment path, signaling that major cloud providers see HPC schedulers and Kubernetes as adjacent layers.
A September 2026 arXiv preprint, not yet peer-reviewed, found that Slurm’s fixed allocation before execution differs sharply from Kubernetes’ more dynamic pod scheduling.
The preprint evaluated resource control across four production supercomputers and reported reducing median additional completion time from 42 seconds to 4 seconds through live queue-aware placement, a 10.5x improvement.
NVIDIA FLARE 2.9 sits precisely at this boundary.
Its portable ‘resource_spec’ lets a job declare GPUs, CPU units, and memory without coupling to a scheduler’s native syntax.
Each launcher then translates those requirements into Docker container limits, Kubernetes pod requests, or Slurm batch allocation directives.
For AI platform teams, the strategic shift is clear: orchestration portability has moved from a DevOps convenience to a data-governance requirement.
A federation can now span an on-premises HPC cluster, a hospital’s Docker host, and a cloud Kubernetes namespace without forcing all participants onto identical infrastructure.
Heterogeneous Infrastructure Becomes the Default, Not the Exception
NVIDIA FLARE 2.9 turns heterogeneous compute from a federation blocker into a configuration detail. For teams tracking these infrastructure shifts and scaling AI content operations in parallel, programmatic SEO AI automation is how Andres SEO Expert approaches it — contact Andres SEO Expert.
Frequently Asked Questions
What is NVIDIA FLARE 2.9?
NVIDIA FLARE 2.9 is a federated learning framework update that adds Slurm support and uses a two-layer architecture separating persistent federation services from dynamically launched job workers. It lets each site run jobs through Docker, Kubernetes, or Slurm while keeping local control over datasets, secrets, images, and scheduling.
How does FLARE decouple coordination from compute?
Long-running server and client parent processes maintain the federation, authenticate connections, and coordinate submitted work. Separate job workers then perform the actual training or analysis and exit when their tasks complete. This keeps federation services available without holding GPUs idle between jobs.
Can one federated learning study span Docker, Kubernetes, and Slurm?
Yes. FLARE 2.9 allows a hospital on Docker, a university on Slurm, and a cloud tenant on Kubernetes to participate in one study without first standardizing on a single platform. Each site’s launcher translates portable resource requirements into native runtime settings.
What is resource_spec in NVIDIA FLARE?
resource_spec is a portable block in a job’s meta.json that describes resource intent using keys like num_of_gpus, num_of_cpus, and memory. An @default profile applies to every targeted site, while named client or server entries override only the values that differ. Launchers convert these into Docker, Kubernetes, or Slurm settings.
How does FLARE support multi-tenant studies without mixing data?
FLARE studies provide a logical multi-tenant boundary within one deployment. Each study defines participating client sites and the admin users allowed to open a session. Site operators use a local study_runtime.yaml file to map studies to dataset mounts, secret-backed environment variables, approved images, and scheduling policies. Kubernetes can resolve secretKeyRef references without storing raw secret values.
Why is heterogeneous infrastructure becoming the default for federated learning?
Because orchestration portability has moved from a DevOps convenience to a data-governance requirement. A federation can span an on-premises HPC cluster, a hospital’s Docker host, and a cloud Kubernetes namespace without forcing all participants onto identical infrastructure. FLARE 2.9 turns heterogeneous compute from a federation blocker into a configuration detail.
What bottleneck moved from models to machines in federated learning?
The bottleneck has shifted from model development to infrastructure coordination. FLARE 2.9 addresses this by separating persistent federation services from dynamically launched job workers, so sites can retain local control over data, secrets, images, and scheduling while still participating in one study.
