Is this page helpful?

The NVIDIA AI Enterprise Infrastructure Software Suite#

The NVIDIA AI Enterprise infrastructure software suite addresses these pain points through a layered suite of products that optimize every stage of the AI lifecycle.

The entire suite is built on Kubernetes, utilizing the Operator framework and Custom Resource Definitions (CRDs) for fine-grained resource management. Additional components for GPU orchestration, optimized hardware virtualization, infrastructure services offloading, and cluster management further enhanced traditional enterprise management frameworks to provide the foundation for the most demanding AI workloads.

_images/software-suite-01.png

GPU-aware microservices#

NVIDIA GPU Operator automates the deployment and life-cycle management of all the software components required to provision and utilize GPUs within a Kubernetes cluster.

When you add a GPU-equipped node to a cluster, Kubernetes is natively unaware of the specialized hardware. The GPU Operator bridges this gap by deploying a sequential, containerized stack of components to make the GPU usable.

  • Node Feature Discovery (NFD): It automatically detects attached GPUs on cluster nodes and labels them accordingly, ensuring software components are only deployed where needed.

  • NVIDIA Driver: It deploys the necessary kernel modules and CUDA libraries directly to the host operating system via a DaemonSet, establishing communication between the hardware and the OS.

  • NVIDIA Container Toolkit: It configures the container runtime (such as containerd or CRI-O) to detect when a container requests a GPU and automatically injects the required CUDA libraries and driver bindings into that container.

  • NVIDIA Device Plugin: It exposes the GPUs to the kubelet API, officially registering them as schedulable, first-class Kubernetes resources (e.g., nvidia.com/gpu).

  • NVIDIA DRA Driver for GPUs: It provides a modern, DRA-based option for GPU allocation and provision ComputeDomains for secure Multi-Node NVLink workloads.

  • MIG Manager: It manages MIG configuration on nodes in your Kubernetes cluster. Multi-Instance GPU (MIG) enables GPUs to be partitioned into separate and secure GPU instances for CUDA applications.

  • Data Center GPU Manager (DCGM) Exporter: It collects rich telemetry, such as GPU usage, temperature, and power consumption, and exposes it to monitoring tools like Prometheus and Grafana.

Network Fabric Configuration#

Just as the GPU Operator abstracts the complexity of compute hardware, the NVIDIA Network Operator abstracts the complexity of the cluster’s network fabric. In a standard Kubernetes environment, workloads default to using the cluster’s overlay network, which routes traffic through standard TCP sockets and the host’s kernel networking stack. While this is fine for standard microservices, it is severely inadequate for the massive data transfers required by AI workloads.

To bridge this gap, the Network Operator uses Kubernetes Custom Resource Definitions (CRDs) and the Operator SDK to orchestrate the following components:

  • Node Feature Discovery (NFD): Automatically scans nodes to detect specialized networking hardware (like Mellanox ConnectX NICs or BlueField DPUs) and labels the nodes accordingly.

  • Advanced Drivers: Automatically deploys highly optimized networking drivers, such as DOCA-OFED, directly to the host.

  • Secondary Networks & CNIs: Deploys secondary Container Network Interfaces (CNIs) like Multus, alongside IP Address Management (IPAM) plugins, allowing pods to attach directly to high-speed physical networks outside the standard Kubernetes overlay.

  • Device Plugins: Installs the necessary Kubernetes device plugins so that advanced network interfaces can be requested and scheduled as manageable resources, much like CPU or memory.

The Network Operator is a critical enabler for the “AI Factory” because it removes the most severe bottleneck in distributed AI architecture: the network.

Seamless Application and Model Deployment#

NVIDIA NIM Operator is a Kubernetes operator designed to automate the deployment, scaling, and lifecycle management of optimized AI models in the form of NIM microservices and their supporting ecosystems.

While the GPU Operator and Network Operator are responsible for configuring the underlying hardware and fabric, the NIM Operator sits at the application layer. It simplifies the highly complex process of turning a raw AI model into a production-ready, highly available API endpoint.

To accomplish this, the NIM Operator extends the Kubernetes API with several specialized Custom Resource Definitions (CRDs):

  • NIMCache: Manages downloading and caching NIM model artifacts on the cluster. The NIMCache resource supports models from NVIDIA NGC, Hugging Face-protocol sources such as Hugging Face Hub, and NGC-mirrored local model registries using S3, HTTPS, or NGC protocols (such as JFrog). It provisions or reuses Persistent Volume Claims (PVCs), stores model weights locally on cluster storage, and supports local or CSI-compatible storage classes.

  • NIMService: Manages the actual deployment and lifecycle management of the inference (NIM) microservice either directly on Kubernetes or on KServe. It automatically orchestrates pod creation, sets up health and readiness probes, handles NGC authentication secrets, requests the appropriate GPU resources via DRA (dynamic resource allocation), sets up ingress routes and configures autoscaling using HPA (Horizontal Pod Autoscaler)

  • NIMPipeline: Manages groups of interconnected NIM services that need to be deployed together as a unified application.

Intelligent Workload and GPU Orchestration#

NVIDIA Run:ai is a Kubernetes-native platform for enterprise AI workload and GPU orchestration. It enables organizations to operate shared GPU infrastructure as a governed service, giving platform teams centralized control while providing users with predictable access to compute.

  • Policy-Based Allocation: Administrators can establish guaranteed GPU allocations for departments, teams, and projects based on organizational priorities. When capacity is available, teams can exceed those guarantees, improving utilization without compromising predictable access.

  • Queueing, Priorities, and Preemption: When demand exceeds supply, Run:ai queues and schedules workloads according to centrally defined policies. Higher-priority workloads can reclaim resources from eligible lower-priority workloads, subject to configured preemption policies.

  • GPU Sharing: Supported workloads can request fractional GPU resources rather than reserving an entire GPU. This allows organizations to accommodate more users and workloads on existing infrastructure when workload and isolation requirements permit.

  • Topology-Aware Scheduling: Run:ai places workloads according to their resource and topology requirements while consolidating smaller workloads where appropriate. This reduces fragmentation and preserves capacity for distributed training and other large-scale jobs.

  • Governance and Visibility: Platform teams gain visibility into resource allocation and consumption across clusters, departments, projects, and workloads, supporting policy enforcement and capacity planning.

Together, these capabilities improve infrastructure utilization while delivering governed, equitable, and policy-driven access to AI infrastructure.

Infrastructure Offloading and Security#

To prevent security and networking tasks from consuming valuable host CPU cycles, NVIDIA DOCA acts as the software platform that unlocks the capabilities of BlueField Data Processing Units (DPUs) and ConnectX SuperNICs. DOCA offloads, accelerates, and isolates data center workloads, enabling inline cybersecurity, software-defined networking, and storage virtualization. By running these services directly in the silicon, DOCA protects AI agents and data at infrastructure speed while freeing up host compute for actual AI workloads.

DOCA Platform Framework (DPF) Operator automates the provisioning, configuration, and orchestration of NVIDIA BlueField Data Processing Units (DPUs) within a cloud-native Kubernetes environment.

While the GPU Operator manages accelerators and the Network Operator configures the high-speed fabric, the DPF Operator is designed to manage the DPUs that sit between the host server and the network. It handles the following key tasks:

  • DPU Provisioning & Lifecycle Management: It automates the complex process of installing OS images and firmware (known as BlueField Bundles or BFBs) onto the DPUs, configuring them, and managing maintenance tasks like controlled node draining and reboots.

  • Service Orchestration: Instead of running infrastructure services on the host server, the DPF Operator deploys containerized applications directly onto the DPU’s onboard ARM cores. It orchestrates a variety of “DOCA Services,” such as Host-Based Networking (HBN), OVN-Kubernetes for software-defined networking, DOCA SNAP for storage virtualization, and DOCA Argus for security.

  • Service Function Chaining (SFC): It allows administrators to connect these DPU services together (e.g., routing traffic from a physical port, through a firewall, and into a networking service) directly on the hardware using declarative Kubernetes manifests.

Hardware Virtualization and Isolation#

NVIDIA vGPU for Compute eliminates the rigid allocation of hardware by allowing multiple virtual machines (VMs) to share a single physical GPU. It provides temporal and spatial isolation through Time-Sliced and Multi-Instance GPU (MIG) backed modes. This ensures that multi-tenant workloads can run securely and in parallel without contention, allowing organizations to support diverse workloads—from lightweight inference to batch jobs—on shared infrastructure.

Even when running virtualized AI workloads in direct hardware passthrough mode, the NVIDIA AI Enterprise infrastructure software suite delivers substantial orchestration, automation, and performance advantages. The NVIDIA GPU Operator can be installed in virtualized node clusters to automatically configure drivers and container runtimes. Inside guest virtual machines where PCI passthrough is configured for SR-IOV devices, the NVIDIA Network Operator automatically provisions high-performance drivers and secondary network interfaces. This natively delivers CPU-bypassing Remote Direct Memory Access (RDMA and GPUDirect RDMA) to guarantee low-latency, high-throughput communication for guest workloads. Finally, for environments employing BlueField DPUs, the DOCA Platform Framework (DPF) supports a dedicated DPU Passthrough mode. This allows administrators to use a simple declarative manifest to configure the DPU as a transparent network pipe for host traffic, while still automating operating system installations, rolling updates, and telemetry collection across the fleet.

Cluster Management#

NVIDIA Base Command Manager is a tool for provisioning, monitoring, and auto-scaling heterogeneous HPC and AI clusters. It provides comprehensive end-to-end management from edge to data center to hybrid-cloud deployments based on upstream Kubernetes.