Inference Provider Platform Requirements on GB300 NVL72 for NCPs
Purpose
This document specifies the requirements for an NVIDIA Cloud Partner (NCP) to build a GB300 NVL72 compute cluster that minimizes friction for Inference Service Providers (ISPs) deploying and running their inference platforms. The result is managed Kubernetes that the NCP delivers to the ISP, which the ISP plans to deploy its stack on.
The document assumes that cluster hardware and facility are built, and that the physical build follows the NVIDIA DSX reference designs. It defines what the NCP does from that point forward: provision bare metal, stand up networking and multi-tenancy, storage, and the managed Kubernetes platform, and expose the telemetry, break-fix, and capacity that operations depend on.
The NVIDIA Requirements for AI Clouds document is the general requirements bar for any NCP delivering GPU capacity. Broader multi-tenant build guidance is in the NCP Software Reference Guide.
Each section has two parts: prose that provides context and a requirements table that outlines the obligations. The tables are the normative part of this document. The prose explains why a requirement exists and how the pieces fit together; it does not itself impose requirements, so it does not repeat the requirement levels. Requirement levels follow IETF RFC 2119:
- MUST and MUST NOT are mandatory.
- SHOULD and SHOULD NOT are strong recommendations that a partner deviates from only with good reason.
- MAY is optional.
- Rows marked INFO are informational and place no obligation on the NCP.
Each table row also shows who owns the work, under the NCP-to-ISP responsibility split:
- Owns: the accountable party.
- Provides: provides the substrate capability.
- Exposes: exposes an interface or topology for the other party to use.
- Consumes: uses what the other party provides.
- Executes: acts on request.
- n/a: not applicable to that party.
Throughout, the NCP is the Operator and owns the substrate; the ISP is the Tenant and owns its inference software stack on the managed cluster.
Reference Designs
The NCP adheres to the following reference designs as a starting point: the DSX Facilities Infrastructure Design Guide for facility, power, and cooling; the GB300 NVL72 Reference Design for cluster architecture, server specification, and network cabling; and the storage partner requirements. These reference designs also specify the network-appliance requirements for edge connectivity.
Access to the linked NVOnline documents must be granted by an NVIDIA representative and may not be available to ISPs.
Requirements Summary
GPU Compute Hardware, Cluster and Capacity
The GPU compute servers must be in the Qualified Systems Catalog. On that hardware, the operating system must support Kubernetes as the compute delivery mechanism, and a supported Linux distribution is recommended; refer to the NVIDIA AI Enterprise support matrix for distributions that include the driver components. This revision targets bare-metal environments, so KVM-based Kubernetes delivery using virtual machines is out of scope.
Requirements Summary
A cluster is defined as a collection of compute nodes, control nodes, networking, and storage mapped to a specific build captured in a single hardware bill of materials. In this document, we assume one managed cluster per region per tenant (or per requested tenant cluster).
A cluster is a collection of compute nodes, control nodes, networking, and storage mapped to a specific build captured in a single hardware bill of materials. This document assumes one managed cluster per region per tenant, or per requested tenant cluster.
Hardware Validation
Hardware validation confirms that each node, and the cluster as a whole, operate correctly before the cluster performs production work. It applies the acceptance-testing bar from the NVIDIA Requirements for AI Clouds to GB300 NVL72, using the NVIDIA AI Cloud-Ready Validation Suite for functional validation and NVIDIA Exemplar Cloud for performance baselines.
NCP-Managed Inventory Services
The NCP maintains a single source of truth for the fleet: an authoritative, real-time record of every GPU, node, region, and NCP, and the current state of each. It tracks identity and location (NCP, region, rack, NVL72 chassis, node, GPU SKU), topology (NVLink Domain membership, NVSwitch fabric ID), software versions (driver, CUDA, NCCL, and fabric-manager state), health (DCGM), and allocation (MIG geometry, assigned pool, current workload). An NCP builds this on its own asset system or from DSX OS components that already emit these fields: NVIDIA Infra Controller for discovery, provisioning, and node state; Fleet Intelligence for health and visibility; DCGM for GPU health; and the fabric manager with topology labeling for the NVLink and NVSwitch fields.
Requirements Summary
Compute Operating System
The NCP provides an infrastructure-as-a-service capability to automate Linux OS installation on compute nodes. The bare-metal lifecycle is documented in the NICo open source project, and the Linux distribution support matrix is covered in the NVIDIA AI Enterprise documentation.
Requirements Summary
Networking and Multi-tenancy
The cluster network is built on the GB300 NVL72 Reference Design, published to NCPs through the NVIDIA NVOnline partner portal, which also specifies the network-appliance requirements for edge connectivity. On top of that hardware, two planes carry the work: a north-south plane on the BlueField-3 DPU that enforces zero trust and hosts multi-tenancy and carries the control plane, storage, and external connectivity, and an east-west plane on the back-end RDMA fabric, InfiniBand or Spectrum-X, that moves tensor-parallel collectives over NCCL, KV-cache over NIXL, and model weights between nodes. The NCP handles IP addressing and inter-host routing, with BGP as the routing protocol. How Kubernetes services meet the edge, and how edge connectivity is sized to inbound request volume, are covered in Kubernetes Services for Edge and the Inference Routing Data Plane.
Multi-tenancy is enforced at the network level, not at the application level. Host networking on the BlueField DPUs and the network appliances, the Spectrum Ethernet switches and Quantum InfiniBand switches, keep tenants isolated end-to-end, and the NVIDIA Infrastructure Controller pairs with the network to enforce that isolation. The design uses a BGP underlay with EVPN, VXLAN overlay encapsulation, access control lists on the DPU, and zero trust between the DPU and the compute host. Kubernetes interaction with multi-tenancy is covered in the Tenancy Model section.
Requirements Summary
Kubernetes interaction with multitenancy is addressed in the Kubernetes Tenancy section.
Storage Architecture
Storage for a GB300 NVL72 cluster follows the NCP Reference Design for Storage, and applies the AI Cloud Requirements storage bar to GB300 NVL72. The goal is high-throughput, low-latency storage that GPU compute nodes can read from and write to directly, delivered in a way that the ISP does not manage the underlying layout.
Requirements Summary
Kubernetes Platform
The NCP provides managed Kubernetes. The ISP does not run its own control plane, etcd, or upgrade pipeline; those are the NCP’s responsibility under the SLA. There is one managed Kubernetes cluster per region per ISP tenant, and this document targets Kubernetes on bare metal. This section applies the AI Cloud Requirements KaaS section to GB300 NVL72.
Requirements Summary
Everything above the managed Kubernetes boundary is owned by the ISP and is out of scope for this document, which specifies only the NCP substrate. The ISP owns its workload-tier CRDs, controllers, operators, admission policies, and schedulers; any replacement or extension of the defaults the managed service ships (for example, swapping the CNI to Cilium where allowed, or adding a custom scheduler extender alongside the default); the model-serving, autoscaling, routing, and observability stack; and GitOps reconciliation against the cluster (for example, Argo CD). This document references these areas only to define the substrate capability the NCP must expose.
Installing Kubernetes
Operational management of Kubernetes is the NCP’s responsibility: selecting the distribution and versions, installing Kubernetes, and verifying the installation.
Requirements Summary
Preparing Kubernetes for Inference
NCPs and tenants can validate the components using the AI Cluster Runtime.
AI Cluster Runtime will create an inference-focused recipe on Ubuntu targeting GB300NVL72:
The recipe can be installed using Helm charts by creating a Helm bundle:
This bundle includes all the necessary components for Kubernetes to run a performant Inference stack.
Requirements Summary
GPU Driver Management
The GPU Operator keeps the GPU software stack consistent across the fleet by managing drivers in the container runtime through the embedded Container Toolkit, and collecting health and telemetry from DCGM. The requirements describe how it behaves on an NCP cluster: it supports GPUDirect RDMA, it allows driver upgrades without competing with other services, such as gpud, for system locks, and its driver management can be disabled when the NCP provisions drivers another way.
For GPUDirect RDMA, the NCP delivers working, validated GPUDirect RDMA; whether it uses dma-buf or nvidia_peermem is an implementation choice set by the kernel and RDMA stack.
Requirements Summary
Kubernetes Networking
The NCP selects the container network interface (CNI) at install time. On a GB300 NVL72 cluster, that choice has to satisfy three demands at once: pods that scale dynamically, tenant isolation that works with the BlueField DPU and the north-south Ethernet, and access to the back-end RDMA network for GPU-to-GPU communication. The specific product is an NCP decision. The NCP can also use Network Operator to manage the GPU-to-GPU communication.
Requirements Summary
Tenancy Model
The tenancy model establishes the boundary between the NCP, as Operator, and the ISP, as Tenant. The baseline is one NCP-managed Kubernetes cluster per ISP tenant per region, each with its own control plane, aligned to the KaaS model in the NVIDIA Software Reference Guide. Tenants do not share a control plane for production inference, and dedicated worker hosts back each tenant, because isolation is enforced at the networking layer. That design is what supports reserved capacity, confidential computing, strict data residency, and enterprise SLOs. The managed service also exposes enough topology, quota, health, and lifecycle information for the ISP to run scheduling, autoscaling, admission control, and break-fix without owning the control plane.
Requirements Summary
Kubernetes Services for Edge
Inbound inference requests cross from the edge-routed network into the Kubernetes network and land on a service running on a cluster node. To keep routing efficient, these services terminate as close to the public edge as possible. The NCP uses the Kubernetes load balancer as the standard, implemented through a custom controller that calls the load balancer’s API (for example, F5, NGINX, or a custom hardware API) to create the virtual server and target group on service creation. Edge-tier services also need anycast IPs for multi-zone deployments, TLS 1.3 termination with HTTP/2 and HTTP/3, WAF, DDoS, and bot mitigation at L3 and L4, and authentication through a bearer key, JWT, or optional mTLS.
Requirements Summary
Image Registry
Serving images are large, and pulling them across regions adds latency to every cold start and scale-out. To avoid that, the NCP provides a native in-region image registry that the ISP uses to host its images close to the compute.
Requirements Summary
Exposing Kubernetes Control Plane - Kubeconfig Acquisition
Once Kubernetes is installed, the ISP needs a way to reach the cluster. The NCP provides access to the Kubernetes API and kubectl, gated by an IAM policy that requires authentication and authorization, so that a tenant can access only its own cluster. To make that access easier to consume, the NCP also exposes interaction with the deployed environment through an AI Server. Today, each NCP delivers kubeconfig in its own way, which is a documented source of friction for ISPs, so a standardized IAM policy and kubeconfig-delivery pattern across NCPs are the goal. Identity aligned with the Software Reference Guide user personas is provided here as well.
Requirements Summary
Topology-Aware Placement and NUMA Alignment
Placement and scheduling policy belongs to the ISP; the NCP’s job is to expose the cluster topology and enable the node-level controls the ISP’s scheduler needs. Neither the GPU Operator nor the Network Operator makes placement or NUMA decisions; they provision node-level GPU and networking capabilities and publish the labels and resources a scheduler reads. The NCP exposes topology as Kubernetes node labels, with Topograph producing the fabric labels (NVLink Domain, NVSwitch fabric, rack, and IB partition), and Node Feature Discovery and NVIDIA GPU Feature Discovery supplying the complementary hardware labels. It enables NUMA alignment through the kubelet Topology Manager, CPU Manager, and Memory Manager, plus device NUMA affinity from the NVIDIA device plugin, and supports Dynamic Resource Allocation and CDI so GPUs and NICs are co-allocated with topology awareness.
Given that substrate, the ISP scheduler (for example, KAI Scheduler, Volcano, or Kueue) enforces the placement policy, and the kubelet Topology Manager enforces NUMA pinning on the nodes the scheduler selects.
Requirements Summary
Additional Kubernetes Components
This layer runs on top of the managed cluster, and the platform, meaning the ISP, owns most of it. It spans platform packaging and networking (Helm, Cilium, Multus), scheduling (KAI Scheduler, Volcano, or Kueue with a custom extender), node lifecycle and OS imaging (Cluster API over NVIDIA Infra Controller, Tinkerbell, or NCP-native bare-metal APIs), GitOps and progressive delivery (Argo), in-region image distribution (Harbor with an NGC pull-through cache, plus Spegel or Dragonfly for peer-to-peer distribution), the driver, CUDA, NCCL, and fabric-manager lifecycle (pinned per node pool through the GPU Operator), health remediation, and model-weight distribution (an object store, a GPUDirect-Storage warm cache, a per-node NVMe cache, and peer-to-peer weight pull over the east-west RDMA fabric).
Because the platform owns most of these, the table below carries only the NCP responsibilities in this layer: enable Dynamic Resource Allocation, own firmware, BIOS, and BMC, provide the node base images, provision and join nodes, expose the managed-Kubernetes and BMaaS APIs that the platform’s health remediation calls, and provide the GPUDirect-Storage filesystem and east-west RDMA fabric that model-weight distribution depends on. The remaining rows are marked as INFO.
Requirements Summary
Inference Routing Data Plane
The inference routing requirements in this section are covered in greater detail in the edge connectivity reference architecture document.
Comprehensive recommendations for inference routing are covered in the Inference Reference Architecture document.
Requirements Summary
Autoscaling and Scheduling
Autoscaling is where the NCP and the ISP intersect: the ISP’s platform drives Kubernetes autoscaling, and the NCP supplies and reclaims the machines underneath. It operates at two scales: replica scaling within a deployment, which the ISP owns, and fleet scaling of the underlying pools, which depends on the NCP. On contraction, scale-down cordons a node, drains it, and removes it from the cluster. The fleet autoscaler tracks free GPU-seconds per pool, SKU, and region; bursts on demand from the NCP when a reserved pool falls below its headroom target; drains the spot pool first; and communicates with the Cluster API to the NCP’s machine controllers. The fleet-facing items are the NCP’s; the replica-scaling behavior is ISP-owned.
Requirements Summary
Telemetry and Observability
The NCP exports the telemetry the ISP needs to operate the cluster and meet its SLAs, applying the AI Cloud Requirements telemetry bar and the Software Reference Guide, Telemetry and Observability to GB300 NVL72. NVIDIA provides the building blocks through DSX OS components: DCGM for GPU health and metrics, Fleet Intelligence for fleet-wide health and visibility, and NVSentinel for fault detection and remediation.
Requirements Summary
Identity Management
The NCP should provide identity aligned with the Software Reference Guide user personas. Obtaining kubectl and the associated identity policies is covered in the Exposing Kubernetes Control Plane - Kubeconfig Acquisition section.
Requirements Summary
Operations
Operational requirements apply the AI Cloud Requirements Service Delivery SLAs to GB300 NVL72. The NCP staffs the cluster with a dedicated technical specialist reachable by NVIDIA, runs a monitored Slack channel or equivalent, and provides 24x7 support under its standard incident-severity procedures. It communicates service-impacting incidents and both planned and unplanned maintenance to NVIDIA, and it allows NVIDIA to schedule planned maintenance windows using APIs or console tools. The NCP also remediates critical vulnerabilities promptly and discloses them transparently, and it runs an incident toolchain built on PagerDuty, Statuspage, a war-room template, and a customer-communications playbook.
NVIDIA-stack-specific operational failure modes the platform must handle automatically:
- DCGM continuously monitors XIDs (driver-level GPU fault codes), and the appropriate response depends on the fault reported. When DCGM detects an XID, the general remediation sequence is to cordon the node, drain its workloads, and raise an RMA ticket. However, the urgency varies by fault type: XID 79 (GPU falling off the bus) and XID 48 (double-bit ECC errors) both require immediate cordoning, while XID 31 and XID 13 (MMU faults) permit a softer response.
- Double-bit ECC errors warrant immediate node cordoning regardless of workload state.
- NCCL hangs, and collective timeouts should be handled by killing the affected replica and attempting a restart. If the same issue repeats more than N times, escalate to on-call. In parallel, check the fabric manager state and verify SHARP tree health, since either can be the root cause.
- InfiniBand and RoCE link flaps call for quarantining the affected node and opening a fabric ticket. UFM events should be pulled and correlated to determine whether the flap is isolated or part of a broader fabric issue.
- If the Fabric Manager service crashes (particularly on NVL72 systems), the first response is automatic restart. If the crashes persist, drain the entire rack rather than individual nodes.
- NVSwitch errors require cordoning off the affected node(s). If rack-scale isolation is needed, drain workloads to non-NVL72 capacity.
- Thermal throttling should trigger load shedding on the affected node. Cooling investigation should be opened with the Network and Compute Platform (NCP) team.
- Silent data corruption is addressed proactively: idle replicas run periodic checksum self-tests, and any node that fails or raises suspicion is cordoned immediately.
- If a node refresh results in a driver or CUDA version mismatch, the admission controller will automatically block scheduling onto that node until the version discrepancy is resolved.
Change management:
- All platform changes flow through GitOps using Argo, so no manual configuration changes reach production outside a reviewed and committed change set.
- New features and model updates are delivered progressively. A rollout starts at canary scale, then moves through 1%, 10%, and 50% of traffic before reaching full deployment. At each stage, SLO metrics are evaluated automatically, and the system rolls back without human intervention if any regression is detected.
- Feature flags are used to isolate risk during launches, allowing new capabilities to be enabled or disabled independently of a deployment boundary.
- A per-model evaluation suite gates model and engine version upgrades. A new version cannot proceed to production if it introduces a quality regression on the golden datasets for that model.
- Driver, CUDA, and NCCL upgrades follow a separate gate: the new version must complete a soak period in the staging pool and pass a benchmark regression check before it is eligible for production rollout.
Requirements Summary
SLOs and Non-Functional Requirements
Published, contractually backed where applicable.
SLO requirements are outlined in the Requirements for AI Clouds Service Delivery SLAs.
Requirements Summary