Design Tenets of NVIDIA Reference Architectures#
NVIDIA Reference Architectures are tailored for enterprise-class deployments, ranging from 32 to 1024 GPUs. Depending on the base technology, they include configurations for 4 up to 128 nodes and 1 up to 8 racks, complete with the appropriate networking topology, switching, and allocations for storage and control plane nodes. Derived from the NCP Reference Architecture but right-sized for enterprise-scale deployments, these RAs provide deployment guides, cluster characterization, provisioning automation using Kubernetes, and sizing guides for common enterprise AI implementations. NVIDIA RAs are designed to support a diverse range of workloads, including fine-tuning, Retrieval-Augmented Generation (RAG), model training, inference, and small-scale High-Performance Computing (HPC) tasks. These designs provide a versatile foundation for enterprise AI with a focus on single-tenant, on-premises deployments.
The following design tenets explain how NVIDIA RAs structure Enterprise AI Factory deployments before the paper moves into the RTX PRO, HGX, and NVL72 family deep dives. These tenets cover the networking model, reference configuration notation, and scalable units used to build repeatable clusters.
Networking#
NVIDIA Reference Architectures include networking guidance that starts at the node and scales to the cluster. The goal is to define the bandwidth, traffic separation, topology, and management requirements needed for production AI performance, security, and supportability. Today, NVIDIA RAs provide Ethernet-based recommendations using the NVIDIA Spectrum-X Ethernet platform, NVIDIA Spectrum switches, NVIDIA ConnectX SuperNICs, and NVIDIA BlueField-3 DPUs. Future RA guidance may also include NVIDIA Quantum InfiniBand options for customers and workloads that require them.
NVIDIA RA networking is organized around distinct traffic types, each with different performance and operational requirements:
East-west networking: Handles GPU compute traffic between nodes. This network is critical for distributed training, fine-tuning, and distributed inference because GPUs must exchange data with high bandwidth and low latency. As clusters scale, east-west networking must scale with the number of compute nodes and GPUs to avoid performance bottlenecks.
North-south networking: Supports customer uplink, storage connectivity, and support and management services. NVIDIA recommends NVIDIA BlueField-3 DPUs for north-south traffic to help provide secure, efficient handling of infrastructure, storage, management, and external communication.
Storage connectivity: Provides the data path for ingestion, retrieval, checkpointing, and result delivery. Storage networking must be sized for the workload so data movement does not limit GPU utilization.
Customer uplink: Connects the AI Factory to customer applications, users, tools, and upstream or downstream enterprise systems. Uplink design should reflect expected inference traffic, data flows, security boundaries, and operational requirements.
Management and support services: Provides connectivity for provisioning, monitoring, orchestration, lifecycle management, and platform operations. These paths should be designed separately from the high-performance data paths so operations remain reliable as compute traffic scales.
As clusters grow, network requirements scale from one node to one scalable unit and then to multi-SU cluster designs. NVIDIA RAs provide topology guidance for different scale points, including consolidated or dedicated north-south and east-west networks, and single-plane or dual-plane designs. This lets partners select the right topology for the customer’s workload, scale, operational model, and budget while preserving predictable performance and supportability.
Reference Configurations#
Each reference architecture is designed around an NVIDIA-Certified server that follows a prescriptive design pattern, called a reference configuration, to ensure optimal performance when deployed in a cluster. Reference configurations standardize the description of compute nodes based on their CPU, GPU, network, and bandwidth configurations. The C-G-N-B nomenclature simplifies system selection by clearly defining compute power, networking capabilities, and bandwidth performance where each field (ex: 2-8-5-200) refers to the ratio of number of CPU sockets, number of GPUs, number of network adapters, and average east-west network bandwidth per GPU (GbE), respectively. With GPU and networking advancements on the horizon, these architectures ensure scalability and future-proofing for enterprise applications.
AI Factory family |
Reference configuration |
CPU |
GPU |
Network adapters |
East-west bandwidth |
Eligible GPUs |
|---|---|---|---|---|---|---|
NVIDIA RTX PRO AI Factory |
2-8-5-200 |
2 CPU |
8 GPU |
5 total: 1 north-south, 4 east-west |
200 GbE per GPU |
NVIDIA RTX PRO 6000 / RTX PRO 4500 BSE |
NVIDIA RTX PRO AI Factory |
2-4-3-200 |
2 CPU |
4 GPU |
3 total: 1 north-south, 2 east-west |
200 GbE per GPU |
NVIDIA RTX PRO 6000 / RTX PRO 4500 BSE |
NVIDIA RTX PRO AI Factory |
2-X-1-NA |
2 CPU |
2, 4, or 8 GPU |
1 total: 1 north-south |
N/A |
NVIDIA RTX PRO 6000 / RTX PRO 4500 BSE |
NVIDIA HGX AI Factory |
2-8-9-800 |
2 CPU |
8 GPU |
9 total: 1 north-south, 8 east-west |
800 GbE per GPU |
NVIDIA HGX B300 |
NVIDIA NVL72 AI Factory |
2-4-5-800 |
2 CPU |
4 GPU |
5 total: 1 north-south, 4 east-west |
800 GbE per GPU |
NVIDIA GB300 NVL72 |
Table 2: Examples of Reference Configuration Node and Networking Patterns: C-G-N-B notation represents CPU sockets, GPUs per server, network adapters, and average east-west bandwidth per GPU. BSE = Blackwell Server Edition. This table represents an exemplary subset of available NVIDIA Reference Architecture configurations.
Scaling Units#
NVIDIA Reference Architectures define how individual nodes become deployable Enterprise AI Factory clusters. The starting point is the node: a validated server design with the right CPU, GPU, memory, storage, NICs, and base networking requirements. The next step is the scalable unit.
A scalable unit, or SU, is the repeatable building block used to size and grow an AI Factory. It is not an arbitrary node count. The SU is the increment around which cluster topology, east-west networking, power, cooling, rack layout, and deployment planning are organized. By scaling in predefined units, partners and customers can add capacity predictably while preserving the balance between compute, networking, and infrastructure.
For the NVIDIA RTX PRO AI Factory and NVIDIA HGX AI Factory families, one scalable unit is four nodes. The minimum viable AI Factory configuration is one SU, or four nodes. Customers can then grow in four-node increments as workload demand increases.
NVIDIA RTX PRO AI Factory: scales from 1 SU to 8 SUs, or 4 to 32 nodes, supporting up to 256 GPUs.
NVIDIA HGX AI Factory: scales from 1 SU to 32 SUs, or 4 to 128 nodes, supporting up to 1,024 GPUs.
This gives customers a practical answer to two common design questions: how small they can start, and how they can add capacity later without redesigning the cluster from scratch.
For the NVIDIA NVL72 AI Factory family, the scalable unit is different. NVL72 scales by rack, not by four-node increments. One NVL72 scalable unit is one full rack with 18 compute trays. The minimum sensible deployment is therefore one rack, and expansion happens one rack at a time. NVIDIA NVL72 AI Factory configurations scale from 1 up to 8 racks.
As clusters grow, the network design must evolve with the scale of the deployment. A topology that works for one node or one scalable unit may not be appropriate across many SUs. This is why the reference architecture progresses from node-level requirements, to one scalable unit, to multi-SU clusters, with different east-west networking patterns introduced as the cluster grows. The goal is to preserve performance, utilization, supportability, and predictable expansion as the AI Factory moves from initial deployment to production scale.
AI Factory family |
Scalable unit |
Minimum deployment |
Expansion increment |
Maximum scale |
|---|---|---|---|---|
NVIDIA RTX PRO AI Factory |
4 nodes |
1 SU / 4 nodes |
4 nodes at a time |
8 SUs / 32 nodes / up to 256 GPUs |
NVIDIA HGX AI Factory |
4 nodes |
1 SU / 4 nodes |
4 nodes at a time |
32 SUs / 128 nodes / up to 1,024 GPUs |
NVIDIA NVL72 AI Factory |
1 rack / 18 compute trays |
1 rack |
1 rack at a time |
8 racks |