Introduction

This document outlines a software reference guide intended to help NVIDIA Cloud Partners (NCPs), Cloud Service Providers (CSPs), and Independent Software Vendors (ISVs) build AI cloud services on NCP hardware platforms. It presents an infrastructure-native Northstar reference that supports multi-tenancy and elastic resource allocation.

As inference workloads grow in importance, organizations increasingly require fungible compute resources that can dynamically satisfy multi-tenant training and inference workloads. This shift has driven the need for a cloud-native approach to building and operating AI service stacks.

This document describes how to build such a solution using Kubernetes and modern AI Platform-as-a-Service (PaaS) offerings. The document also provides guidance on optimizing the performance of AI inference and training within a virtualized environment, where workloads can run on either shared or dedicated physical hosts.

Built on top of the NCP Hardware Reference Design, the architecture stack follows a layered approach:

• Infrastructure-as-a-Service (IaaS) for bare metal and virtual machine provisioning
• Container-as-a-Service (CaaS) with managed Kubernetes
• AI Platform-as-a-Service (PaaS) for tenant-facing AI workloads

The layered design enables NCPs to deliver dynamic, multi-tenant AI services competitive with hyperscale cloud service providers. The architecture is infrastructure-native; compute, storage, and networking resources are allocated in an on-demand model rather than statically provisioned.

NCPs can work with an ecosystem of integrated services vendors (ISVs) or implement open-source tools to deliver security and workload isolation. This enables the operation of a concurrent multitenant private cloud that leverages the performance-optimized stack outlined in this reference architecture. NVIDIA certifies the performance of third-party solutions so that NCPs can confidently choose their partner of choice.

Document Structure

The document is organized into two sections:

• Part 1: Software Reference Guide: Describes a layered multi-tenant AI cloud architecture, covering the capabilities that are desired at each layer; IaaS, CaaS, and PaaS. Considerations for network management, workload isolation, observability, and break-fix are mentioned.
• Part 2: Software Components: Provides detailed information on NVIDIA-provided software addressing specific capabilities mentioned in the Software Reference Architecture section. Each entry explains the software component and maps it back to the architectural capabilities. The use of software provided by NVIDIA is optional and depends on architectural decisions.

Audience and Scope

This document is a reference guide, not an implementation manual. It describes capabilities required at each layer and identifies where software provided by NVIDIA may be integrated.

• NCPs: Use this as a checklist of capabilities needed for operating a multi-tenant AI Cloud service. Each section describes what key capabilities are desired in each layer. This reference can be used to assess your current software offering stack to identify gaps and make informed build and buy decisions.
• ISVs: Map your software offerings to this layered architecture framework. The NVIDIA software identified in each section represents integration points and complementary components for your provided solutions.

User Personas

User personas listed in the table below are the roles that interact with, administer, or use the system implemented using the NCP Software Reference Guide.

User Personas

Terms and Definitions

Terms and definitions in this document are defined in the table below.

Terms and Definitions