NVIDIA Inference Reference Architecture

Introduction

This reference architecture describes an inference-focused software stack for NVIDIA Cloud Partners. It helps NCP substrate operators and ISP inference-platform operators build a cloud-native platform that can host large language model services, multimodal services, traditional machine learning inference, asynchronous GPU tasks, and partner-specific AI platforms on a shared NVIDIA accelerated infrastructure.

The design assumes the provider platform behaves more like a cloud service than a static cluster. Inference-service consumers should be able to request endpoints, models, Kubernetes capacity, GPU workers, model storage, and service support without understanding every physical detail in the data center. NCP substrate operators and ISP inference-platform operators still need clear architectural interfaces for placement, isolation, performance, health, and lifecycle events.

The architecture uses a layered approach. It starts with data center assumptions, then shows the inference-service consumer view and the NCP-to-ISP architecture boundary, then maps those views to infrastructure, Kubernetes, AI platform services, inference serving, model data movement, validation, telemetry, fault effects, performance, and security.

The Inference Provider Platform Requirements define the normative NCP requirements, NCP/ISP responsibility boundary, component requirements, validation obligations, and Day-2 operational requirements for the selected provider profile. This RA defines the end-to-end inference architecture, workload behavior, component relationships, and inference request and data flows. Where the documents touch the same subject, this RA describes how the architecture consumes the platform capability and links to the Platform Requirements instead of repeating requirements, ownership tables, validation gates, or runbooks.

Executive Summary

NVIDIA inference deployments now need to handle large language models, multimodal models, traditional ML services, asynchronous GPU tasks, and platform APIs across many GPU nodes. This architecture defines a repeatable stack for those needs: delivered provider capabilities at the base, topology-aware scheduling and health-aware placement in the inference platform, model and cache movement in the data plane, model-serving orchestration above inference engines, and workload benchmarking around the full system.

The architecture defines component roles and integration points for a complete inference platform. Claims that require lab results are framed as validation work to run, not as completed proof.

Target Audience

• NVIDIA Cloud Partner platform teams building inference services on GPU Kubernetes clusters.
• Solution architects mapping NVIDIA software components to partner infrastructure.
• Inference architects and platform engineers who need to understand how inference services, workloads, and components interact on an accepted provider platform.
• Performance and validation teams that need repeatable source, version, and benchmark evidence.
• Product and field teams that need a current architecture view tied to validated components.

Customer Problem

Inference platforms are no longer a single model server behind a load balancer. Large models can require multi-node placement, separate prefill and decode pools, fast model-weight movement, cache-aware routing, GPU-aware scheduling, and operational remediation when GPU or network health changes. Partner teams need a way to assemble these capabilities without losing traceability to the component versions and design boundaries that define the stack.

Design Goals

• Provide a repeatable full-stack blueprint for partner inference deployments.
• Separate infrastructure, model data, serving, optimization, validation, and operations concerns.
• Make common deployment combinations explicit so teams can adopt the full stack or a narrower subset.
• Document reference component versions, workload-validation inputs, deployment flow, and architecture boundaries.
• Make backend disposition, tenancy, routing, model-data movement, cache behavior, and release validation visible instead of leaving them as local implementation choices.
• Keep unproven performance, support, and validation claims out of the reader-facing architecture.

Architecture Decision Guardrails

The base RA should record inference-architecture decisions and show where provider evidence enters those decisions, not mandate provider ownership or a single implementation. Treat partner-selected ecosystem software, physical cluster shape, DPU placement, fabric topology, storage product, gateway implementation, and provisioning tool as architecture inputs governed by the selected provider profile.

Data Center Architecture

The provider platform is an external contract consumed by this RA. The Inference Provider Platform Requirements define how the NCP builds, validates, versions, and operates the infrastructure. The RA records only the delivered capabilities and interfaces that affect inference architecture: GPU resource shape, topology, NUMA locality, network attachments, storage classes, health signals, and lifecycle events.

GPU Compute Node

GPU compute nodes run the endpoint workers, prefill workers, decode workers, model data-plane services, benchmark jobs, and supporting sidecars. The node design should account for GPU memory capacity, CPU memory, local NVMe capacity, NIC count, NIC speed, PCIe topology, and GPU-to-GPU topology. For large models, node placement must be considered part of the architecture because a poor placement decision can turn a valid software stack into a slow service.

The RA describes how inference workloads use the delivered GPU, NIC, topology, NUMA, storage, and health interfaces. It does not redefine the provider bill of materials, firmware, operating system, driver deployment, inventory, or node lifecycle.

Networking

The inference platform uses multiple networks with different responsibilities.

• Tenant access traffic carries user APIs, control plane calls, endpoint ingress, storage access, registry access, and shared-service traffic.
• Secure management traffic carries provider administration, out-of-band management, provisioning, and provider automation; its requirements and procedures remain in the Platform Requirements.
• Cluster interconnect traffic carries east-west model serving traffic, distributed inference coordination, model data movement, and high-volume cache or tensor movement when the workload profile requires it.
• NVLink traffic carries local scale-up GPU communication inside the supported GPU topology.

For disaggregated serving, network design must be reviewed together with serving topology. Prefill pools, decode pools, routers, cache services, and model-weight transfer services should be placed so the highest-volume traffic remains on the most appropriate fabric. RDMA, GPUDirect RDMA, separate east-west compute fabric, DPU offload, and dual-plane topology are decisions to validate against the selected RTX PRO, HGX, GB300 NVL72, cloud-hosted, or lab profile.

The Platform Requirements own the network requirements, acceptance evidence, ownership, and operational procedures. This RA maps inference traffic classes to delivered interfaces and explains their architectural effect on request routing, distributed serving, collective communication, KV transfer, latency, throughput, and recovery behavior.

Storage

Inference needs more than one storage tier. Model artifacts often start in object or file storage. Hot models may need node-local cache, shared cache, or GPU-to-GPU transfer paths. Logs, traces, metrics, benchmark outputs, and validation reports have different access and retention characteristics from model storage. Local NVMe can support image cache, model cache, temporary tensors, and short-lived logs when that capability is part of the provider handoff.

The architecture should distinguish persistent model artifact storage from ephemeral cache and from telemetry retention. ModelExpress, NIXL, Velo, FlexTensor, and model streaming components sit in this boundary between storage, memory, and runtime workers.

The Platform Requirements own storage qualification, evidence, sanitization, media-wear, break-fix, and operational procedures. This RA maps model artifacts, caches, KV data, benchmark outputs, and telemetry to delivered storage interfaces and explains their effect on model loading, caching, recovery, latency, and throughput.

Data Center View

At the data center level, the inference platform is a set of control planes and data planes connected through delivered platform interfaces. The RA records which ISP-owned services run as shared control services, per-tenant services, DaemonSets, or endpoint-specific workers and which inference components consume GPUs, NICs, topology, or local storage. Provider control-plane and storage-service placement remains in the Platform Requirements and selected provider profile.

NCP Inference Reference Architecture Introduction

Inference Service Consumer View

From a tenant point of view, the platform exposes AI services, Kubernetes resources, model endpoints, and operational status. The tenant should not need to understand the entire data center, but the platform must preserve enough topology and resource intent to run high-performance inference.

NCP-to-ISP Architecture Boundary

The Inference Provider Platform Requirements are the source of truth for NCP and ISP ownership. In this RA, NCP substrate operator means the provider role that delivers the managed platform; ISP inference-platform operator means the role that builds and runs the inference stack; and inference-service consumer means the application or user consuming the service. The RA describes the interfaces and information flows between these roles without redefining their responsibilities.

NCP Inference Reference Architecture

Solution Overview

The architecture uses a layered model. Platform APIs and workload endpoints sit above the serving layer. Serving frameworks coordinate inference engines and request flow. Optimization tools prepare models and execution paths. Model data services move weights, tensors, KV cache blocks, and large payloads. Kubernetes infrastructure components provide GPU enablement, networking, scheduling, placement, health, and repeatable recipes. Benchmarking and validation tools close the loop.

Component Layers

The Platform Requirements define which platform components are required or conditional, who owns them, their supported versions, their validation requirements, and their lifecycle obligations. This RA records the provider capabilities consumed by the inference architecture and describes the relationships among inference components.

Common Component Combinations

The Platform Requirements define required and conditional platform components, ownership, supported versions, validation, and lifecycle obligations. The combinations below are non-normative architecture patterns that show how inference components relate and interact.

Visual Architecture Views

The following views show the same inference platform from different operating angles. Use them during design review to confirm that the architecture has a complete layered stack, a clean control-plane and data-plane boundary, explicit network and storage interfaces, multi-cluster routing semantics, and a clear handoff from provider evidence into workload decisions. Review each view with the reference component snapshot and applicable Platform Requirements so the diagram remains an architecture aid rather than a static illustration.

Layered Inference Factory View

Use this view to confirm that every tenant-facing endpoint has a path through platform API, serving orchestration, model data movement, Kubernetes orchestration, accelerated infrastructure, and workload validation. Missing inference-component relationships or unlinked platform prerequisites indicate an incomplete architecture.

Control Plane And Data Plane Split

Use this view to separate policy, planning, scheduling, and health control from the hot path that routes requests, moves weights, manages KV cache, and executes inference. The split should be explicit in namespace design, service-account policy, placement policy, telemetry, and failure handling.

Network And Storage Readiness View

Show the provider network and storage capabilities consumed by each inference flow and link to the Platform Requirements for the applicable requirements and evidence. Do not reproduce provider qualification tests or pass/fail criteria.

Multi-Cluster Routing View

Use this view when an endpoint can run in more than one cluster, region, or availability zone. The architecture should define the routing inputs, cache-locality signals, health checks, compliance tags, and fallback behavior before endpoint traffic is distributed across clusters.

Platform Evidence Handoff View

Show the handoff from Platform validation evidence into the inference architecture. Link each prerequisite to its Platform requirement, then use the RA to explain the affected workload, component relationship, or inference flow.

Infrastructure-as-a-Service Architecture

The external provider contract defined in Data Center Architecture also applies to bare metal, virtual machines, managed Kubernetes clusters, and integrated AI platforms. This section focuses on the IaaS interfaces that the inference architecture consumes.

Cloud And Cluster Control Plane

The inference architecture consumes provider APIs that express GPU capacity, model endpoint capacity, network capability, storage access, isolation boundaries, service objectives, health, and lifecycle events. The Platform Requirements own the provider control-plane implementation and its operating procedures.

Where NVIDIA AI Cluster Runtime is part of the selected provider profile, its delivered recipes and snapshots are provider evidence consumed at the handoff. The RA does not make AI Cluster Runtime or its operational workflow a universal platform requirement.

Compute Service

Apply the workload-facing scope defined in GPU Compute Node to this service; provider provisioning, sanitization, and lifecycle requirements remain outside the RA.

For inference, compute placement must account for GPU topology, NIC topology, model size, prefill and decode roles, cache locality, endpoint isolation, and failure domains. Full-node allocation is simplest for large LLM services. Smaller services may use MIG or virtualized GPU paths, but those choices must be validated against latency, throughput, and isolation requirements.

Software Defined Network Layer

The inference architecture maps tenant access, distributed serving, collective communication, model-data movement, and KV transfer to the network interfaces delivered by the selected provider profile. RDMA, GPUDirect RDMA, DPU offload, dual-plane networking, and local NVLink scale-up remain conditional capabilities whose architectural effect depends on the workload.

The Platform Requirements own physical fabric, addressing, CNI, tenant-isolation, Network Operator, and acceptance requirements. The RA describes how inference components consume network attachments, locality, bandwidth, latency, congestion, and health signals without redefining provider ownership or configuration.

Software Defined Storage Layer

The inference architecture maps persistent model artifacts, endpoint configuration, benchmark artifacts, logs, traces, metrics, and ephemeral cache to the file, object, block, and local storage interfaces delivered by the provider platform.

The model data-plane components in this architecture do not replace storage. They improve the path from storage to serving workers by coordinating cache state, streaming tensors, moving weights, and staging large payloads across GPU and host memory tiers.

Container-as-a-Service: Kubernetes

Kubernetes is the primary orchestration layer for cloud-native inference workloads. It provides declarative APIs, controllers, scheduling, service discovery, horizontal scaling, namespace isolation, and a consistent packaging model for model-serving services and platform services.

Kubernetes Usage And Personas In ML/AI

AI practitioners need reproducible environments, model endpoints, benchmark feedback, and access to GPU capacity. Developers need stable APIs, deployment workflows, traffic routing, and observability. ISP inference-platform engineers need placement controls, workload isolation, serving lifecycle, and the provider health and lifecycle interfaces that affect inference services.

The inference architecture uses Kubernetes for three jobs:

• Consuming the Kubernetes API, quotas, GPU and network resources, topology, storage, health, and lifecycle interfaces exposed by the provider.
• Hosting ISP-owned workload controllers, schedulers, serving components, and tenant or endpoint-specific inference workloads.
• Hosting workload telemetry, model data-plane, routing, and inference-service automation components.

Kubernetes Architecture For Inference

The RA begins at the provider-to-ISP handoff defined by the Inference Provider Platform Requirements. It consumes the Kubernetes API, quotas, GPU and network resources, topology, storage, health, and lifecycle interfaces exposed by the provider. ISP-owned workload controllers, schedulers, and serving components use those interfaces to implement queueing, gang scheduling, placement, startup ordering, routing, and autoscaling.

KAI Scheduler provides GPU-aware queueing and allocation policy. Grove provides gang scheduling, startup ordering, and topology-aware placement for multi-pod inference units. GPU Operator and Network Operator appear here only as platform capabilities exposed through the provider handoff; their requirement status, ownership, installation, validation, and lifecycle remain in the Platform Requirements.

These components matter because inference services are often not independent pods. A large service may need routers, prefill workers, decode workers, cache services, sidecars, and model-data services to start and scale coherently. If only part of the service schedules, GPUs can sit idle while the endpoint remains unhealthy.

The Kubernetes Gateway API Inference Extension can be included as a cloud-native interoperability layer when the platform needs Kubernetes-native InferencePool and InferenceModel resources. Treat that extension as an API integration decision, not as a required scheduler, serving backend, or replacement for the Dynamo serving control path.

The RA uses CRDs, scheduler events, pod-group placement, GPU allocation, network attachments, namespace boundaries, service accounts, node selectors, topology keys, and queue policy to explain workload behavior. Required installation, validation, upgrade, rollback, and cluster-acceptance procedures remain in the Platform Requirements.

AI Platform-as-a-Service

The AI platform layer turns GPU infrastructure into user-facing inference services. It owns endpoint lifecycle, function or task lifecycle, model-serving APIs, routing policy, rate limits, identity hooks, user observability, and integration with model artifacts.

Backend selection is an architectural decision. Record which model classes use TensorRT-LLM, TensorRT, Dynamo-managed workers, or another compatible path; document known limitations; and define a rollback path when a runtime change affects serving behavior.

Multi-cluster routing should separate endpoint routing, cluster routing, model routing, and cache routing. A request should cross clusters only when policy, locality, compliance tags, health, capacity, trace context, and cache state are visible to the routing layer.

Cloud-Native Inference Gateway Integration

Inference Serving Flow

1. A user or application calls a platform API or inference endpoint.
2. The platform authenticates the call, applies policy, and routes it to the endpoint control plane.
3. The serving layer selects runtime workers and applies routing, batching, prefill, decode, and cache policy.
4. The model data layer supplies model artifacts, weight transfer, KV or tensor movement, and large-payload staging.
5. Kubernetes and scheduler components maintain worker placement, readiness, and scale.
6. Telemetry and benchmark systems compare live behavior with accepted baselines.

Model Optimization And Runtime Preparation

Model optimization is a separate architectural layer because it changes the artifact that enters serving. The optimization path should be selected before final benchmark acceptance, not after a service is already in production.

New model bring-up should use versioned recipes rather than one-off tuning notes. Each recipe should define the quantization path, backend path, serving topology, benchmark profile, accuracy gate, artifact provenance check, and rollback path before it is recommended for a partner service profile.

Model Data And Memory Architecture

Large-scale inference can bottleneck on model movement, cache locality, GPU memory pressure, and startup time. The model data plane sits between storage and serving workers and should be designed as deliberately as the serving runtime.

KV cache ownership, transfer, eviction, recovery, and observability must be explicit in the design. Host memory, local SSD, remote memory, and peer-to-peer transfer should be treated as planned tiers with accepted failure behavior rather than emergency overflow paths. Validation should include cache-aware routing behavior, worker restart behavior, cache-hit-rate tracking, and local-SSD wear review where SSD-backed cache or offload is used.

Model startup should be decomposed into artifact discovery, cache warmup, weight movement, container startup, backend initialization, and first-ready signaling. Record model download time, cache-hit time, peer transfer time, container ready time, backend ready time, first token time, and restart recovery time for every accepted service profile.

Data Flow Diagrams

The inference stack supports multiple data paths, not one monolithic service path. These diagrams show the major GenAI, traditional ML, and deployment flows that architecture and workload teams should evaluate.

GenAI/LLM Inference Flow

Traditional ML Inference Flow

Model Deployment Flow

Key Component Interactions

Disaggregated LLM Serving

The following is an example disaggregated serving pattern using publicly documented Dynamo-ecosystem components. Partners may substitute equivalent interfaces if they preserve routing, placement, cache, and transfer ownership boundaries.

In this example, the critical interactions are Planner with Grove for right-sized prefill and decode capacity, Router with the KV Event plane and KVIndexer for cache-state updates and cache-aware placement, and KV Block Manager with NIXL for memory-tier movement.

Kubernetes Infrastructure Stack

Component Interaction Matrix

Getting Started

Choose an adoption path based on the first inference-service problem the ISP inference-platform operator needs to solve.

Full Stack Deployment

1. Select the inference architecture and applicable provider profile.
2. Obtain the managed-platform handoff and links to the Platform Requirements, evidence, and operating procedures.
3. Deploy the ISP-owned workload, serving, routing, model-data, and runtime components.
4. Apply the workload topology, placement, and scaling design described by the RA.
5. Use the Platform Requirements validation and Day-2 procedures instead of restating them in the RA.

Traditional ML Inference Only

1. Optimize model artifacts with TensorRT, Model Optimizer, AITune, or DALI where relevant.
2. Deploy the service on GPU-enabled Kubernetes.
3. Validate latency, throughput, and preprocessing behavior with AIPerf.

GenAI/LLM Inference Only

1. Select the serving backend and runtime path.
2. Use AIConfigurator to narrow prefill, decode, backend, and GPU-count choices.
3. Deploy Dynamo with TensorRT-LLM or the selected backend.
4. Add KV, NIXL, ModelExpress, Grove, and KAI Scheduler when the service needs multi-node scale or fast model movement.

Kubernetes Integration Only

1. Select the provider profile and obtain the managed-Kubernetes handoff defined by the Platform Requirements.
2. Map the delivered Kubernetes API, quotas, GPU and network resources, topology, storage, health, and lifecycle interfaces to the inference workload.
3. Add KAI Scheduler for GPU-aware allocation and Grove when workloads require coordinated multi-pod placement.
4. Link provider validation and Day-2 procedures rather than duplicating them in the RA.

Example Workload: Large MoE LLM Inference

A large mixture-of-experts LLM service is the clearest example of why this architecture is disaggregated. The workload can require separate compute profiles for prefill and decode, cache-aware routing, topology-aware placement, fast model-weight movement, and benchmark-driven configuration.

Core Design Philosophy

The service should separate control plane, request routing, prefill, decode, KV cache management, model artifact movement, and operations. Each subsystem scales against a different bottleneck. Treating them as one monolithic deployment hides the bottleneck and makes capacity planning harder.

Key Architectural Components

• Dynamo coordinates distributed serving and backend workers.
• Planner and AIConfigurator narrow the prefill/decode and parallelism choices before broad benchmark runs.
• Router, KV Event plane, and KVIndexer use cache-state updates to reduce redundant prefill work through cache-aware placement.
• KV Block Manager and NIXL manage cache blocks and move them across memory tiers.
• NIXL and ModelExpress accelerate model data and cache movement.
• Grove and KAI Scheduler keep multi-pod serving units schedulable and topology aware.
• AIPerf and the validation suite turn the design into acceptance evidence.

Reference Architecture

Deployment Recommendations

1. Start with disaggregated serving for high-throughput or long-context services.
2. Keep prefill and decode placement topology aware.
3. Measure model load, first-ready time, time to first token, inter-token latency, and throughput together.
4. Use cache-aware routing only with enough telemetry to prove cache locality helps the target workload.
5. Treat performance claims as workload-specific until the partner validation report records the hardware, model, backend, traffic profile, and versions.

Logical Architecture

1. Clients call a platform API, an invocation endpoint, or a model-serving endpoint.
2. The API and serving layer applies routing, authentication hooks, admission policy, and endpoint-level workload controls.
3. The serving layer selects an inference backend and coordinates request routing, scaling, prefill, decode, and runtime workers.
4. The model data layer stages model artifacts, streams weights, exchanges large payloads, and manages memory-tier movement where the selected components support those paths.
5. Kubernetes orchestration places the workload on GPU nodes, configures GPU and network resources, schedules related pods, and exposes health information.
6. Benchmarking and validation tools measure latency, throughput, startup, configuration, and environment readiness.
7. Operations tooling feeds health events, remediation workflows, and release-state changes back into the next architecture refresh.

Use this sequence to clarify inference-component boundaries. Record which inference component makes each control-plane decision, which component performs each data-plane movement, which signal makes the transition observable, and which architectural rollback returns the service to the last working state. Link provider-owned decisions and procedures to the Platform Requirements.

Review the logical flow when the serving design changes. Resolve inference-component gaps in the RA and track provider ownership questions through the external-decision gate and the Platform Requirements.

Physical Architecture

The physical deployment starts with one or more GPU Kubernetes clusters. A production partner environment maps the logical layers to concrete availability zones, clusters, racks, GPU nodes, NICs, storage services, container registries, and Kubernetes operator namespaces. This architecture does not assume one fixed rack shape because the component set can run on cloud, self-managed, and lab environments.

Select the hardware profile before turning topology into requirements. RTX PRO, HGX, and GB300 NVL72 profiles differ in GPU form factor, rack density, local scale-up, power, cooling, fabric, DPU placement, and tenancy. Confirm implementation details with the NVIDIA-approved, access-controlled hardware reference for the selected profile.

Provider-profile constraints apply only when that profile is selected. RTX PRO, HGX, cloud-hosted, VM, and lab profiles retain their own infrastructure assumptions while sharing the RA’s inference architecture patterns.

Workload-facing physical inputs for an architecture review:

• GPU node type, GPU count, memory capacity, CPU memory, local storage, and PCIe or NVLink topology.
• East-west network type, RDMA readiness, GPUDirect RDMA readiness, and switch-domain boundaries.
• Storage source for model artifacts, local cache capacity, shared cache options, and air-gapped behavior.
• Delivered Kubernetes, GPU, network, storage, health, and topology interfaces plus ISP-owned scheduler configuration.
• Placement rules for multi-node model instances, prefill pools, decode pools, control-plane services, and telemetry.

Record the delivered node labels, scheduler topology keys, storage classes, network attachments, GPU topology, and failure domains that the inference architecture consumes. The Platform Requirements and selected provider profile own the bill of materials, firmware baseline, rack and rail topology, storage qualification, and management-plane requirements.

Deployment Model

The Deployment Model describes the architectural progression from provider-platform handoff to an operating inference service: delivered platform capabilities, ISP inference components, workload placement, model and cache data, serving, routing, and application integration. The Platform Requirements own the NCP/ISP operational responsibilities, validation gates, maintenance process, incident process, upgrades, remediation, and Day-2 runbooks.

Use the RA to select the serving design, map delivered interfaces to workload needs, place the ISP-owned components, trace request and data flows, and explain the architectural effect of rollback. Link deployment evidence and operating procedures to the owning Platform Requirements instead of reproducing them here.

Validation Methodology

The Inference Provider Platform Requirements define required platform validation, accountable owners, evidence, release gates, and operational acceptance criteria. The RA may show where that evidence enters an architectural decision, but it does not repeat the requirement or test procedure.

Workload-performance validation remains part of the RA. For each workload study, record benchmark provenance including the model, tokenizer, backend, container image, hardware profile, network mode, storage path, prompt profile, output profile, concurrency, cache state, and software versions. Use the results to interpret time to first token, inter-token latency, throughput, startup time, routing behavior, model-data movement, and the effect of architecture changes; do not present them as provider certification criteria.

Performance Guidance

Start with configuration search before exhaustive load testing. Use the configuration tool to narrow prefill, decode, parallelism, backend, and GPU-count choices. Then use endpoint benchmarks to measure token latency, request latency, throughput, and concurrency behavior under realistic traffic. For model-start performance, collect model download, cache-hit, weight-transfer, and first-ready timestamps. For operations, track GPU health events, recovery time, failed placements, and capacity headroom.

Do not compare configurations unless the model, tokenizer, prompt distribution, output length distribution, GPU type, driver, network, serving backend, and concurrency profile are recorded together.

Speculative decoding, custom kernels, quantization, and model-specific tuning should be treated as controlled optimization inputs. Measure acceptance rate, accuracy delta, latency impact, backend compatibility, and rollback behavior before turning an optimization on by default.

Maintain a per-backend performance baseline and compare new results only against baselines that share the same model, hardware, traffic profile, and software versions.

Sizing Guidance

Use inference software patterns and provider profiles together. The inference pattern describes component relationships and workload behavior. The selected provider profile supplies the validated GPU, network, storage, topology, and isolation capabilities the pattern consumes.

• Single-node inference pattern: TensorRT or TensorRT-LLM can combine with AITune, Model Optimizer, DALI, and AIPerf while consuming the delivered GPU interface.
• Multi-node LLM inference pattern: Dynamo can combine with ModelExpress, NIXL, Grove, AIConfigurator, and scheduler controls while consuming delivered topology and network interfaces.
• Partner platform inference pattern: NVIDIA Cloud Functions can provide the platform API while the inference stack consumes provider-exposed Kubernetes, GPU, network, health, and lifecycle capabilities.
• RTX PRO profile: use for agentic AI, visual computing, physical AI, simulation, data processing, and small or medium LLM services where PCIe GPU server modularity and enterprise scalability are the starting point.
• HGX profile: use for large GPU nodes and multi-node LLM profiles that need HGX topology and high-performance cluster networking.
• GB300 NVL72 profile: use for rack-scale services that need NVL72 local scale-up, liquid cooling, dual-plane networking, and tightly controlled model-parallel placement.

Scale only after measuring the bottleneck. Add GPU capacity for compute saturation, add model-data acceleration for cold-start and artifact movement, add scheduling controls for placement failures, and add network controls for high east-west transfer pressure.

For each workload profile, document the starting GPU count, target concurrency, prompt and output shape, model size, cache policy, consumed network and storage capabilities, and expected autoscaling trigger. Measure the profile with AIPerf or an equivalent endpoint benchmark before applying its conclusions to another model family.

Review sizing profiles after each benchmark run and update the accepted profile only when the measured bottleneck, mitigation, and rollback threshold are recorded.

Telemetry And Observability

The RA shows how provider health signals and ISP application signals flow into inference decisions such as routing, autoscaling, placement, admission, cache recovery, and service availability. The Platform Requirements define which signals are required, who exposes and consumes them, and how they are used in validation and Day-2 operations. Logs, metrics, and traces remain correlated with model, endpoint, tenant, GPU, node, scheduler, and network context in the inference architecture.

The hot path supports real-time operations: dashboards, alerting, incident response, and service debugging. The cold path supports planning: capacity analysis, regression tracking, cost attribution, validation history, and long-term trend analysis. Keep both paths connected through stable identifiers for tenant, endpoint, model, node, GPU, and request.

Cache-hit-rate, prefill saturation, decode saturation, model-load state, queue depth, backend version, and routing decision context should be first-class operating signals. These metrics are required to distinguish a serving bottleneck from a cache, model-data, scheduler, or network bottleneck.

Break-Fix Architecture

The RA may illustrate the architectural effect of a node, GPU, fabric, or workload fault and the resulting inference-service flow. Required detection, escalation, cordon/drain, remediation, reboot, RMA, and return-to-service procedures belong in the Inference Provider Platform Requirements.

Node Level Health Checks

The RA correlates provider node, GPU, fabric, and lifecycle signals with Kubernetes workload state and inference endpoint signals so that faults can be translated into routing, admission, placement, cache-recovery, and availability effects. Required health checks, thresholds, escalation, remediation, and return-to-service procedures remain in the Platform Requirements.

Performance Requirements

Inference performance depends on GPU access, network paths, model artifact availability, scheduling locality, and runtime configuration. Measure each workload profile rather than assuming one benchmark generalizes to every model.

Virtual Machine And Container Networking

High-performance inference workers should avoid unnecessary network abstraction on latency-sensitive or high-bandwidth paths. Where the design requires direct NIC access, use SR-IOV, RDMA, GPUDirect RDMA, or equivalent platform mechanisms. Standard CNI networking may still be appropriate for control traffic, user APIs, and lower-volume service calls.

GPU Exposure

Model And Storage Performance

Model load and cache behavior can dominate service readiness. Measure download time, cache-hit time, disk-to-GPU time, peer-to-peer transfer time, first-ready time, and first-token behavior. Use ModelExpress, NIXL, model streaming, and local cache only where they map to an observed bottleneck.

Serving Performance

Serving tests should record time to first token, inter-token latency, request latency, output throughput, concurrency, error rate, model size, prompt distribution, output distribution, backend, GPU type, and software versions. Use AIConfigurator to reduce the configuration search space and AIPerf to measure endpoint behavior.

For workload guidance, keep the command or workload definition, environment metadata, benchmark output, and comparison criteria together. Comparisons that omit backend version, model artifact revision, network mode, or cache state are not sufficient for an architectural conclusion.

Isolation And Security

Isolation and security span the infrastructure, Kubernetes, AI platform, model artifact, telemetry, and user-access layers. The RA shows how delivered isolation capabilities affect inference workloads and how ISP inference-platform actions remain auditable; normative provider controls belong in the Platform Requirements.

Security And Compliance

Security review should cover identity boundaries between platform APIs, model-serving services, ISP inference-platform operators, and cluster agents. Secrets, model artifacts, network policies, and telemetry boundaries affect inference-service architecture and must align with the delivered provider controls. The Platform Requirements own normative provider security and validation requirements.

Workload Isolation

The RA records the tenant boundaries delivered at the infrastructure, Kubernetes, platform API, model artifact, telemetry, and network layers and explains how those boundaries affect placement, routing, model-data access, and service design. Bare metal, virtual machines, and Kubernetes namespaces provide different isolation properties; the selected provider profile defines which mechanisms and requirements apply.

For managed Kubernetes and AI platform services, per-tenant control planes or strongly isolated control-plane partitions can reduce cross-tenant blast radius. The inference architecture consumes the tenant-aware access interfaces delivered by shared registries, identity services, metadata services, and observability backends.

Tenancy should be visible to the platform API, scheduler, router, model-data services, tracing, and telemetry. Validate tenant-aware routing, service-account scope, secret access, model artifact access, trace partitioning, quota behavior, and noisy-neighbor behavior under load.

Boot And Attestation

When the selected provider profile exposes boot or attestation state, the RA may use that state in workload admission, placement, and isolation decisions. The Platform Requirements own secure-boot, measured-boot, remote-attestation, firmware, DPU, BMC, and node-handoff requirements.

Shared Responsibility Model

The Inference Provider Platform Requirements are the source of truth for NCP and ISP ownership. See NCP-to-ISP Architecture Boundary for the role definitions used in this RA.

External Decisions Required

Resolve the following items in the Platform Requirements or another authoritative decision record before using them to change RA wording. Until resolved, the RA must describe the affected interface or information flow without assigning ownership or inventing a requirement.

• GPU software pinning and canary ownership, including the relationship between KUB-3, KUB-10, KUB-11, and KUB-31 INFO.
• The meaning and scope of GPU Operator “can be disabled.”
• CNI ownership after installation and the meaning of “where allowed.”
• NVSentinel and remediation-controller ownership across TEL-1, KUB-36 INFO, and OPS-8.
• Validated component-combination ownership and approval authority.
• Validation and SLO ownership where requirement IDs or owner fields are absent.
• Document precedence and the authoritative cross-linking rule.

Operations And Lifecycle Management

The RA tracks inference runtime, model-data service, scheduler, routing, workload, and API relationships as the inference stack evolves. Keep workload benchmark provenance for each architecture pattern and explain the service effect and architectural rollback when component relationships or delivered provider interfaces change.

The Platform Requirements own NCP/ISP operational responsibilities, validation gates, maintenance, incidents, upgrades, remediation, and Day-2 runbooks. Link those procedures from the selected provider profile rather than reproducing them in the RA.

Reference Component Version And Source Snapshot

This table records the components used to explain the reference architecture. It is not a requirements, support, or compatibility matrix. Validated component combinations, supported versions, and ownership belong in the Inference Provider Platform Requirements. Exact pins belong in the provider-profile evidence, not in this public reference architecture.

Design Alternatives And Tradeoffs

• Full stack versus subset: the full stack gives platform coverage, but a narrower subset is appropriate for single-node or traditional ML services.
• Aggregated serving versus disaggregated serving: aggregated serving is simpler, while disaggregated serving gives separate control over prefill, decode, and model-data movement.
• Static placement versus topology-aware scheduling: static placement is easier to reason about, while topology-aware scheduling is better for multi-node and tightly coupled inference units.
• Shared storage versus peer-to-peer transfer: shared storage is familiar, while peer-to-peer transfer can reduce duplicate downloads and cold-start pressure when the environment supports it.
• Online benchmarking versus offline configuration search: offline search reduces the test space, but final acceptance still requires measured workload behavior.
• RTX PRO versus HGX versus GB300 NVL72: each hardware profile has different GPU, rack, network, DPU, power, cooling, and tenancy assumptions, so choose the profile before writing mandatory topology requirements.
• Direct platform API versus Kubernetes Gateway API Inference Extension: direct platform APIs can keep endpoint behavior fully inside the NCP control plane, while the Gateway API option provides Kubernetes-native InferencePool and InferenceModel integration for teams that need that API surface.
• Required fabric versus validated fabric decision: large disaggregated services may require high-performance east-west paths, while single-node or pure endpoint profiles may not need the same compute fabric.
• DPU-enabled design versus host-only design: DPU offload can improve isolation and infrastructure control for selected profiles. Confirm DPU placement and validation requirements with the NVIDIA-approved, access-controlled hardware reference for the selected profile.
• Base RA versus partner implementation: the base RA should name NVIDIA components and cloud-native API standards, while partner-selected ecosystem tools should be recorded as implementation decisions.

Known Limitations

• This reference architecture is not a substitute for partner lab validation.
• The reference component version and source snapshot is not a universal support statement.
• Physical topology, performance numbers, and support boundaries must be supplied by the partner validation process.
• Benchmark results must be tied to the model, hardware, backend, traffic profile, and software versions used in the test.
• Hardware guidance must be reconciled with the NVIDIA-approved, access-controlled hardware reference for the selected profile.
• The Kubernetes Gateway API Inference Extension provides an interoperability path; it does not replace Dynamo serving design decisions, backend disposition, or KV-aware routing validation.
• Non-NVIDIA implementation software should not become a base RA dependency unless the partner records it as a local decision outside the reference stack.

Next Steps

1. Select the provider profile and link its Platform Requirements.
2. Record the delivered interfaces and capabilities consumed by the architecture.
3. Define the inference components, relationships, workload topology, and request and data flows.
4. Link operational and validation decisions to the owning Platform Requirements and runbooks.