Introduction

NVIDIA NIMs simplify the deployment and management of inference services across a wide range of AI models. Delivered as pre-packaged, containerized inference servers, NIMs are readily available through the NVIDIA NGC catalog and are designed for rapid integration into enterprise AI infrastructure. Each NVIDIA NIM has a minimum number of GPUs that are needed to run Inference against a specific model. The minimum no. of GPUs that can support a specific number of users/workloads that can run concurrent Inference sessions against NIM, based on how the NIM is optimized (latency v/s throughput). NIMs can be scaled easily by adding more GPUs as workload demands grow.

The overall workload on a NIM service can vary at any point in the day, the challenge however is all the GPUs that are allocated to a particular NIM are consumed by that NIM regardless of the workload on that NIM, or in case of scale, the system GPUs might be scaled for max capacity, however only a percentage of users are consuming that NIM so the GPUs can be scaled down and can be given to another workload.

At times, enterprises might have powerful GPUs that can fractionally accommodate more than one NIM workload. While the GPUs are idle, they cannot be deallocated and allocated to other workloads because these GPUs are statically assigned.

NVIDIA Run:ai is an intelligent workload manager that helps orchestrate AI workloads across a resource pool of GPUs. It can automatically scale up/down an inference service based on workload and can allocate fractional GPUs for various NIMs, Run:ai can also prioritize GPU allocation based on users, workload etc.

Scope

This paper covers how to optimize and scale NVIDIA GPUs to run NVIDIA NIMs using Run:ai on an Enterprise Reference Architecture. The following NIMs and GPUs are covered.

Table 1: Scope of NIMs and GPUs

NVIDIA NIM for LLMs	NVIDIA GPUs
Meta Llama 3.1 8B Instruct	H100 NVL
DeepSeek R1 Distill Llama 8B	H100 NVL

Intended Audience

This Guide is meant to help NVIDIA partners architecting Gen AI-based Large Language Models (LLMs) on Enterprise Infrastructure and sizing for cluster-level deployments. The guide can be used for both new and existing deployments to determine the capacity and scale needed based on the LLM workload. It can also be used by the following Personas as they architect their Enterprise solutions.

Enterprise Architects: Enterprise Architects tasked with designing and defining servers, GPU’s and Networking gear to determine what Infrastructure resources will be needed to support specific workloads around LLMs based on NIMs

MLOps Engineer: MLOps Engineers can use this to define Infrastructure requirements as they talk to Infrastructure/Cloud teams to carve out resources to run Inference Services on LLMs based on NIMs

Platform Engineer: Platform Engineers can use this guide to determine how to design their Container Environment around Kubernetes, and also what kind of resources will be needed by the cluster to support various LLM workloads based on NIMs