Evo 2 Prerequisites
To get started with the Evo 2 NIM, you must complete the necessary prerequisites.
Models
The following table lists the Evo 2 models available in the NIM.
Hardware
Run the Evo 2 NIM on an NVIDIA GPU that meets the following hardware requirements.
40B Model (Default)
The following table lists the supported GPU configurations for the 40B model.
7B Model
The following table lists the supported GPU configurations for the 7B model.
The GPU Memory, CPU RAM, and Disk Space values are in GB; Disk Space is for both the container and the model.
If the precision is mixed, it means the model features layers with FP32, BF16, and FP8 floating-point precision tensors.
Supported CPU architectures: x86_64 / amd64.
Software
Install the following software dependencies and verify that your container runtime can access the GPU.
- Begin with Linux-based Docker-supported operating system
- Install Docker - minimum version: 23.0.1
- Install NVIDIA Drivers - minimum version: 550.90.07
- Install the NVIDIA Container Toolkit - minimum version: 1.13.5
- Verify your container runtime supports NVIDIA GPUs by running
Example output:
For more information on enumerating multi-GPU systems, refer to the NVIDIA Container Toolkit GPU enumeration topic.
NGC Account
The Evo 2 NIM pulls its container image and model weights from NVIDIA NGC, so the host needs outbound network access to nvcr.io and an NGC account.
-
Log in to Docker with your NGC API key.
Enter the key as the password when prompted.
Next Steps
When your prerequisites are in place, you are ready to start the NIM. Refer to the Quickstart Guide for Evo 2 NIM.