Prerequisites and Support Matrix for Evo 2 NIM#
Models#
Model Name |
Parameters |
Context Length |
Publisher |
|---|---|---|---|
Evo 2 |
40 Billion |
1 Million |
Arc Institute |
Evo 2 |
7 Billion |
1 Million |
Arc Institute |
Hardware Support#
Run your Evo 2 NIM on any NVIDIA GPU that meets following minimum hardware requirements:
40B Model (Default)#
GPU |
GPU Memory (GB) |
Precision |
# of GPUs |
Disk Space (GB) |
CPU RAM (GB) |
|---|---|---|---|---|---|
H100 |
80 |
Mixed |
2 |
110 |
16 |
H200 |
141 |
Mixed |
1 |
110 |
16 |
7B Model#
GPU |
GPU Memory (GB) |
Precision |
# of GPUs |
Disk Space (GB) |
CPU RAM (GB) |
|---|---|---|---|---|---|
H100 |
80 |
Mixed |
1 |
50 |
16 |
H200 |
141 |
Mixed |
1 |
50 |
16 |
RTX 6000 Ada |
48 |
Mixed |
1 |
50 |
16 |
L40S |
48 |
Mixed |
1 |
50 |
16 |
The GPU Memory, CPU RAM, and Disk Space values are in GB; Disk Space is for both the container and the model.
If the precision is mixed, it means the model features layers with FP32, BF16, and FP8 floating-point precision tensors.
Supported CPU architectures: x86_64 / amd64.
Software Prerequisites#
Begin with Linux-based Docker-supported operating system
Install Docker - minimum version: 23.0.1
Install NVIDIA Drivers - minimum version: 550.90.07
Install the NVIDIA Container Toolkit - minimum version: 1.13.5
Verify your container runtime supports NVIDIA GPUs by running
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
Example output:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H100 80GB HBM3 On | 00000000:1B:00.0 Off | 0 |
| N/A 33C P0 75W / 700W | 1MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA H100 80GB HBM3 On | 00000000:43:00.0 Off | 0 |
| N/A 36C P0 70W / 700W | 1MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Note
For more information on enumerating multi-GPU systems, see the NVIDIA Container Toolkit’s GPU Enumeration Docs
NGC (NVIDIA GPU Cloud) Account#
Log in to Docker with your NGC API (enter the key as password when prompted.)
docker login nvcr.io --username='$oauthtoken'