Prerequisites and Support Matrix for Evo 2 NIM#

Models#

Model Name	Parameters	Context Length	Publisher
Evo 2	40 Billion	1 Million	Arc Institute

Hardware Support#

Run your Evo 2 NIM on any NVIDIA GPU that meets following minimum hardware requirements:

GPU	GPU Memory (GB)	Precision	# of GPUs	Disk Space (GB)	CPU RAM (GB)
H100	80	Mixed	2	100	16
H200	144	Mixed	1	100	16

The GPU Memory, CPU RAM, and Disk Space values are in GB; Disk Space is for both the container and the model.

If the precision is mixed, it means the model features layers with FP32, BF16, and FP8 floating-point precision tensors.

Supported CPU architectures: x86_64 / amd64.

Software Prerequisites#

Begin with Linux-based Docker-supported operating system
Install Docker - minimum version: 23.0.1
Install NVIDIA Drivers - minimum version: 535
Install the NVIDIA Container Toolkit - minimum version: 1.13.5
Verify your container runtime supports NVIDIA GPUs by running

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Example output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.78.01    Driver Version: 525.78.01    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 41%   30C    P8     1W / 260W |   2244MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Note

For more information on enumerating multi-GPU systems, see the NVIDIA Container Toolkit’s GPU Enumeration Docs

NGC (NVIDIA GPU Cloud) Account#

Create an account on NGC
Generate an API Key
Log in to Docker with your NGC API (enter the key as password when prompted.)

docker login nvcr.io --username='$oauthtoken'