Skip to navigation

Evo 2 Prerequisites

View as MarkdownOpen in Claude

To get started with the Evo 2 NIM, you must complete the necessary prerequisites.

Models

The following table lists the Evo 2 models available in the NIM.

Model NameParametersContext LengthPublisher
Evo 240 Billion1 MillionArc Institute
Evo 27 Billion1 MillionArc Institute

Hardware

Run the Evo 2 NIM on an NVIDIA GPU that meets the following hardware requirements.

40B Model (Default)

The following table lists the supported GPU configurations for the 40B model.

GPUGPU Memory (GB)Precision# of GPUsDisk Space (GB)CPU RAM (GB)
NVIDIA H100 80GB HBM380Mixed211016
NVIDIA H200141Mixed111016

7B Model

The following table lists the supported GPU configurations for the 7B model.

GPUGPU Memory (GB)Precision# of GPUsDisk Space (GB)CPU RAM (GB)
NVIDIA H100 80GB HBM380Mixed15016
NVIDIA H200141Mixed15016
NVIDIA RTX 6000 Ada Generation48Mixed15016
NVIDIA L40S48Mixed15016

The GPU Memory, CPU RAM, and Disk Space values are in GB; Disk Space is for both the container and the model.

If the precision is mixed, it means the model features layers with FP32, BF16, and FP8 floating-point precision tensors.

Supported CPU architectures: x86_64 / amd64.

Software

Install the following software dependencies and verify that your container runtime can access the GPU.

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Example output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H100 80GB HBM3 On | 00000000:1B:00.0 Off | 0 |
| N/A 33C P0 75W / 700W | 1MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA H100 80GB HBM3 On | 00000000:43:00.0 Off | 0 |
| N/A 36C P0 70W / 700W | 1MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

For more information on enumerating multi-GPU systems, refer to the NVIDIA Container Toolkit GPU enumeration topic.

NGC Account

The Evo 2 NIM pulls its container image and model weights from NVIDIA NGC, so the host needs outbound network access to nvcr.io and an NGC account.

  1. Create an account on NGC

  2. Generate an API key

  3. Log in to Docker with your NGC API key.

    Enter the key as the password when prompted.

docker login nvcr.io --username='$oauthtoken'

Next Steps

When your prerequisites are in place, you are ready to start the NIM. Refer to the Quickstart Guide for Evo 2 NIM.