Installation Guide#

NeMo Evaluator provides multiple installation paths depending on your needs. Choose the approach that best fits your use case.

Choose Your Installation Path#

Table 3 Installation Path Comparison#
Installation Path	Best For	Key Features
NeMo Evaluator Launcher (Recommended)	Most users who want unified CLI and orchestration across backends	• Unified CLI for 100+ benchmarks • Multi-backend execution (local, Slurm, cloud) • Built-in result export to MLflow, W&B, etc. • Configuration management with examples
NeMo Evaluator Core	Developers building custom evaluation pipelines	• Programmatic Python API • Direct container access • Custom framework integration • Advanced adapter configuration
Container Direct	Users who prefer container-based workflows	• Pre-built NGC evaluation containers • Guaranteed reproducibility • No local installation required • Isolated evaluation environments

Prerequisites#

System Requirements#

Python 3.10 or higher (supports 3.10, 3.11, 3.12, and 3.13)
CUDA-compatible GPU(s) (tested on RTX A6000, A100, H100)
Docker (for container-based workflows)

Recommended Environment#

Python 3.12
CUDA 12.9
Ubuntu 24.04

Installation Methods#

Launcher (Recommended)

Install NeMo Evaluator Launcher for unified CLI and orchestration:

# Create and activate virtual environment
python3 -m venv nemo-eval-env
source nemo-eval-env/bin/activate

# Install launcher with all exporters (recommended)
pip install nemo-evaluator-launcher[all]

Quick verification:

# Verify installation
nemo-evaluator-launcher --version

# Test basic functionality - list available tasks
nemo-evaluator-launcher ls tasks | head -10

Core Library

Install NeMo Evaluator Core for programmatic access:

# Create and activate virtual environment
python3 -m venv nemo-eval-env
source nemo-eval-env/bin/activate

# Install core library with dependencies
pip install nemo-evaluator

# Install evaluation frameworks
pip install nvidia-simple-evals nvidia-lm-eval

Quick verification:

# Verify installation
nemo-evaluator ls | head && echo '✓ CLI available' || exit 1
python3 -c "from nemo_evaluator.api import evaluate; print('✓ Python API available')" || exit 1
echo "✓ NeMo Evaluator Core installed successfully"

NGC Containers

Use pre-built evaluation containers from NVIDIA NGC for guaranteed reproducibility:

# Pull evaluation containers (no local installation needed)
docker pull nvcr.io/nvidia/eval-factory/simple-evals:25.09
docker pull nvcr.io/nvidia/eval-factory/lm-evaluation-harness:25.09
docker pull nvcr.io/nvidia/eval-factory/bigcode-evaluation-harness:25.09

# Run container interactively
docker run --rm -it \
    -v $(pwd)/results:/workspace/results \
    nvcr.io/nvidia/eval-factory/simple-evals:25.09 bash

# Or run evaluation directly
docker run --rm \
    -v $(pwd)/results:/workspace/results \
    -e NGC_API_KEY=nvapi-xxx \
    nvcr.io/nvidia/eval-factory/simple-evals:25.09 \
    nemo-evaluator run_eval \
        --eval_type mmlu_pro \
        --model_url https://integrate.api.nvidia.com/v1/chat/completions \
        --model_id meta/llama-3.1-8b-instruct \
        --api_key_name NGC_API_KEY \
        --output_dir /workspace/results

Quick verification:

# Test container access
docker run --rm nvcr.io/nvidia/eval-factory/simple-evals:25.09 \
    nemo-evaluator ls | head -5
echo " Container access verified"

Clone the Repository#

Clone the NeMo Evaluator repository to get easy access to our ready-to-use examples:

git clone https://github.com/NVIDIA-NeMo/Evaluator.git

Run the example:

cd Evaluator/

export NGC_API_KEY=nvapi-...  # API Key with access to build.nvidia.com
nemo-evaluator-launcher run \
  --config-dir packages/nemo-evaluator-launcher/examples \
  --config-name local_nvidia_nemotron_nano_9b_v2 \
  --override execution.output_dir=nemotron-eval

Add Evaluation Harnesses to Your Environment#

Build your custom evaluation pipeline by adding evaluation harness packages to your environment of choice:

pip install nemo-evaluator <evaluation-package>

Available PyPI Packages#

Package Name	PyPI URL
nvidia-bfcl	https://pypi.org/project/nvidia-bfcl/
nvidia-bigcode-eval	https://pypi.org/project/nvidia-bigcode-eval/
nvidia-compute-eval	https://pypi.org/project/nvidia-compute-eval/
nvidia-eval-factory-garak	https://pypi.org/project/nvidia-eval-factory-garak/
nvidia-genai-perf-eval	https://pypi.org/project/nvidia-genai-perf-eval/
nvidia-crfm-helm	https://pypi.org/project/nvidia-crfm-helm/
nvidia-hle	https://pypi.org/project/nvidia-hle/
nvidia-ifbench	https://pypi.org/project/nvidia-ifbench/
nvidia-livecodebench	https://pypi.org/project/nvidia-livecodebench/
nvidia-lm-eval	https://pypi.org/project/nvidia-lm-eval/
nvidia-mmath	https://pypi.org/project/nvidia-mmath/
nvidia-mtbench-evaluator	https://pypi.org/project/nvidia-mtbench-evaluator/
nvidia-eval-factory-nemo-skills	https://pypi.org/project/nvidia-eval-factory-nemo-skills/
nvidia-safety-harness	https://pypi.org/project/nvidia-safety-harness/
nvidia-scicode	https://pypi.org/project/nvidia-scicode/
nvidia-simple-evals	https://pypi.org/project/nvidia-simple-evals/
nvidia-tooltalk	https://pypi.org/project/nvidia-tooltalk/
nvidia-vlmeval	https://pypi.org/project/nvidia-vlmeval/

Note

Evaluation harnessess that require complex environments are not available as packages but only as containers.