Run Evaluations

Follow step-by-step guides for different evaluation scenarios and methodologies in NeMo Evaluation.

Before You Start

Ensure you have:

1. Completed the initial getting started guides for Installation Guide and Quickstart.

2. Chosen a Model Deployment option:

   • Launcher-Orchestrated Deployment (recommended)

   • Bring-Your-Own-Endpoint

3. Reviewed the evaluation parameters available for optimization.

Environment Requirements

# Core evaluation framework (pre-installed in NeMo container)
pip install nvidia-lm-eval==25.7.1

# Optional harnesses (install as needed)
pip install nvidia-simple-evals>=25.6      # Baseline/simple evaluations
pip install nvidia-bigcode-eval>=25.6      # Advanced code evaluation  
pip install nvidia-safety-harness>=25.6    # Safety evaluation
pip install nvidia-bfcl>=25.6              # Function calling
pip install nvidia-eval-factory-garak>=25.6  # Security scanning

Authentication Requirements

Some evaluations require additional authentication:

# Hugging Face token for gated datasets
export HF_TOKEN="your_hf_token"

# NVIDIA Build API key for judge models (safety evaluation)
export JUDGE_API_KEY="your_nvidia_api_key"

Evaluations

Select an evaluation type to measure capabilities such as text generation, log-probability scoring, code generation, safety and security, and function calling.

Text Generation

Measure model performance through natural language generation for academic benchmarks, reasoning tasks, and general knowledge assessment.

Text Generation Evaluation

Log-Probability

Assess model confidence and uncertainty using log-probabilities for multiple-choice scenarios without text generation.

Log-Probability Evaluation

Code Generation

Measure programming capabilities through code generation, completion, and algorithmic problem solving.

Code Generation Evaluation

Safety & Security

Test AI safety, alignment, and security vulnerabilities using specialized safety harnesses and probing techniques.

Safety and Security Evaluation

Function Calling

Assess tool use capabilities, API calling accuracy, and structured output generation for agent-like behaviors.

Function Calling Evaluation

Selection Guide

Use this section to choose recommended evaluations by model type or by use case.

By Model Type

Model Type	Recommended Evaluations
Base Models (Pre-trained)	Log-Probability Evaluation - No instruction following required Text Generation Evaluation - With academic prompting Avoid chat-specific evaluations
Instruction-Tuned Models	Text Generation Evaluation - Instruction following tasks Code Generation Evaluation - Programming tasks and algorithmic problem solving Safety and Security Evaluation - Alignment testing and vulnerability scanning Function Calling Evaluation - Tool use scenarios and API integration
Chat Models	All evaluation types with appropriate chat formatting Conversational benchmarks and multi-turn evaluations

By Use Case

Use Case	Recommended Evaluations
Academic Research	Text Generation Evaluation for MMLU, reasoning benchmarks Log-Probability Evaluation for baseline comparisons Specialized domains for research-specific metrics (documentation coming soon)
Production Deployment	Safety and Security Evaluation for alignment validation and vulnerability testing Function Calling Evaluation for agent capabilities and tool use Code Generation Evaluation for programming assistants and code completion
Model Development	Text Generation Evaluation for general capability assessment Multiple evaluation types for comprehensive analysis Custom benchmarks for specific improvements