NeMo Skills Integration

Use NeMo Skills benchmarks as SkillsEnvironment instances with full per-request observability.

Setup

$
pip install -e ".[skills]"  # or: pip install nemo-skills
$
ns prepare_data gpqa
$
ns prepare_data aime24
$
export NEMO_MODEL_URL="https://integrate.api.nvidia.com/v1/chat/completions"
$
export NEMO_MODEL_ID="nvidia/nemotron-3-super-120b-a12b"
$
export NVIDIA_API_KEY="your-api-key-here"

CLI

$
nel eval run --bench skills://gpqa --repeats 4 --max-problems 100
$
nel eval run --bench skills://aime24 --repeats 8
$
nel eval run --bench skills://mmlu --repeats 1

The skills:// URI scheme resolves to a SkillsEnvironment, which:

1. Loads the dataset via nemo_skills.dataset.utils
2. Auto-prepares missing datasets
3. Selects the correct scoring method from the benchmark’s METRICS_TYPE

How It Works

SkillsEnvironment is a standard EvalEnvironment. The eval loop calls seed(idx) and verify(response, expected) like any other environment. NEL owns the model call, so you get full observability.

Scoring

The environment automatically selects the correct scoring method based on METRICS_TYPE:

Custom Prompt Templates

Override the default prompt for any Skills benchmark:

$
nel eval run --bench skills://gpqa --system-prompt "Think step by step."

Or via Python:

1
from nemo_evaluator.environments.skills import SkillsEnvironment
2
3
env = SkillsEnvironment(
4
    "gpqa",
5
    prompt_template="Answer the following question.\n\n{problem}\n\nAnswer:",
6
)

Available Benchmarks

$
nel list  # shows built-in + skills if installed

Math Reasoning

Knowledge and Reasoning

Code

Instruction Following

Tool Calling

Long Context

Distributed Evaluation

Combine with sharding for large benchmarks:

Use SLURM sharding via a config file (see Distributed Evaluation), or use environment variables directly:

$
NEL_SHARD_IDX=0 NEL_TOTAL_SHARDS=8 nel eval run --bench skills://mmlu --repeats 4

Python API

1
import asyncio
2
from nemo_evaluator.environments.skills import SkillsEnvironment
3
from nemo_evaluator import run_evaluation, ChatSolver, ModelClient
4
5
env = SkillsEnvironment("gpqa")
6
client = ModelClient(base_url="https://api.example.com/v1", model="my-model")
7
solver = ChatSolver(client)
8
9
bundle = asyncio.run(run_evaluation(env, solver, n_repeats=4, max_problems=100))