NeMo Skills Integration

Use NeMo Skills benchmarks as SkillsEnvironment instances with full per-request observability.

Setup

pip install -e ".[skills]"  # or: pip install nemo-skills
ns prepare_data gpqa
ns prepare_data aime24
export NEMO_MODEL_URL="https://integrate.api.nvidia.com/v1/chat/completions"
export NEMO_MODEL_ID="nvidia/nemotron-3-super-120b-a12b"
export NVIDIA_API_KEY="your-api-key-here"

CLI

nel eval run --bench skills://gpqa --repeats 4 --max-problems 100
nel eval run --bench skills://aime24 --repeats 8
nel eval run --bench skills://mmlu --repeats 1

The skills:// URI scheme resolves to a SkillsEnvironment, which:

1. Loads the dataset via nemo_skills.dataset.utils
2. Auto-prepares missing datasets
3. Selects the correct scoring method from the benchmark’s METRICS_TYPE

How It Works

SkillsEnvironment is a standard EvalEnvironment. The eval loop calls seed(idx) and verify(response, expected) like any other environment. NEL owns the model call, so you get full observability.

Scoring

The environment automatically selects the correct scoring method based on METRICS_TYPE:

Custom Prompt Templates

Override the default prompt for any Skills benchmark:

nel eval run --bench skills://gpqa --system-prompt "Think step by step."

Or via Python:

from nemo_evaluator.environments.skills import SkillsEnvironment
env = SkillsEnvironment(
    "gpqa",
    prompt_template="Answer the following question.\n\n{problem}\n\nAnswer:",
)

Available Benchmarks

nel list  # shows built-in + skills if installed

Math Reasoning

Knowledge and Reasoning

Code

Instruction Following

Tool Calling

Long Context

Distributed Evaluation

Combine with sharding for large benchmarks:

Use SLURM sharding via a config file (see Distributed Evaluation), or use environment variables directly:

NEL_SHARD_IDX=0 NEL_TOTAL_SHARDS=8 nel eval run --bench skills://mmlu --repeats 4

Python API

import asyncio
from nemo_evaluator.environments.skills import SkillsEnvironment
from nemo_evaluator import run_evaluation, ChatSolver, ModelClient
env = SkillsEnvironment("gpqa")
client = ModelClient(base_url="https://api.example.com/v1", model="my-model")
solver = ChatSolver(client)
bundle = asyncio.run(run_evaluation(env, solver, n_repeats=4, max_problems=100))