Quickstart

Run a complete evaluation in under 5 minutes.

Step 1: Set your model endpoint

$
export NVIDIA_API_KEY="your-api-key-here"

Step 2: Run an evaluation

$
nel eval run --bench gsm8k \
>
  --model-url https://integrate.api.nvidia.com/v1/chat/completions \
>
  --model-id nvidia/nemotron-3-super-120b-a12b \
>
  --api-key $NVIDIA_API_KEY \
>
  --repeats 2 --max-problems 20

Live progress output:

gsm8k  [12/20]  60.0%  ████████████░░░░░░░░  acc=0.750  1.2k tok/s  ETA 0:32

Step 3: Read the results

After completion, nel prints a summary and writes artifacts:

gsm8k
  problems=20  repeats=2
  pass@1: 0.7500  [0.5500, 0.9000]
  pass@2: 0.8500  [0.6500, 0.9500]
  tokens=12,400  speed=2.1/s
  Output: ./eval_results/
    bundle: eval-20260225T143012Z-gsm8k.json
    results: results.jsonl
    trajectories: trajectories.jsonl
    runtime_stats: runtime_stats.json
    failure_analysis: failure_analysis.json

Artifact reference

Step 4: Compare against a baseline

$
nel eval run --bench gsm8k \
>
  --model-url https://integrate.api.nvidia.com/v1/chat/completions \
>
  --model-id nvidia/nemotron-3-super-120b-a12b \
>
  --api-key $NVIDIA_API_KEY \
>
  --repeats 4 -o ./results/candidate
$
$
nel compare ./results/baseline ./results/candidate --strict

nel compare accepts directories or bundle files. It pairs results at the problem level, runs a McNemar exact test for regression detection, and generates an investigation report.

When scipy is installed (pip install nemo-evaluator[stats]), comparisons include McNemar significance testing and confidence intervals on effect sizes.

See Comparing Evaluation Runs for the full walkthrough.

Step 5: Gate a release across multiple benchmarks

For release qualification, use nel gate with a policy file:

1
# gate_policy.yaml
2
version: 1
3
defaults:
4
  tier: supporting
5
  metric: mean_reward
6
  max_drop: 0.015
7
benchmarks:
8
  mmlu_pro:
9
    tier: critical
10
    max_drop: 0.01
11
  gpqa:
12
    tier: critical
13
    max_drop: 0.01

$
nel gate ./results/baseline ./results/candidate --policy gate_policy.yaml --strict

The gate applies per-benchmark thresholds and returns GO, NO-GO, or INCONCLUSIVE. With --strict, exit codes work directly in CI (0/1/2).

See Implementing Quality Gates for the full walkthrough.

Step 6: Use a config file

For reproducible multi-benchmark evaluations:

1
# eval_config.yaml
2
services:
3
  nemotron:
4
    type: api
5
    url: https://integrate.api.nvidia.com/v1/chat/completions
6
    protocol: chat_completions
7
    model: nvidia/nemotron-3-super-120b-a12b
8
    api_key: ${NVIDIA_API_KEY}
9
10
benchmarks:
11
  - name: gsm8k
12
    repeats: 4
13
    solver:
14
      type: simple
15
      service: nemotron
16
      system_prompt: "Solve step by step. Put your final answer in \\boxed{}."
17
18
  - name: triviaqa
19
    repeats: 1
20
    max_problems: 100
21
    solver:
22
      type: simple
23
      service: nemotron

$
nel eval run eval_config.yaml

Each task gets its own output directory with the full artifact suite.

Step 7: Resume a failed suite

If a benchmark fails mid-suite (e.g., a network error on benchmark 3/5), the remaining benchmarks still execute. Re-run with --resume to retry only the failed ones:

$
nel eval run eval_config.yaml --resume

Completed benchmarks are skipped. Failed benchmarks are retried.

Next Steps

• Comparing Evaluation Runs — Deep dive into comparing runs and investigating regressions
• Implementing Quality Gates — Set up multi-benchmark release gates with policy files
• Write Your Own Benchmark (BYOB) — Write your own benchmark with @benchmark + @scorer
• Gym Integration — Serve benchmarks for Gym training
• Distributed Evaluation — Scale to thousands of problems
• Index — Understand how the system works