Local Deployment | NVIDIA NeMo Evaluator

Run evaluations directly on your workstation.

Single benchmark (CLI)

$ export NVIDIA_API_KEY="your-api-key-here"
$ 
$ nel eval run --bench gsm8k --repeats 4 \
>     --model-url https://integrate.api.nvidia.com/v1/chat/completions \
>     --model-id nvidia/nemotron-3-super-120b-a12b \
>     --api-key $NVIDIA_API_KEY \
>     --output-dir ./results

Config file

1 # eval.yaml
2 services:
3   nemotron:
4     type: api
5     url: https://integrate.api.nvidia.com/v1/chat/completions
6     protocol: chat_completions
7     model: nvidia/nemotron-3-super-120b-a12b
8     api_key: ${NVIDIA_API_KEY}
9 
10 benchmarks:
11   - name: gsm8k
12     repeats: 4
13     solver:
14       type: simple
15       service: nemotron
16       system_prompt: "Solve step by step. Put your final answer in \\boxed{}."
17 
18   - name: triviaqa
19     repeats: 1
20     max_problems: 200
21     solver:
22       type: simple
23       service: nemotron

$ nel eval run eval.yaml

Multi-model evaluation

Use named services to evaluate multiple models in the same config:

1 services:
2   solver:
3     type: api
4     url: https://integrate.api.nvidia.com/v1/chat/completions
5     protocol: chat_completions
6     model: nvidia/nemotron-3-super-120b-a12b
7     api_key: ${NVIDIA_API_KEY}
8 
9   judge:
10     type: api
11     url: https://integrate.api.nvidia.com/v1/chat/completions
12     protocol: chat_completions
13     model: nvidia/nemotron-3-super-120b-a12b
14     api_key: ${NVIDIA_API_KEY}
15 
16 benchmarks:
17   - name: simpleqa
18     solver:
19       type: simple
20       service: solver
21     scoring:
22       metrics:
23         - type: judge
24           name: correctness
25           service: judge

Resume a partially completed suite

If a benchmark fails mid-suite, the remaining benchmarks still execute. Re-run with --resume to retry only the failed ones:

$ nel eval run eval.yaml --resume

Completed benchmarks are skipped automatically. Without --resume, all benchmarks are re-run from scratch.

Serve for Gym

$ nel serve -b gsm8k --gym-compat --port 9090

Validate a benchmark

Quick sanity check (5 samples by default, prints pass/fail per sample):

$ export NVIDIA_API_KEY="your-api-key-here"
$ 
$ nel validate -b gsm8k \
>   --model-url https://integrate.api.nvidia.com/v1/chat/completions \
>   --model-id nvidia/nemotron-3-super-120b-a12b \
>   --api-key $NVIDIA_API_KEY \
>   --samples 5

Environment variables

Variable	Purpose	Example
`NVIDIA_API_KEY`	API key for NVIDIA endpoints	`nvapi-...`
`OPENAI_API_KEY`	API key for OpenAI endpoints	`sk-...`
`NEL_SHARD_IDX`	Shard index for distributed eval	`0`
`NEL_TOTAL_SHARDS`	Total shards for distributed eval	`8`