Interactive Walkthrough

A guided tour of NeMo Evaluator’s main features, explained through real config files. Each section builds on the previous one — by the end you will understand services, solvers, scoring, sandboxes, suites, cluster deployment, and sharding.

Prerequisites

$
pip install -e ".[scoring]"
$
export NVIDIA_API_KEY="your-api-key-here"

All examples below reference configs in examples/configs/. You can run any of them directly:

$
nel eval run examples/configs/01_gsm8k_chat.yaml

1. The Simplest Evaluation

Config: 01_gsm8k_chat.yaml

1
services:
2
  nemotron:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/chat/completions
5
    protocol: chat_completions
6
    model: nvidia/nemotron-3-super-120b-a12b
7
    api_key: ${NVIDIA_API_KEY}
8
9
benchmarks:
10
  - name: gsm8k
11
    max_problems: 5
12
    solver:
13
      type: simple
14
      service: nemotron

Every config has two required sections:

services declares the model endpoints you want to evaluate. Each service has:

• type — api means an external endpoint you connect to (not one NEL deploys).
• url — The full endpoint URL, including the path (/v1/chat/completions).
• protocol — How to format requests (chat_completions, completions, or responses). This is separate from the URL because experimental endpoints may use a different path than the standard one.
• model — The model identifier sent in the API request.
• api_key — Supports ${ENV_VAR} expansion so you never hardcode secrets.

benchmarks is a list of evaluations to run. Each benchmark has:

• name — A built-in benchmark name (gsm8k), a URI (skills://gpqa, lm-eval://mmlu), or a remote server (gym://host:port).
• solver — Tells NEL how to interact with the model. type: simple sends the prompt and parses the response. service: nemotron points to the service declared above.

The GSM8K benchmark has a built-in scorer that extracts the last number from the model’s response and compares it to the ground truth. No extra scoring config is needed.

$
nel eval run examples/configs/01_gsm8k_chat.yaml

2. Completions Protocol (Base Models)

Config: 02_mmlu_completions.yaml

1
services:
2
  nemotron:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/completions
5
    protocol: completions
6
    model: nvidia/nemotron-3-super-120b-a12b
7
    api_key: ${NVIDIA_API_KEY}
8
    generation:
9
      max_tokens: 64
10
11
benchmarks:
12
  - name: mmlu
13
    max_problems: 5
14
    solver:
15
      type: simple
16
      service: nemotron
17
    scoring:
18
      metrics:
19
        - type: scorer
20
          name: mcq_extract

Two new concepts here:

generation controls LLM generation parameters — max_tokens, temperature, top_p, etc. For multiple-choice, we only need a short answer, so max_tokens: 64 keeps things fast.

scoring.metrics overrides the benchmark’s default scorer. Here we use mcq_extract which extracts a letter (A-D) from the response. Each metric has a type and name — the scorer type runs a built-in scoring function.

Notice the URL and protocol changed — /v1/completions with protocol: completions. The simple solver automatically adapts: for chat_completions it sends a messages array, for completions it sends a raw text prompt.

3. External Benchmark Sources

Config: 03_mmlu_lmeval.yaml and 04_mmlu_pro_skills.yaml

NEL isn’t limited to its built-in benchmarks. URI schemes plug in external benchmark libraries:

1
benchmarks:
2
  # lm-evaluation-harness (pip install nemo-evaluator[lm-eval])
3
  - name: lm-eval://mmlu_generative
4
    solver:
5
      type: simple
6
      service: nemotron
7
8
  # NeMo Skills (pip install nemo-evaluator[skills])
9
  - name: skills://mmlu-pro
10
    solver:
11
      type: simple
12
      service: nemotron

The service and solver config is identical regardless of the benchmark source — you are always in control of which model gets called and how.

4. LLM-as-Judge (Two Services)

Config: 05_simpleqa_judge.yaml

1
services:
2
  solver:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/chat/completions
5
    protocol: chat_completions
6
    model: nvidia/nemotron-3-super-120b-a12b
7
    api_key: ${NVIDIA_API_KEY}
8
9
  judge:
10
    type: api
11
    url: https://integrate.api.nvidia.com/v1/chat/completions
12
    protocol: chat_completions
13
    model: nvidia/nemotron-3-super-120b-a12b
14
    api_key: ${NVIDIA_API_KEY}
15
16
benchmarks:
17
  - name: simpleqa
18
    max_problems: 5
19
    solver:
20
      type: simple
21
      service: solver
22
    scoring:
23
      include_defaults: false
24
      metrics:
25
        - type: judge
26
          name: correctness
27
          service: judge

This config declares two separate services: one for solving and one for judging. In production you would use a stronger model as the judge.

The scoring section introduces:

• include_defaults: false — Disables the benchmark’s built-in scorer. Only the explicitly listed metrics run.
• type: judge — An LLM-as-judge metric. The service field points to the judge model, and NEL automatically constructs a judge prompt from the response and expected answer.

5. Code Execution in Docker Sandboxes

Config: 06_humaneval_docker.yaml

1
benchmarks:
2
  - name: humaneval
3
    max_problems: 5
4
    solver:
5
      type: simple
6
      service: nemotron
7
    sandbox:
8
      type: docker
9
      image: python:3.12-slim
10
      timeout: 60.0

HumanEval generates Python code. To score it, NEL executes the generated code inside a Docker container:

• sandbox.type: docker — Each problem gets its own isolated container.
• sandbox.image — The Docker image to use. HumanEval needs Python, so python:3.12-slim.
• sandbox.timeout — Kill the container after 60 seconds (prevents infinite loops).

The sandbox is infrastructure-only — it knows nothing about scoring. The HumanEval benchmark’s scorer extracts code from the model response, writes it into the container alongside test cases, and checks the exit code.

Other sandbox options: memory, cpus, network (bridge/host/none), concurrency (max parallel containers).

6. Agentic Evaluation with Harbor

Config: 07_swebench_harbor.yaml

1
services:
2
  nemotron:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/chat/completions
5
    protocol: chat_completions
6
    model: nvidia/nemotron-3-super-120b-a12b
7
    api_key: ${NVIDIA_API_KEY}
8
    interceptors:
9
      - name: turn_counter
10
        config:
11
          max_turns: 100
12
13
benchmarks:
14
  - name: harbor://swebench-verified@1.0
15
    max_problems: 3
16
    solver:
17
      type: harbor
18
      service: nemotron
19
      agent: openhands-sdk
20
      agent_kwargs:
21
        max_iterations: 100
22
    sandbox:
23
      type: ecs_fargate
24
      cluster: harbor-cluster
25
      subnets: [subnet-abc123]
26
      # ... (ECS configuration)

This is a significant step up. Three new concepts:

interceptors are LiteLLM proxy callbacks that intercept every request between the agent and the model. Here, turn_counter limits the agent to 100 turns. Other interceptors can log requests, modify payloads, or cache responses. Each service gets its own set of interceptors.

solver.type: harbor runs a full autonomous agent (OpenHands, Terminus-2, etc.) instead of a single model call. The agent field selects which agent framework, and agent_kwargs passes configuration to it.

sandbox.type: ecs_fargate runs each problem in an AWS ECS Fargate container — no local Docker required. Each SWE-bench problem gets its own container with the correct codebase pre-installed. An SSH sidecar enables bidirectional tunnels between the orchestrator and the container.

7. Gym Integration

Config: 09_gym_runtime.yaml

1
services:
2
  nemotron:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/chat/completions
5
    protocol: chat_completions
6
    model: nvidia/nemotron-3-super-120b-a12b
7
    api_key: ${NVIDIA_API_KEY}
8
9
  swe_gym:
10
    type: gym
11
    url: http://localhost:11413
12
13
benchmarks:
14
  - name: swebench-multilingual
15
    solver:
16
      type: gym_delegation
17
      service: nemotron
18
      gym_service: swe_gym
19
      gym_agent: openhands
20
      trust_reward: false
21
    sandbox:
22
      type: docker
23
      timeout: 1800.0

Here we see a type: gym service — a connection to a running NeMo Gym server. The gym_delegation solver delegates the agent execution to the Gym server while NEL handles verification independently.

Key fields:

• gym_service — Points to the swe_gym service (the Gym server).
• trust_reward: false — NEL verifies results independently rather than trusting Gym’s reward. Set to true for Gym-trusted scoring.

8. NEL-Driven Tool Calling (ReActSolver)

Configs: 21_tool_calling_gym.yaml, 22_tool_calling_sandbox.yaml, 23_tool_calling_combined.yaml

The gym_delegation and harbor solvers delegate the agentic loop to an external framework (Gym Agent Server, Harbor agent). This is convenient but opaque — NEL only sees the final response, not the individual model calls, tool calls, and conversation turns.

The tool_calling solver changes this: NEL drives the loop itself. It calls the model, parses tool calls, dispatches them, and records every turn — giving full observability.

Gym HTTP Tools

1
services:
2
  model:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/chat/completions
5
    protocol: chat_completions
6
    model: nvidia/llama-3.3-nemotron-super-49b-v1
7
    api_key: ${NVIDIA_API_KEY}
8
  tools:
9
    type: gym
10
    url: http://localhost:8000
11
12
benchmarks:
13
  - name: swebench-multilingual
14
    solver:
15
      type: tool_calling
16
      service: model
17
      resource_service: tools
18
      max_turns: 50

Here, resource_service: tools points to a Gym Resource Server. NEL discovers available tools via /openapi.json and dispatches tool calls via POST /{tool_name}. Every model call, tool result, and conversation turn is captured in the ATIF trajectory.

Sandbox Tools (NEL as the Agent)

1
benchmarks:
2
  - name: swebench-verified
3
    solver:
4
      type: tool_calling
5
      service: model
6
      sandbox_tools: true
7
      max_turns: 100
8
      response_mode: sandbox_artifact
9
      system_prompt: |
10
        You are a software engineer fixing a bug in a Python repository.
11
        You have these tools: bash, str_replace_editor, file_read, file_write.
12
        Use bash to explore the repo and run tests. Use str_replace_editor to
13
        make targeted edits. When done, the sandbox state is captured automatically.
14
    sandbox:
15
      type: docker
16
      timeout: 1800.0

With sandbox_tools: true, NEL provides a canonical set of tools that execute inside the Docker sandbox:

File operations use sandbox.upload() / sandbox.download() internally — no content is ever embedded in shell commands.

Key fields:

• sandbox_tools: true — enables the four sandbox tools listed above. Requires a sandbox config.
• response_mode: sandbox_artifact — the “answer” is the sandbox state (e.g. a git patch), not the last model message. Use for SWE-bench.
• max_turns — maximum model call rounds before stopping. The solver sets error: "max_turns_exhausted" if hit.
• tool_timeout — per-tool-call timeout in seconds (default: 180).
• max_output_chars — truncation limit for tool output fed back to the model (default: 16384). Large outputs are truncated with head+tail preservation.

Combined (HTTP + Sandbox)

1
benchmarks:
2
  - name: research-bench
3
    solver:
4
      type: tool_calling
5
      service: model
6
      resource_service: tools       # web_search, find_in_page from Gym
7
      sandbox_tools: true            # bash, file_read, file_write
8
      max_turns: 30
9
    sandbox:
10
      type: docker

Both tool sources can be combined: the model sees tools from the Gym Resource Server and sandbox tools. Tool calls are routed to the correct backend by name.

Observability Gains

9. Vision-Language Models

Config: 13_vlmevalkit_mmbench.yaml

1
services:
2
  vila:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/chat/completions
5
    protocol: chat_completions
6
    model: nvidia/vila-3b
7
    api_key: ${NVIDIA_API_KEY}
8
9
benchmarks:
10
  - name: vlmevalkit://MMBench_DEV_EN
11
    max_problems: 5
12
    solver:
13
      type: simple
14
      service: vila
15
      image_detail: high

The simple solver handles VLM benchmarks automatically when the dataset includes images. The image_detail field (auto, low, high) controls the resolution of images sent to the model. VLMEvalKit datasets provide image data alongside text prompts.

10. Multi-Benchmark Suites

Config: 17_suite_release.yaml

1
services:
2
  nemotron:
3
    type: api
4
    url: https://integrate.api.nvidia.com/v1/chat/completions
5
    protocol: chat_completions
6
    model: nvidia/nemotron-3-super-120b-a12b
7
    api_key: ${NVIDIA_API_KEY}
8
9
sandboxes:
10
  humaneval_code:
11
    type: docker
12
    image: python:3.12-slim
13
    timeout: 60.0
14
15
benchmarks:
16
  - name: gsm8k
17
    max_problems: 5
18
    solver:
19
      type: simple
20
      service: nemotron
21
22
  - name: mmlu
23
    max_problems: 5
24
    solver:
25
      type: simple
26
      service: nemotron
27
28
  - name: humaneval
29
    max_problems: 5
30
    solver:
31
      type: simple
32
      service: nemotron
33
    sandbox: humaneval_code
34
35
  - name: simpleqa
36
    max_problems: 5
37
    solver:
38
      type: simple
39
      service: nemotron

Two important features here:

sandboxes is a top-level dict of named sandbox configurations. Instead of inlining the sandbox config in every benchmark, you define it once and reference it by name: sandbox: humaneval_code. This avoids duplication when multiple benchmarks share the same sandbox setup.

Multiple benchmarks in a single config run sequentially. Each gets its own output directory. If one fails, the others still run (failure isolation). You can resume a partially completed suite:

$
nel eval run examples/configs/17_suite_release.yaml --resume

11. Auto-Deployed vLLM on SLURM

Config: 15_slurm_gsm8k_vllm.yaml

1
services:
2
  model:
3
    type: vllm
4
    model: nvidia/Llama-3.1-70B-Instruct
5
    protocol: chat_completions
6
    tensor_parallel_size: 4
7
    port: 8000
8
    node_pool: compute
9
10
benchmarks:
11
  - name: gsm8k
12
    max_problems: 50
13
    solver:
14
      type: simple
15
      service: model
16
17
cluster:
18
  type: slurm
19
  walltime: "02:00:00"
20
  node_pools:
21
    compute:
22
      partition: batch
23
      nodes: 1
24
      ntasks_per_node: 1
25
      gres: "gpu:4"

This is where NEL becomes a “docker-compose for evals”:

type: vllm — NEL deploys the model server for you. It starts a vLLM process, waits for health, runs the benchmark, then tears it down. Same for sglang and trt_llm.

cluster declares infrastructure. Instead of running locally, this generates an sbatch script and submits it to SLURM.

node_pools defines named groups of resources. Here, compute is 1 node with 4 GPUs. Services reference pools via node_pool: compute.

tensor_parallel_size: 4 tells vLLM to shard the model across 4 GPUs — matching the gres: "gpu:4" in the node pool. NEL validates this consistency.

$
nel eval run examples/configs/15_slurm_gsm8k_vllm.yaml

This generates eval.sbatch, submits it, and monitors progress.

12. Heterogeneous SLURM Jobs

Config: 16_slurm_swebench_harbor.yaml

1
services:
2
  model:
3
    type: vllm
4
    model: nvidia/Llama-3.1-70B-Instruct
5
    protocol: chat_completions
6
    tensor_parallel_size: 4
7
    port: 8000
8
    node_pool: gpu
9
10
sandboxes:
11
  swebench:
12
    type: slurm
13
    node_pool: sandbox
14
    concurrency: 4
15
    slots_per_node: 4
16
    timeout: 1800.0
17
18
benchmarks:
19
  - name: swebench-verified
20
    max_problems: 10
21
    solver:
22
      type: harbor
23
      service: model
24
      agent: openhands
25
    sandbox: swebench
26
27
cluster:
28
  type: slurm
29
  walltime: "04:00:00"
30
  node_pools:
31
    gpu:
32
      partition: batch
33
      nodes: 1
34
      ntasks_per_node: 1
35
      gres: "gpu:4"
36
    sandbox:
37
      partition: batch
38
      nodes: 2

This uses two node pools: gpu (1 node with 4 GPUs for vLLM) and sandbox (2 CPU nodes for agent sandboxes). NEL generates a SLURM heterogeneous job that allocates both pools simultaneously.

The swebench sandbox uses type: slurm — containers run via Pyxis/Enroot on the sandbox nodes. With slots_per_node: 4 across 2 nodes, there are 8 concurrent sandbox slots.

13. Sharded Evaluation

Config: 15a_slurm_gsm8k_vllm_sharded.yaml

1
cluster:
2
  type: slurm
3
  walltime: "02:00:00"
4
  shards: 4
5
  node_pools:
6
    compute:
7
      partition: batch
8
      nodes: 1
9
      ntasks_per_node: 1
10
      gres: "gpu:4"

Adding shards: 4 to the cluster config turns the SLURM job into an array job. NEL generates #SBATCH --array=0-3, and each array task evaluates a disjoint slice of the dataset.

After all tasks complete, merge the results:

$
nel eval merge ./eval_results

This discovers shard_0/, shard_1/, etc., deduplicates any overlapping results, and produces a single merged bundle with recomputed pass@k and confidence intervals.

$
NEL_SHARD_IDX=0 NEL_TOTAL_SHARDS=4 nel eval run config.yaml -o ./results/shard_0
$
NEL_SHARD_IDX=1 NEL_TOTAL_SHARDS=4 nel eval run config.yaml -o ./results/shard_1
$
# ... after all complete:
$
nel eval merge ./results

14. Legacy Container Harnesses

Config: 14_container_nemo_skills.yaml

1
benchmarks:
2
  - name: "container://nvcr.io/nvidia/eval-factory/nemo-skills:26.03#ns_ifeval"
3
    params:                          # legacy params passed to the container
4
      temperature: 1.0
5
      top_p: 0.95
6
      parallelism: 1
7
      limit_samples: 10              # test run on 10 samples for quick verification
8
    solver:
9
      type: container
10
      service: nemotron
11
      adapter_config:                # passed verbatim to harness's adapter pipeline
12
        process_reasoning_traces: true
13
      env_vars: {}
14
      mounts: {}

The container:// scheme runs an legacy eval harness container as an opaque box. The container owns the full eval loop — NEL parses results.yml and eval_factory_metrics.json from the output.

This is the least observable mode (no per-request trajectories), but it supports any legacy harness without modification.

15. MLflow Exporter

Per-sample traces

The mlflow exporter emits scalar metrics plus per-sample MLflow GenAI Traces (AGENT / LLM / TOOL spans) built from the ATIF trajectory.

1
output:
2
  export:
3
    - type: mlflow
4
      tracking_uri: http://localhost:5000
5
      experiment: nemotron-eval
6
      emit_traces: true              # default
7
      emit_traces_max_samples: 200   # cap per benchmark; null = no cap
8
      emit_traces_content_max: 4000  # truncate long strings

Tiers (auto-selected): rich (agentic + tools) → messages (chat) → response (extractive) → meta (no trajectory / no response). For agent-only trajectories (e.g. PinchBench), a leading user span is synthesised from the prompt. The root span carries nel.scoring_details, per-scorer rewards (nel.scorer.<name>), and task metadata (nel.task_id, nel.task_name, nel.category, …). Traces are skipped when MLflow is not installed, emit_traces=false, or the run is reused on re-export.

Artifact upload filtering

exclude_patterns adds user-defined patterns to the always-on EXCLUDED_PATTERNS defaults (*cache*, *.db, *.lock, synthetic, debug.json). Match is on basename, case-insensitive, using fnmatch.fnmatchcase (so *, ?, and [seq] character classes all work). The patterns are applied in all artifact paths the exporter walks: only_required=true (gates which required-glob hits get uploaded), only_required=false (shutil.copytree filter), and copy_logs=true (per-file gate under logs/).

1
output:
2
  export:
3
    - type: mlflow
4
      tracking_uri: http://localhost:5000
5
      copy_artifacts: true
6
      only_required: false             # upload everything except excluded
7
      copy_logs: true
8
      exclude_patterns:
9
        - deliverables                 # drop the gdpval deliverables sub-tree
10
        - "*.parquet"                  # drop large columnar dumps

Config Anatomy Cheat Sheet

Every NeMo Evaluator config follows this structure:

1
# 1. Services — what endpoints you are evaluating
2
services:
3
  <name>:
4
    type: api | vllm | sglang | trt_llm | gym | nat_agent
5
    url: <full endpoint URL>
6
    protocol: chat_completions | completions | responses
7
    model: <model identifier>
8
    # Optional:
9
    api_key: ${ENV_VAR}
10
    generation: { temperature, top_p, max_tokens, seed, stop, frequency_penalty, presence_penalty }
11
    interceptors: [{ name, config }, ...]
12
    proxy_verbose: false
13
    depends_on: [<other service names>]
14
15
# 2. Sandboxes (optional) — reusable sandbox configs
16
sandboxes:
17
  <name>:
18
    type: docker | slurm | ecs_fargate | apptainer | local | none
19
    # type-specific fields...
20
21
# 3. Benchmarks — what to evaluate
22
benchmarks:
23
  - name: <benchmark name or URI>
24
    solver:
25
      type: simple | harbor | tool_calling | gym_delegation | openclaw | container
26
      service: <service name>
27
      # solver-specific fields...
28
    # Optional:
29
    sandbox: <sandbox name or inline config>
30
    scoring:
31
      metrics: [{ type, name, service }, ...]
32
    max_problems: <int>
33
    repeats: <int>
34
    max_concurrent: <int>
35
    timeout: <float>
36
37
# 4. Cluster (optional) — where to run
38
cluster:
39
  type: local | slurm | docker
40
  # cluster-specific fields...
41
42
# 5. Output (optional) — reporting and export
43
output:
44
  dir: ./eval_results
45
  export: [inspect, wandb, mlflow]

What’s Next?