Interactive Walkthrough
A guided tour of NeMo Evaluator’s main features, explained through real config files. Each section builds on the previous one — by the end you will understand services, solvers, scoring, sandboxes, suites, cluster deployment, and sharding.
Prerequisites
All examples below reference configs in examples/configs/. You can run any of them directly:
1. The Simplest Evaluation
Config: 01_gsm8k_chat.yaml
Every config has two required sections:
services declares the model endpoints you want to evaluate. Each service has:
type—apimeans an external endpoint you connect to (not one NEL deploys).url— The full endpoint URL, including the path (/v1/chat/completions).protocol— How to format requests (chat_completions,completions, orresponses). This is separate from the URL because experimental endpoints may use a different path than the standard one.model— The model identifier sent in the API request.api_key— Supports${ENV_VAR}expansion so you never hardcode secrets.
benchmarks is a list of evaluations to run. Each benchmark has:
name— A built-in benchmark name (gsm8k), a URI (skills://gpqa,lm-eval://mmlu), or a remote server (gym://host:port).solver— Tells NEL how to interact with the model.type: simplesends the prompt and parses the response.service: nemotronpoints to the service declared above.
The GSM8K benchmark has a built-in scorer that extracts the last number from the model’s response and compares it to the ground truth. No extra scoring config is needed.
What happens under the hood
- NEL loads the GSM8K dataset (1,319 math problems).
max_problems: 5limits the run to the first 5.- For each problem, the
simplesolver sends the prompt to thenemotronservice via/v1/chat/completions. - The benchmark’s built-in scorer extracts the answer and compares to ground truth.
- Results are written to
./eval_results/with pass@k scores, trajectories, and runtime stats.
2. Completions Protocol (Base Models)
Config: 02_mmlu_completions.yaml
Two new concepts here:
generation controls LLM generation parameters — max_tokens, temperature, top_p, etc. For multiple-choice, we only need a short answer, so max_tokens: 64 keeps things fast.
scoring.metrics overrides the benchmark’s default scorer. Here we use mcq_extract which extracts a letter (A-D) from the response. Each metric has a type and name — the scorer type runs a built-in scoring function.
Notice the URL and protocol changed — /v1/completions with protocol: completions. The simple solver automatically adapts: for chat_completions it sends a messages array, for completions it sends a raw text prompt.
3. External Benchmark Sources
Config: 03_mmlu_lmeval.yaml and 04_mmlu_pro_skills.yaml
NEL isn’t limited to its built-in benchmarks. URI schemes plug in external benchmark libraries:
The service and solver config is identical regardless of the benchmark source — you are always in control of which model gets called and how.
4. LLM-as-Judge (Two Services)
Config: 05_simpleqa_judge.yaml
This config declares two separate services: one for solving and one for judging. In production you would use a stronger model as the judge.
The scoring section introduces:
include_defaults: false— Disables the benchmark’s built-in scorer. Only the explicitly listed metrics run.type: judge— An LLM-as-judge metric. Theservicefield points to the judge model, and NEL automatically constructs a judge prompt from the response and expected answer.
Self-judge prevention
NEL validates that the judge service is different from the solver’s service. If you accidentally point both to the same service, NEL raises an error — unless you explicitly set allow_self_judge: true on the metric.
5. Code Execution in Docker Sandboxes
Config: 06_humaneval_docker.yaml
HumanEval generates Python code. To score it, NEL executes the generated code inside a Docker container:
sandbox.type: docker— Each problem gets its own isolated container.sandbox.image— The Docker image to use. HumanEval needs Python, sopython:3.12-slim.sandbox.timeout— Kill the container after 60 seconds (prevents infinite loops).
The sandbox is infrastructure-only — it knows nothing about scoring. The HumanEval benchmark’s scorer extracts code from the model response, writes it into the container alongside test cases, and checks the exit code.
Other sandbox options: memory, cpus, network (bridge/host/none), concurrency (max parallel containers).
6. Agentic Evaluation with Harbor
Config: 07_swebench_harbor.yaml
This is a significant step up. Three new concepts:
interceptors are LiteLLM proxy callbacks that intercept every request between the agent and the model. Here, turn_counter limits the agent to 100 turns. Other interceptors can log requests, modify payloads, or cache responses. Each service gets its own set of interceptors.
solver.type: harbor runs a full autonomous agent (OpenHands, Terminus-2, etc.) instead of a single model call. The agent field selects which agent framework, and agent_kwargs passes configuration to it.
sandbox.type: ecs_fargate runs each problem in an AWS ECS Fargate container — no local Docker required. Each SWE-bench problem gets its own container with the correct codebase pre-installed. An SSH sidecar enables bidirectional tunnels between the orchestrator and the container.
Harbor benchmark URIs
harbor://swebench-verified@1.0 downloads task definitions from the Harbor registry via sparse git checkout. Each task includes a Dockerfile, test script, and metadata. Tasks are cached locally after first download.
7. Gym Integration
Config: 09_gym_runtime.yaml
Here we see a type: gym service — a connection to a running NeMo Gym server. The gym_delegation solver delegates the agent execution to the Gym server while NEL handles verification independently.
Key fields:
gym_service— Points to theswe_gymservice (the Gym server).trust_reward: false— NEL verifies results independently rather than trusting Gym’s reward. Set totruefor Gym-trusted scoring.
8. NEL-Driven Tool Calling (ReActSolver)
Configs: 21_tool_calling_gym.yaml, 22_tool_calling_sandbox.yaml, 23_tool_calling_combined.yaml
The gym_delegation and harbor solvers delegate the agentic loop to an external framework (Gym Agent Server, Harbor agent). This is convenient but opaque — NEL only sees the final response, not the individual model calls, tool calls, and conversation turns.
The tool_calling solver changes this: NEL drives the loop itself. It calls the model, parses tool calls, dispatches them, and records every turn — giving full observability.
Gym HTTP Tools
Here, resource_service: tools points to a Gym Resource Server. NEL discovers available tools via /openapi.json and dispatches tool calls via POST /{tool_name}. Every model call, tool result, and conversation turn is captured in the ATIF trajectory.
Sandbox Tools (NEL as the Agent)
With sandbox_tools: true, NEL provides a canonical set of tools that execute inside the Docker sandbox:
File operations use sandbox.upload() / sandbox.download() internally — no content is ever embedded in shell commands.
Key fields:
sandbox_tools: true— enables the four sandbox tools listed above. Requires a sandbox config.response_mode: sandbox_artifact— the “answer” is the sandbox state (e.g. a git patch), not the last model message. Use for SWE-bench.max_turns— maximum model call rounds before stopping. The solver setserror: "max_turns_exhausted"if hit.tool_timeout— per-tool-call timeout in seconds (default: 180).max_output_chars— truncation limit for tool output fed back to the model (default: 16384). Large outputs are truncated with head+tail preservation.
Combined (HTTP + Sandbox)
Both tool sources can be combined: the model sees tools from the Gym Resource Server and sandbox tools. Tool calls are routed to the correct backend by name.
Observability Gains
9. Vision-Language Models
Config: 13_vlmevalkit_mmbench.yaml
The simple solver handles VLM benchmarks automatically when the dataset includes images. The image_detail field (auto, low, high) controls the resolution of images sent to the model. VLMEvalKit datasets provide image data alongside text prompts.
10. Multi-Benchmark Suites
Config: 17_suite_release.yaml
Two important features here:
sandboxes is a top-level dict of named sandbox configurations. Instead of inlining the sandbox config in every benchmark, you define it once and reference it by name: sandbox: humaneval_code. This avoids duplication when multiple benchmarks share the same sandbox setup.
Multiple benchmarks in a single config run sequentially. Each gets its own output directory. If one fails, the others still run (failure isolation). You can resume a partially completed suite:
11. Auto-Deployed vLLM on SLURM
Config: 15_slurm_gsm8k_vllm.yaml
This is where NEL becomes a “docker-compose for evals”:
type: vllm — NEL deploys the model server for you. It starts a vLLM process, waits for health, runs the benchmark, then tears it down. Same for sglang and trt_llm.
cluster declares infrastructure. Instead of running locally, this generates an sbatch script and submits it to SLURM.
node_pools defines named groups of resources. Here, compute is 1 node with 4 GPUs. Services reference pools via node_pool: compute.
tensor_parallel_size: 4 tells vLLM to shard the model across 4 GPUs — matching the gres: "gpu:4" in the node pool. NEL validates this consistency.
This generates eval.sbatch, submits it, and monitors progress.
12. Heterogeneous SLURM Jobs
Config: 16_slurm_swebench_harbor.yaml
This uses two node pools: gpu (1 node with 4 GPUs for vLLM) and sandbox (2 CPU nodes for agent sandboxes). NEL generates a SLURM heterogeneous job that allocates both pools simultaneously.
The swebench sandbox uses type: slurm — containers run via Pyxis/Enroot on the sandbox nodes. With slots_per_node: 4 across 2 nodes, there are 8 concurrent sandbox slots.
13. Sharded Evaluation
Config: 15a_slurm_gsm8k_vllm_sharded.yaml
Adding shards: 4 to the cluster config turns the SLURM job into an array job. NEL generates #SBATCH --array=0-3, and each array task evaluates a disjoint slice of the dataset.
After all tasks complete, merge the results:
This discovers shard_0/, shard_1/, etc., deduplicates any overlapping results, and produces a single merged bundle with recomputed pass@k and confidence intervals.
Sharding without SLURM
You can shard on any infrastructure using environment variables:
14. Legacy Container Harnesses
Config: 14_container_nemo_skills.yaml
The container:// scheme runs an legacy eval harness container as an opaque box. The container owns the full eval loop — NEL parses results.yml and eval_factory_metrics.json from the output.
This is the least observable mode (no per-request trajectories), but it supports any legacy harness without modification.
15. MLflow Exporter
Per-sample traces
The mlflow exporter emits scalar metrics plus per-sample MLflow GenAI Traces (AGENT / LLM / TOOL spans) built from the ATIF trajectory.
Tiers (auto-selected): rich (agentic + tools) → messages (chat) → response (extractive) → meta (no trajectory / no response). For agent-only trajectories (e.g. PinchBench), a leading user span is synthesised from the prompt. The root span carries nel.scoring_details, per-scorer rewards (nel.scorer.<name>), and task metadata (nel.task_id, nel.task_name, nel.category, …). Traces are skipped when MLflow is not installed, emit_traces=false, or the run is reused on re-export.
Artifact upload filtering
exclude_patterns adds user-defined patterns to the always-on EXCLUDED_PATTERNS defaults (*cache*, *.db, *.lock, synthetic, debug.json). Match is on basename, case-insensitive, using fnmatch.fnmatchcase (so *, ?, and [seq] character classes all work). The patterns are applied in all artifact paths the exporter walks: only_required=true (gates which required-glob hits get uploaded), only_required=false (shutil.copytree filter), and copy_logs=true (per-file gate under logs/).
Config Anatomy Cheat Sheet
Every NeMo Evaluator config follows this structure: