NeMo Evaluator: Deployment Patterns

Architecture

Pattern 1: Direct Evaluation (Local Environment)

Who: Research / training owner running benchmarks against a model endpoint.

What: Evaluator owns the full loop. Seeds problems from a local EvalEnvironment, calls the model, verifies, produces all artifacts.

$
# Quick validation
$
nel validate -b gsm8k --samples 10
$
$
# Full run with n-repeats
$
nel eval run --bench gsm8k --repeats 4 --max-problems 100 -o ./results
$
$
# From YAML config
$
nel eval run eval_config.yaml

1
# eval_config.yaml
2
services:
3
  model:
4
    type: api
5
    url: https://integrate.api.nvidia.com/v1/chat/completions
6
    protocol: chat_completions
7
    model: nvidia/nemotron-3-super-120b-a12b
8
    api_key: ${NVIDIA_API_KEY}
9
10
benchmarks:
11
  - name: gsm8k
12
    repeats: 4
13
    max_problems: 100
14
    solver:
15
      type: simple
16
      service: model

Multi-benchmark configs support --resume for checkpoint-based recovery:

$
nel eval run eval_config.yaml --resume

Artifacts produced: All. Full trajectory with request/response bindings, token counts, latency breakdown per phase, scoring details, failure categorization.

Environments: Any registered EvalEnvironment subclass — GSM8K, TriviaQA, or any BYOB benchmark.

Pattern 2: Serve for Gym Training

Who: Environment developer making an Evaluator benchmark available to Gym’s RL pipeline.

What: An EvalEnvironment is exposed as an HTTP service speaking Gym’s /seed_session + /verify protocol. Gym agents consume it during training. Evaluator independently evaluates the same benchmark with full observability.

$
# Serve with evaluator protocol (for nel eval run --adapter)
$
nel serve -b gsm8k --port 9090
$
$
# Serve with gym-compatible protocol (accepts NeMoGymResponse in /verify)
$
nel serve -b gsm8k --port 9090 --gym-compat
$
$
# Also export JSONL for ng_collect_rollouts
$
nel serve -b gsm8k --port 9090 --gym-compat --export-data /data

Dual-consumer: Same service, two readers. Gym reads reward. Evaluator reads everything.

Pattern 3: Consume Remote Environment via Adapter

Who: Evaluation owner who wants Evaluator’s statistical rigor on an environment running elsewhere.

What: GymEnvironment connects to any server speaking seed_session/verify. Evaluator owns the model call in between, giving full trajectory capture even though the environment is remote.

$
# Consume a remote environment
$
nel eval run --bench gym://staging-server:8080 --repeats 4 --max-problems 50
$
$
# Consume another nel serve instance
$
nel eval run --bench gym://127.0.0.1:9090 --repeats 2

Key: Evaluator owns the model call → full trajectory, tokens, latency regardless of where the environment runs.

Pattern 4: VLMEvalKit Benchmarks

Who: Research team evaluating vision-language models against VLMEvalKit’s 100+ benchmarks with Evaluator’s full observability.

What: VLMEvalKitEnvironment wraps VLMEvalKit datasets, handling image loading and scoring (MCQ, VQA, Y/N). The VLMSolver sends images + text to the model.

$
nel eval run --bench vlmevalkit://MMBench_DEV_EN --repeats 1 --max-problems 50

1
from nemo_evaluator.environments.vlmevalkit import VLMEvalKitEnvironment
2
from nemo_evaluator.engine import run_evaluation, ModelClient
3
from nemo_evaluator.solvers import VLMSolver
4
5
env = VLMEvalKitEnvironment("MMBench_DEV_EN")
6
client = ModelClient(base_url="...", model="...", api_key="...")
7
solver = VLMSolver(client)
8
bundle = await run_evaluation(env, solver, n_repeats=1)

Pattern 5: Serve for ng_collect_rollouts

Who: Training team that needs batch rollout collection using Gym’s infrastructure.

What: nel serve exposes any EvalEnvironment as a Gym-compatible HTTP server. Gym agents and ng_collect_rollouts consume it via standard /seed_session + /verify endpoints.

$
# Serve gsm8k as a Gym-compatible resource server
$
nel serve -b gsm8k --gym-compat --port 9090
$
# Gym training nodes connect via: gym://localhost:9090

1
# Or programmatically
2
import uvicorn
3
from nemo_evaluator.environments.registry import get_environment
4
from nemo_evaluator.serving.app import generate_app
5
6
env = get_environment("gsm8k")
7
app = generate_app(env, gym_compat=True)
8
uvicorn.run(app, host="0.0.0.0", port=9090)

JSONL row format (matches ng_collect_rollouts input spec):

1
{
2
  "responses_create_params": {
3
    "input": [{"role": "user", "content": "Solve: ..."}]
4
  },
5
  "expected_answer": "42",
6
  "uuid": "gsm8k-0",
7
  "eval_type": "gsm8k",
8
  "metadata": {"category": "math"}
9
}

Pattern 6: Regression Comparison

Who: CI pipeline or evaluation owner comparing model versions.

What: Two run bundles → score deltas with CI overlap check, Mann-Whitney U p-values, per-category deltas, runtime deltas.

1
from nemo_evaluator.engine.comparison import compare_runs, write_regression
2
3
report = compare_runs("results/v1/eval-*.json", "results/v2/eval-*.json")
4
write_regression(report, "results/regression.json")

Output:

1
{
2
  "score_deltas": {
3
    "pass@1": {"baseline": 0.85, "candidate": 0.88, "delta": 0.03, "ci_overlap": true, "p_value": 0.031, "significant": true}
4
  },
5
  "category_deltas": {
6
    "algebra": {"baseline": 0.92, "candidate": 0.95, "delta": 0.03},
7
    "geometry": {"baseline": 0.71, "candidate": 0.68, "delta": -0.03}
8
  },
9
  "runtime_deltas": {
10
    "tokens_per_second": {"baseline": 41.0, "candidate": 38.5, "delta": -2.5}
11
  }
12
}

Artifact Summary

Every evaluation run (all patterns) produces:

Environment Compatibility Matrix

BYOB: Writing a New Benchmark

1
from nemo_evaluator import benchmark, scorer, ScorerInput, answer_line
2
3
@benchmark(
4
    name="my_benchmark",
5
    dataset="hf://my-org/my-data?split=test",
6
    prompt="Question: {question}\nAnswer:",
7
    target_field="answer",
8
)
9
@scorer
10
def my_scorer(sample: ScorerInput) -> dict:
11
    return answer_line(sample)

Then:

$
nel validate -b my_benchmark --samples 10             # sanity check
$
nel eval run --bench my_benchmark --repeats 4          # full evaluation
$
nel serve -b my_benchmark --gym-compat                 # serve for Gym

Pattern 7: Distributed Evaluation on SLURM

User story: “I need to evaluate a 14k-problem benchmark with n=8 repeats. Running serially would take days. I want to shard across 16 SLURM nodes and merge results.”

Architecture

                           SLURM Controller
                                 │
                    sbatch --array=0-15
                    ┌────────────┼────────────┐
                    ▼            ▼             ▼
               Shard 0      Shard 1  ...  Shard 15
              [p0-p874]    [p875-p1749]  [p13125-p13999]
            nel eval run   nel eval run   nel eval run
                    │            │             │
                    └────────────┼────────────┘
                                 ▼
                          Merge Job
                     (afterok dependency)
                                 │
                                 ▼
                     merged/eval-*.json
                     merged/results.jsonl
                     merged/trajectories.jsonl

Config-Driven SLURM

SLURM evaluations are driven by a YAML config with a cluster: { type: slurm } block:

1
# slurm_eval.yaml
2
services:
3
  model:
4
    type: vllm
5
    model: nvidia/Llama-3.1-70B-Instruct
6
    protocol: chat_completions
7
    tensor_parallel_size: 4
8
    port: 8000
9
    node_pool: compute
10
11
benchmarks:
12
  - name: gsm8k
13
    repeats: 8
14
    solver:
15
      type: simple
16
      service: model
17
18
cluster:
19
  type: slurm
20
  walltime: "04:00:00"
21
  shards: 16
22
  node_pools:
23
    compute:
24
      partition: batch
25
      nodes: 1
26
      ntasks_per_node: 1
27
      gres: "gpu:4"
28
29
output:
30
  dir: ./eval_results/gsm8k_distributed

$
# Submit via config (16 shards, auto-merge after completion)
$
nel eval run slurm_eval.yaml
$
$
# Generate scripts to inspect first (set submit: false in config)
$
nel eval run slurm_eval.yaml

Shard merging is handled automatically when all array tasks complete.

How Sharding Works

Each SLURM array task gets SLURM_ARRAY_TASK_ID and SLURM_ARRAY_TASK_COUNT set automatically. nel eval run detects these (or NEL_SHARD_IDX/NEL_TOTAL_SHARDS) and computes its problem range:

Each shard writes its own artifacts to shard_N/. The merge job combines all results, recomputes global metrics (pass@k, CI), and aggregates runtime stats.

Serve on SLURM

For long-running Gym training, serve an environment as a SLURM service:

$
nel serve -b gsm8k --gym-compat --port 9090
$
$
# Wrap in an sbatch script for SLURM submission
$
# Allocated endpoint is written to eval_results/endpoint.txt
$
# Gym training nodes connect via: gym://$(cat eval_results/endpoint.txt)

Environment Variables

Pattern 8: Docker / Docker Compose

User story: “I want to run evaluations in containers for reproducibility, or spin up a serve+eval stack locally.”

$
# Build the image
$
docker build -t nemo-evaluator .
$
$
# Single eval
$
docker run -e NEMO_API_KEY=$KEY nemo-evaluator eval run \
>
    --bench gsm8k --repeats 2 --output-dir /results
$
$
# Serve + eval (docker compose)
$
cd deploy
$
NEMO_API_KEY=$KEY docker compose up serve eval-remote
$
$
# Sharded (manually, 4 shards)
$
for i in 0 1 2 3; do
$
  NEL_SHARD_IDX=$i NEL_TOTAL_SHARDS=4 docker compose \
>
    --profile sharded run -d eval-shard
$
done

Files: Dockerfile, deploy/docker-compose.yaml

Pattern 9: Kubernetes

User story: “I need to run evaluations on our K8s cluster, with sharded jobs for large benchmarks and a persistent serve endpoint for Gym training.”

Single Evaluation Job

$
kubectl apply -f deploy/k8s/eval-job.yaml
$
kubectl logs -f job/nel-eval

Sharded (Indexed Job)

Uses K8s completionMode: Indexed — each pod gets JOB_COMPLETION_INDEX mapped to NEL_SHARD_IDX.

$
kubectl apply -f deploy/k8s/eval-indexed-job.yaml
$
# 8 pods run in parallel, each evaluating its shard
$
kubectl wait --for=condition=complete job/nel-eval-sharded --timeout=2h
$
# Then run the merge job
$
kubectl apply -f deploy/k8s/eval-merge.yaml

Serve as K8s Service

$
kubectl apply -f deploy/k8s/serve-deployment.yaml
$
# Gym training pods connect via: gym://nel-serve.default.svc:9090

Includes readiness/liveness probes on /health, ClusterIP service for internal discovery.

Files: deploy/k8s/eval-job.yaml, deploy/k8s/eval-indexed-job.yaml, deploy/k8s/serve-deployment.yaml

Pattern 10: Ray (Distributed)

User story: “Our Gym training already runs on Ray. I want to run distributed evaluation on the same Ray cluster.”

$
# Submit as a Ray job
$
ray job submit --working-dir . -- python -m nemo_evaluator.engine.ray_launcher \
>
    --bench gsm8k --shards 8 --repeats 5 \
>
    --model-url https://inference-api.nvidia.com/v1 \
>
    --model-id azure/openai/gpt-5.2 \
>
    --api-key $NVIDIA_API_KEY \
>
    --output-dir ./eval_results/ray
$
$
# Or from within a Ray script
$
import ray
$
from deploy.ray_eval import run_shard
$
futures = [run_shard.remote("gsm8k", i, 8, ...) for i in range(8)]
$
results = ray.get(futures)

Each run_shard is a Ray remote task that runs run_evaluation() with a computed problem_range. Results are merged locally after all tasks complete. Works on existing Ray clusters used by Gym training.

File: src/nemo_evaluator/engine/ray_launcher.py

Pattern 11: GitLab CI Regression Gate

User story: “I want every MR to automatically evaluate the candidate model against the baseline and block merge if scores regress.”

1
# Include in your .gitlab-ci.yml:
2
include:
3
  - local: deploy/gitlab-ci-eval.yml

Pipeline stages:

1. eval:baseline — checks out target branch, runs eval
2. eval:candidate — runs eval on MR branch
3. regression:check — compares bundles, fails if any metric drops >5%

Produces regression.json artifact with per-metric deltas, CI overlap, and category breakdowns.

File: deploy/gitlab-ci-eval.yml

Deployment Matrix

All targets use the same nel eval run / nel serve commands, same sharding env vars (NEL_SHARD_IDX, NEL_TOTAL_SHARDS), and same artifact format. The only difference is the orchestration layer.