Skip to navigation
Architecture

Architecture

View as Markdown

System Overview

CLI (nel) Environments Engine Config Orchestration Executors Sandbox Proxy Observability Scoring nel eval run nel serve nel validate nel eval report nel compare nel gate Registry @benchmark + @scorer GymEnvironment SkillsEnvironment LMEvalEnvironment VLMEvalKitEnvironment eval_loop Solver ModelClient EvalConfig compose ModelServer Orchestrator SlurmGen LocalExecutor DockerExecutor SlurmExecutor Sandbox Protocol DockerSandbox SlurmSandbox LocalSandbox ECSFargateSandbox SandboxManager LiteLLM Proxy Interceptors StepRecordModelResponse ArtifactCollector ConsoleProgress judge.py json_schema.py

Package Map

PackageResponsibilityKey types
environments/Base class, registry, @benchmark BYOB API, environment typesEvalEnvironment, SeedResult, VerifyResult, BenchmarkDefinition
benchmarks/17 built-in benchmarks (@benchmark + @scorer or @register)Scorer functions
solvers/Pluggable inference strategiesSolver, ChatSolver, VLMSolver, HarborSolver, GymSolver, ReActSolver, NatSolver, OpenClawSolver (config type: simple → ChatSolver/VLMSolver, harbor, tool_calling → ReActSolver, gym_delegation, etc.)
engine/Core eval loop, model client, checkpoint, comparison, export pluginsrun_evaluation(), ModelClient, CheckpointManager, InspectExporter, WandBExporter
config/Pydantic config schemas, env expansion, YAML compositionEvalConfig, parse_eval_config(), compose_config(), services.py, sandboxes.py, solvers.py, scoring.py, clusters.py
orchestration/Suite orchestration, model server management, SLURM generation, proxy lifecyclerun_local(), DeployConfig, generate_sbatch(), start_proxy()
serving/HTTP server for environments (evaluator + Gym protocol)generate_app()
executors/Executor protocol and backends (local, Docker, SLURM)Executor, get_executor(), detect_executor(), ProcessState
sandbox/Per-problem isolated execution and strategy patternsSandbox, SandboxSpec, SandboxManager, ECSFargateSandbox, NoSandbox, StatefulSandbox, StatelessSandbox
interceptors/LiteLLM proxy callback plugins for request/response interceptionresolve_interceptors(), BaseInterceptor
scoring/Verification scorers, judge pipeline, JSON schemaexact_match, code_sandbox, needs_judge
observability/Rich telemetry captureStepRecord, ModelResponse, RuntimeStats, ArtifactCollector
metrics/Statistical aggregationpass_at_k(), bootstrap_ci(), category_breakdown()
cli/CLI commandsnel eval run, nel eval report, nel eval merge, nel serve, nel validate, nel list, nel compare, nel gate, nel config, nel package

Environment Abstraction

Everything is an EvalEnvironment. Built-in benchmarks, remote Gym servers, NeMo Skills tasks, lm-eval harness tasks, and VLMEvalKit benchmarks all implement the same contract:

<<abstract>> EvalEnvironment + name: str + seed(idx): SeedResult + verify(response, expected, sandbox?): VerifyResult + sandbox_specs(): list~SandboxSpec~ | None + dataset_size(): int + close() ByobEnvironment + definition: BenchmarkDefinition GymEnvironment + endpoint: str SkillsEnvironment + benchmark: str + eval_type: str LMEvalEnvironment + task_name: str VLMEvalKitEnvironment + dataset_name: str ContainerEnvironment + image_uri: str + task_name: str

Resolution

The registry resolves environment names in order:

  1. URI scheme — lm-eval://task, skills://name, gym://host:port, gym://name, vlmevalkit://dataset, container://image#task
  2. Built-in registry — names registered via @benchmark or @register
from nemo_evaluator import get_environment
env = get_environment("mmlu")                        # built-in
env = get_environment("lm-eval://aime25")            # lm-eval task
env = get_environment("skills://gpqa")               # NeMo Skills
env = get_environment("gym://localhost:9090")         # remote Gym
env = get_environment("vlmevalkit://MMBench_DEV_EN") # VLMEvalKit

Evaluation Flow

opt [proxy configured] opt [sandbox configured] opt [sandbox acquired] loop [For each problem x repeat] executor.run(config) run_local(config) start() model_url start_proxy(model_url) run_evaluation(env, solver, config, sandbox_manager) seed(idx) SeedResult(prompt, expected, sandbox_spec?) manager.acquire(spec) solve(task, sandbox?) SolveResult(response) verify(response, expected, sandbox?) VerifyResult(reward, details) manager.release(sandbox) record(StepRecord) bundle stop() model_url rewritten to proxy URL nel eval run Executor Orchestrator ModelServer eval_loop Solver Environment ArtifactCollector

Solver Protocol

Solvers decouple inference strategy from benchmark logic. The eval loop calls solver.solve(task) and receives a response. In YAML configs, solvers are configured via solver.type in each benchmark.

SolverConfig typeProtocolUse case
ChatSolversimple/chat/completionsStandard benchmarks (default)
VLMSolversimple (images)/chat/completions + imagesVision-language benchmarks
HarborSolverharborHarbor agent SDKAgentic evaluation (OpenHands, SWE-agent)
ReActSolvertool_callingNEL-driven ReAct loopFull-observability tool use (Gym HTTP tools, sandbox tools, or both)
GymSolvergym_delegationHTTP to nemo-gymDelegate solve to gym server
NatSolver(via service)SSE /generate/fullNAT agent benchmarks
OpenClawSolveropenclawOpenClaw CLIOpenClaw benchmarks
class Solver(Protocol):
    async def solve(self, task: SeedResult) -> SolveResult: ...

Executor Protocol

All execution backends implement the Executor protocol (executors/__init__.py):

class Executor(Protocol):
    name: str
    def run(self, config, *, dry_run=False, resume=False,
            background=False, submit=False) -> None: ...
    def status(self, output_dir) -> ProcessState: ...
    def stop(self, output_dir) -> bool: ...
    @staticmethod
    def detect(output_dir) -> bool: ...

The CLI dispatches via get_executor(config.cluster.type) and detect_executor(output_dir) — no if/elif trees.

ExecutorConfigMetadata fileWhat it does
LocalExecutorcluster.type: localnel.pidIn-process eval with optional model deployment, checkpointing, and failure isolation
DockerExecutorcluster.type: dockerdocker.jsonRuns eval inside a Docker container with the correct per-harness image
SlurmExecutorcluster.type: slurmslurm_job.jsonGenerates self-contained sbatch scripts with per-benchmark containers

SLURM uses node_pools to declare resource topology. Services and sandboxes reference pools by name, enabling heterogeneous jobs (e.g., GPU nodes for model serving + CPU nodes for sandboxes).

Adding a new executor (e.g. Kubernetes) requires only a new class, a metadata file convention, and a registry entry.

# Local with external API
nel eval run --bench mmlu --model-url https://api.example.com/v1
# Docker
nel eval run config.yaml    # with cluster.type: docker
# SLURM (generates + submits sbatch)
nel eval run config.yaml    # with cluster.type: slurm

Model Deployment

The orchestration/model_server.py module manages model server lifecycle:

Config typeInternal classDescription
apiAPIDeploymentExternal API, no server management
vllmProcessModelDeploymentLocal vLLM process
sglangProcessModelDeploymentLocal SGLang process
trt_llmProcessModelDeploymentLocal TensorRT-LLM process

All deployments implement start() -> url, health_wait(), and stop().

Observability Data Model

StepRecord + step_id: str + problem_idx: int + repeat: int + prompt: str + expected_answer: str + model_response: ModelResponse + reward: float + extracted_answer: str + scoring_method: str + scoring_details: dict + seed_ms: float + model_ms: float + verify_ms: float + failure_category: str ModelResponse + content: str + model: str + finish_reason: str + prompt_tokens: int + completion_tokens: int + reasoning_tokens: int + latency_ms: float + raw_response: dict RuntimeStats + total_steps: int + total_tokens: int + elapsed_seconds: float + latency_percentiles_ms: dict + tokens_per_second: float + steps_per_second: float + model_errors: int FailureReport + total_failures: int + failure_rate: float + categories: dict + exemplars: list

Resilience and Resume

Multi-benchmark suites use CheckpointManager to track per-benchmark completion:

completed FAILED completed nel eval run suite.yaml Benchmark 1 checkpoint: done Benchmark 2 checkpoint: failed Benchmark 3 checkpoint: done Summary: 2 done, 1 failed nel eval run suite.yaml --resume Retry Benchmark 2
  • Failure isolation: A failing benchmark is caught, logged, and skipped. The suite continues to the next benchmark.
  • Checkpoint tracking: Each benchmark’s status (completed/failed) is persisted to disk under the output directory.
  • Resume: --resume skips completed benchmarks and retries failed ones. Without --resume, checkpoints are cleared for a fresh run.

Sharding

Environment Variables get_shard_range(total, idx, shards) run_evaluation(...,problem_range=(start, end)) shard_N/results.jsonl merge_results(shard_dirs) merged/eval-*.jsonRecomputed pass@k, CI NEL_SHARD_IDXor SLURM_ARRAY_TASK_ID NEL_TOTAL_SHARDSor SLURM_ARRAY_TASK_COUNT

Sharding is transparent to the eval loop. problem_range changes the iteration bounds. The merge step recomputes global metrics from concatenated per-shard results.