Architecture

System Overview

Package Map

Environment Abstraction

Everything is an EvalEnvironment. Built-in benchmarks, remote Gym servers, NeMo Skills tasks, lm-eval harness tasks, and VLMEvalKit benchmarks all implement the same contract:

Resolution

The registry resolves environment names in order:

1. URI scheme — lm-eval://task, skills://name, gym://host:port, gym://name, vlmevalkit://dataset, container://image#task
2. Built-in registry — names registered via @benchmark or @register

1
from nemo_evaluator import get_environment
2
3
env = get_environment("mmlu")                        # built-in
4
env = get_environment("lm-eval://aime25")            # lm-eval task
5
env = get_environment("skills://gpqa")               # NeMo Skills
6
env = get_environment("gym://localhost:9090")         # remote Gym
7
env = get_environment("vlmevalkit://MMBench_DEV_EN") # VLMEvalKit

Evaluation Flow

Solver Protocol

Solvers decouple inference strategy from benchmark logic. The eval loop calls solver.solve(task) and receives a response. In YAML configs, solvers are configured via solver.type in each benchmark.

1
class Solver(Protocol):
2
    async def solve(self, task: SeedResult) -> SolveResult: ...

Executor Protocol

All execution backends implement the Executor protocol (executors/__init__.py):

1
class Executor(Protocol):
2
    name: str
3
    def run(self, config, *, dry_run=False, resume=False,
4
            background=False, submit=False) -> None: ...
5
    def status(self, output_dir) -> ProcessState: ...
6
    def stop(self, output_dir) -> bool: ...
7
    @staticmethod
8
    def detect(output_dir) -> bool: ...

The CLI dispatches via get_executor(config.cluster.type) and detect_executor(output_dir) — no if/elif trees.

SLURM uses node_pools to declare resource topology. Services and sandboxes reference pools by name, enabling heterogeneous jobs (e.g., GPU nodes for model serving + CPU nodes for sandboxes).

Adding a new executor (e.g. Kubernetes) requires only a new class, a metadata file convention, and a registry entry.

$
# Local with external API
$
nel eval run --bench mmlu --model-url https://api.example.com/v1
$
$
# Docker
$
nel eval run config.yaml    # with cluster.type: docker
$
$
# SLURM (generates + submits sbatch)
$
nel eval run config.yaml    # with cluster.type: slurm

Model Deployment

The orchestration/model_server.py module manages model server lifecycle:

All deployments implement start() -> url, health_wait(), and stop().

Observability Data Model

Resilience and Resume

Multi-benchmark suites use CheckpointManager to track per-benchmark completion:

• Failure isolation: A failing benchmark is caught, logged, and skipped. The suite continues to the next benchmark.
• Checkpoint tracking: Each benchmark’s status (completed/failed) is persisted to disk under the output directory.
• Resume: --resume skips completed benchmarks and retries failed ones. Without --resume, checkpoints are cleared for a fresh run.

Sharding

Sharding is transparent to the eval loop. problem_range changes the iteration bounds. The merge step recomputes global metrics from concatenated per-shard results.