Legacy Evaluator Containers

Run any existing NeMo Evaluator harness container (simple-evals, lm-evaluation-harness, nemo-skills, mtbench, etc.) through NEL’s unified interface.

How It Works

The container adapter does not decompose the harness. The harness owns the model call, which means you get aggregate scores and response stats from the adapter interceptors, but not per-request trajectories. For full observability, use the NeMo Skills Integration (native mode) instead.

Available Harnesses

$
nel list --source lm-eval

Plus 10+ more (see nel list --source lm-eval).

Python API

1
from nemo_evaluator.environments.container import ContainerEnvironment
2
from nemo_evaluator.engine.eval_loop import run_evaluation
3
from nemo_evaluator.solvers import ChatSolver
4
from nemo_evaluator.engine.model_client import ModelClient
5
6
env = ContainerEnvironment(
7
    image="nvcr.io/.../simple-evals:26.01",
8
    task="GPQA_diamond",
9
)
10
11
client = ModelClient(
12
    base_url="https://inference-api.nvidia.com/v1",
13
    model="azure/openai/gpt-5.2",
14
)
15
solver = ChatSolver(client)
16
bundle = await run_evaluation(env, solver)

Output Format

The container adapter parses results.yml and eval_factory_metrics.json from the container output:

1
{
2
  "source": "container",
3
  "image": "nvcr.io/.../simple-evals:26.01",
4
  "task": "simple_evals.AIME_2025",
5
  "scores": {
6
    "AIME_2025/score/micro": {"value": 0.4, "stats": {"stddev": 0.49, "stderr": 0.16}}
7
  },
8
  "runtime": {
9
    "elapsed_seconds": 120.5,
10
    "inference_time_seconds": 98.2,
11
    "scoring_time_seconds": 22.3
12
  },
13
  "response_stats": {
14
    "avg_latency_ms": 1250.0,
15
    "avg_prompt_tokens": 320,
16
    "avg_completion_tokens": 450,
17
    "count": 50,
18
    "successful_count": 48
19
  }
20
}

Observability Trade-offs

For full observability, see NeMo Skills Integration which wraps Skills benchmarks natively.