External Environments | NVIDIA NeMo Evaluator

All external integrations are EvalEnvironment subclasses resolved through the unified registry.

Environment Types

Environment	URI	Source
`GymEnvironment`	`gym://host:port`	Remote HTTP server (Gym protocol)
`ManagedGymEnvironment`	`gym://name`	Auto-started Gym server (name auto-detected)
`SkillsEnvironment`	`skills://name`	NeMo Skills benchmarks
`LMEvalEnvironment`	`lm-eval://task`	lm-evaluation-harness tasks
`VLMEvalKitEnvironment`	`vlmevalkit://dataset`	VLMEvalKit VLM benchmarks
`ContainerEnvironment`	`container://image#task`	Legacy eval-factory containers

$ nel eval run --bench gym://host:port
$ nel eval run --bench skills://gpqa
$ nel eval run --bench lm-eval://aime2025
$ nel eval run --bench vlmevalkit://MMBench_DEV_EN

See the dedicated tutorials for each:

Gym Integration — Gym environments
NeMo Skills Integration — NeMo Skills benchmarks

Writing a Custom Environment

Implement EvalEnvironment directly:

1 from nemo_evaluator import EvalEnvironment, SeedResult, VerifyResult, register
2 
3 @register("my_platform")
4 class MyPlatformEnvironment(EvalEnvironment):
5     def __init__(self, config: str = "default"):
6         super().__init__()
7         self._dataset = self._load(config)
8 
9     async def seed(self, idx: int) -> SeedResult:
10         row = self._dataset[idx]
11         return SeedResult(prompt=row["prompt"], expected_answer=row["answer"])
12 
13     async def verify(self, response: str, expected: str, **meta) -> VerifyResult:
14         correct = response.strip().lower() == expected.strip().lower()
15         return VerifyResult(reward=1.0 if correct else 0.0)
16 
17     def _load(self, config):
18         ...

Or use the @benchmark + @scorer API for simpler cases (see Write Your Own Benchmark (BYOB)).