Evaluator NeMo Platform SDK Resources

The nemo_evaluator_sdk package provides context-agnostic objects for defining metrics, datasets, evaluation configuration, and result handling. When you want to execute those evaluations through the NeMo Platform Evaluator plugin, use the Evaluator SDK resource mounted on the nemo_platform SDK. This page explains the NeMo Platform-specific objects used to submit durable platform jobs and retrieve evaluator job results.

Evaluator

The Evaluator resource is the sync SDK object for working with the Evaluator plugin on NeMo Platform. It is accessed directly from a NeMoPlatform instance:

1 import os
2 from nemo_evaluator.sdk import Evaluator
3 from nemo_platform import NeMoPlatform
4 
5 client = NeMoPlatform(
6     base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
7     workspace="default",
8 )
9 evaluator: Evaluator = client.evaluator  # this object is an Evaluator resource

The primary execution method is submit, which creates a durable remote platform job whose lifecycle you manage through the returned job resource. For local in-process evaluation that returns a completed EvaluationResult without the platform, use nemo_evaluator_sdk.Evaluator directly.

Method	Description	Returns
`submit()`	Submits one metric evaluation as a durable platform job.	`EvaluatorJobResource`
`plugin_status()`	Returns Evaluator plugin health information from the service.	`dict[str, object]`
`get_job_resource(job_name: str, workspace: str \	None = None)`	Returns a resource for an existing Evaluator plugin job.

The dataset argument accepts inline rows, local dataset paths, local glob paths, and fileset references with optional fragment selectors. Use config for evaluator runtime settings, and target plus prompt_template when the evaluator should generate model or agent responses before scoring.

`submit()` arguments

Argument	Type	Required	Description
`metric`	`Metric`	Yes	Metric configuration serialized into the durable platform job.
`dataset`	`PluginDatasetInput`	Yes	Inline rows, local dataset paths, local glob paths, or fileset references with optional fragment selectors.
`config`	`RunConfig \	RunConfigOnline \	RunConfigOnlineModel \
`target`	`Model \	ModelRef \	Agent \
`prompt_template`	`str \	dict[str, Any] \	None`

Submit a platform job

1 from nemo_evaluator_sdk import ExactMatchMetric
2 
3 metric = ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}")
4 dataset = [
5     {"expected": "Paris", "output": "Paris"},
6     {"expected": "Berlin", "output": "Munich"},
7 ]
8 
9 job = evaluator.submit(metric=metric, dataset=dataset)
10 job.wait_until_done()
11 result = job.get_result()
12 print(result.aggregate_scores)

AsyncEvaluator

The AsyncEvaluator resource provides the same Evaluator plugin surface for AsyncNeMoPlatform. Async methods must be awaited:

1 import os
2 from nemo_evaluator.sdk import AsyncEvaluator
3 from nemo_platform import AsyncNeMoPlatform
4 
5 client = AsyncNeMoPlatform(
6     base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
7     workspace="default",
8 )
9 evaluator: AsyncEvaluator = client.evaluator

Method	Description	Returns
`submit()`	Submits one metric evaluation as a durable platform job.	`AsyncEvaluatorJobResource`
`plugin_status()`	Returns Evaluator plugin health information from the service.	`dict[str, object]`
`get_job_resource(job_name: str, workspace: str \	None = None)`	Returns a resource for an existing Evaluator plugin job.

AsyncEvaluator.submit() accepts the same arguments as the sync method above.

1 import asyncio
2 
3 from nemo_evaluator_sdk import ExactMatchMetric
4 
5 metric = ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}")
6 dataset = [
7     {"expected": "Paris", "output": "Paris"},
8     {"expected": "Berlin", "output": "Munich"},
9 ]
10 
11 async def main() -> None:
12     job = await evaluator.submit(metric=metric, dataset=dataset)
13     await job.wait_until_done()
14     result = await job.get_result()
15     print(result.aggregate_scores)
16 
17 asyncio.run(main())

EvaluatorJobResource

The EvaluatorJobResource is the sync job handle returned by Evaluator.submit. You can also reconnect to an existing job with Evaluator.get_job_resource.

Some of the most useful methods and properties are described below.

Method or property	Description
`name`	Returns the evaluator job name.
`job`	Returns the raw evaluator job payload captured at resource creation.
`get_job_status()`	Fetches the current evaluator job status from the Evaluator plugin API.
`check_if_complete(raise_if_not_complete: bool = False)`	Returns whether the job is complete. When `raise_if_not_complete` is true, raises for any status other than `completed`.
`wait_until_done()`	Polls the job until it reaches a terminal platform status. Raises if the job fails or times out.
`get_result(aggregate_fields=None)`	Downloads aggregate-score and row-score artifacts and returns an `EvaluationResult`. Optional `aggregate_fields` shapes the returned aggregate scores only.
`download_artifacts(path=None)`	Downloads and extracts the full job artifacts archive under a job-specific directory.
`as_async()`	Returns an `AsyncEvaluatorJobResource` view over the same job.

AsyncEvaluatorJobResource

The AsyncEvaluatorJobResource is the async job handle returned by AsyncEvaluator.submit. It mirrors EvaluatorJobResource, but status and result methods are awaited.