DocumentationEvaluate Agents & ModelsMetrics

Model Configuration

View as Markdown

Online evaluations use Model objects for model endpoints. A model can be the evaluation target that produces outputs, or it can be part of a judge-style metric such as LLM-as-a-Judge, RAG, or agentic metrics.

The Evaluator plugin SDK uses inline model objects from nemo_evaluator_sdk. Pass the model either as target=... or as a field on the metric class that needs a judge or embeddings model.

Initialize the SDK

1import os
2
3from nemo_evaluator.sdk import Evaluator
4from nemo_platform import NeMoPlatform
5
6client = NeMoPlatform(
7    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
8    workspace="default",
9)
10evaluator: Evaluator = client.evaluator  # this object is an Evaluator resource

Inline Model

Define the endpoint URL and model name directly:

1from nemo_evaluator_sdk import Model, SecretRef
2
3model = Model(
4    url="https://integrate.api.nvidia.com/v1",
5    name="nvidia/nemotron-3.5-lightning-30b-a3b",
6    api_key_secret=SecretRef(root="nvidia-api-key"),
7)
FieldRequiredDescription
urlYesBase URL of the inference endpoint.
nameYesModel name to send in inference requests.
formatNoDeprecated and ignored. Structured output support is detected from the endpoint during preflight rather than inferred from this label.
api_key_secretNoModel API key reference. See Model API Authentication.

Model API Authentication

api_key_secret is an optional property on the Model object. Omit it when the endpoint does not require API-key authentication.

For nemo_evaluator_sdk.Evaluator runs, the SecretRef root names an environment variable available to the local Python process. For example, api_key_secret=SecretRef(root="NVIDIA_API_KEY") reads os.environ["NVIDIA_API_KEY"].

For remote evaluator.submit(...) jobs, the SecretRef root must name a NeMo platform secret in the target workspace. Create the secret before submitting the job:

1client.secrets.create(
2    name="nvidia-api-key",
3    value=os.environ["NVIDIA_API_KEY"],
4)

Model as the Evaluation Target

Use target=model when the evaluator should call the model to generate the sample output before scoring.

1from nemo_evaluator_sdk import (
2    RunConfigOnlineModel,
3    ExactMatchMetric,
4    InferenceParams,
5    Model,
6    SecretRef,
7)
8
9model = Model(
10    url="https://integrate.api.nvidia.com/v1",
11    name="nvidia/nemotron-3.5-lightning-30b-a3b",
12    api_key_secret=SecretRef(root="nvidia-api-key"),
13)
14
15metric = ExactMatchMetric(reference="{{item.expected_answer}}")
16dataset = [
17    {"question": "What is the capital of France?", "expected_answer": "Paris"},
18]
19
20job = evaluator.submit(
21    metric=metric,
22    dataset=dataset,
23    config=RunConfigOnlineModel(
24        parallelism=4,
25        inference=InferenceParams(temperature=0.1, max_tokens=64),
26    ),
27    target=model,
28    prompt_template={
29        "messages": [
30            {
31                "role": "user",
32                "content": "Answer this question concisely: {{item.question}}",
33            },
34        ],
35    },
36)
37job.wait_until_done()
38result = job.get_result()

Model on a Judge Metric

Use a model field on the metric when the metric itself calls an LLM to score existing outputs.

1from nemo_evaluator_sdk import LLMJudgeMetric, Model, RangeScore, SecretRef
2
3judge_model = Model(
4    url="https://integrate.api.nvidia.com/v1",
5    name="nvidia/nemotron-3.5-lightning-30b-a3b",
6    api_key_secret=SecretRef(root="nvidia-api-key"),
7)
8metric = LLMJudgeMetric(
9    model=judge_model,
10    scores=[
11        RangeScore(
12            name="correctness",
13            description="Correctness from 1 to 5.",
14            minimum=1,
15            maximum=5,
16        ),
17    ],
18    prompt_template={
19        "messages": [
20            {
21                "role": "system",
22                "content": "Return JSON with a correctness score from 1 to 5.",
23            },
24            {
25                "role": "user",
26                "content": "Question: {{item.question}}\nAnswer: {{item.output}}\nExpected: {{item.expected_answer}}",
27            },
28        ],
29    },
30)
31
32job = evaluator.submit(
33    metric=metric,
34    dataset=[
35        {
36            "question": "What is the capital of France?",
37            "output": "Paris",
38            "expected_answer": "Paris",
39        },
40    ],
41)
42job.wait_until_done()
43result = job.get_result()

Runtime Parameters

Use RunConfigOnlineModel for model-target evaluations:

1from nemo_evaluator_sdk import (
2    RunConfigOnlineModel,
3    InferenceParams,
4    ReasoningParams,
5)
6
7params = RunConfigOnlineModel(
8    parallelism=4,
9    request_timeout=60,
10    max_retries=2,
11    ignore_request_failure=False,
12    inference=InferenceParams(temperature=0.2, max_tokens=256),
13    reasoning=ReasoningParams(end_token="</think>"),
14)

Use plain RunConfig for offline evaluations where the dataset already contains the output to score.

Model References

You can supply the evaluation target two ways. Which one is valid depends on whether you submit the evaluation as a durable platform job or run it locally with nemo_evaluator_sdk.Evaluator.

Inline Model

An inline Model carries the resolved endpoint details. Always pass an inline Model as the target (or as a judge/embeddings field on the metric). If your deployment stores platform model entities, resolve the entity into endpoint details before constructing the Model:

1from nemo_evaluator_sdk import Model, RunConfigOnlineModel, SecretRef
2
3model_entity = client.models.retrieve("my-model", workspace="default")
4model = Model(
5    url=client.models.get_model_entity_route_openai_url(model_entity),
6    name="my-model",
7    api_key_secret=SecretRef(root="nvidia-api-key"),
8)
9
10job = evaluator.submit(
11    metric=metric,
12    dataset=dataset,
13    config=RunConfigOnlineModel(parallelism=4),
14    target=model,
15    prompt_template={
16        "messages": [
17            {
18                "role": "user",
19                "content": "Answer this question concisely: {{item.question}}",
20            },
21        ],
22    },
23)
24job.wait_until_done()
25result = job.get_result()

ModelRef (supported by evaluator.submit(...))

Durable remote evaluator.submit(...) jobs additionally accept a ModelRef target. A ModelRef names a platform model entity (workspace/model-name) and is resolved by the evaluator backend when the job runs, so you do not have to resolve the endpoint yourself. Use this for platform-managed model routing. A ModelRef target generates outputs online, so it requires an online run config (RunConfigOnlineModel):

1from nemo_evaluator_sdk import ModelRef, RunConfigOnlineModel
2
3job = evaluator.submit(
4    metric=metric,
5    dataset=dataset,
6    config=RunConfigOnlineModel(),
7    target=ModelRef(root="default/my-model"),
8    prompt_template={
9        "messages": [
10            {
11                "role": "user",
12                "content": "Answer this question concisely: {{item.question}}",
13            },
14        ],
15    },
16)

evaluator.submit(...) accepts either a Model or a ModelRef. nemo_evaluator_sdk.Evaluator resolves no platform entities, so pass it an inline Model. See the Define and Run Custom Python Metrics tutorial for an end-to-end ModelRef + FilesetRef submit example.

For evaluating agentic systems, use an Agent request target instead of a Model. See Agent Configuration.