> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Model Configuration

Online evaluations use `Model` objects for model endpoints. A model can be the evaluation target that produces outputs, or it can be part of a judge-style metric such as LLM-as-a-Judge, RAG, or agentic metrics.

The Evaluator plugin SDK uses inline model objects from `nemo_evaluator_sdk`. Pass the model either as `target=...` or as a field on the metric class that needs a judge or embeddings model.

## Initialize the SDK

```python
import os

from nemo_evaluator.sdk import Evaluator
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)
evaluator: Evaluator = client.evaluator  # this object is an Evaluator resource
```

## Inline Model

Define the endpoint URL and model name directly:

```python
from nemo_evaluator_sdk import Model, SecretRef

model = Model(
    url="https://integrate.api.nvidia.com/v1",
    name="nvidia/nemotron-3.5-lightning-30b-a3b",
    api_key_secret=SecretRef(root="nvidia-api-key"),
)
```

| Field            | Required | Description                                                                                                                                |
| ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `url`            | Yes      | Base URL of the inference endpoint.                                                                                                        |
| `name`           | Yes      | Model name to send in inference requests.                                                                                                  |
| `format`         | No       | **Deprecated and ignored.** Structured output support is detected from the endpoint during preflight rather than inferred from this label. |
| `api_key_secret` | No       | Model API key reference. See [Model API Authentication](#model-api-authentication).                                                        |

## Model API Authentication

`api_key_secret` is an optional property on the `Model` object. Omit it when the endpoint does not require API-key authentication.

For `nemo_evaluator_sdk.Evaluator` runs, the `SecretRef` root names an environment variable available to the local Python process. For example, `api_key_secret=SecretRef(root="NVIDIA_API_KEY")` reads `os.environ["NVIDIA_API_KEY"]`.

For remote `evaluator.submit(...)` jobs, the `SecretRef` root must name a NeMo platform secret in the target workspace. Create the secret before submitting the job:

```python
client.secrets.create(
    name="nvidia-api-key",
    value=os.environ["NVIDIA_API_KEY"],
)
```

## Model as the Evaluation Target

Use `target=model` when the evaluator should call the model to generate the sample output before scoring.

```python
from nemo_evaluator_sdk import (
    RunConfigOnlineModel,
    ExactMatchMetric,
    InferenceParams,
    Model,
    SecretRef,
)

model = Model(
    url="https://integrate.api.nvidia.com/v1",
    name="nvidia/nemotron-3.5-lightning-30b-a3b",
    api_key_secret=SecretRef(root="nvidia-api-key"),
)

metric = ExactMatchMetric(reference="{{item.expected_answer}}")
dataset = [
    {"question": "What is the capital of France?", "expected_answer": "Paris"},
]

job = evaluator.submit(
    metric=metric,
    dataset=dataset,
    config=RunConfigOnlineModel(
        parallelism=4,
        inference=InferenceParams(temperature=0.1, max_tokens=64),
    ),
    target=model,
    prompt_template={
        "messages": [
            {
                "role": "user",
                "content": "Answer this question concisely: {{item.question}}",
            },
        ],
    },
)
job.wait_until_done()
result = job.get_result()
```

## Model on a Judge Metric

Use a model field on the metric when the metric itself calls an LLM to score existing outputs.

```python
from nemo_evaluator_sdk import LLMJudgeMetric, Model, RangeScore, SecretRef

judge_model = Model(
    url="https://integrate.api.nvidia.com/v1",
    name="nvidia/nemotron-3.5-lightning-30b-a3b",
    api_key_secret=SecretRef(root="nvidia-api-key"),
)
metric = LLMJudgeMetric(
    model=judge_model,
    scores=[
        RangeScore(
            name="correctness",
            description="Correctness from 1 to 5.",
            minimum=1,
            maximum=5,
        ),
    ],
    prompt_template={
        "messages": [
            {
                "role": "system",
                "content": "Return JSON with a correctness score from 1 to 5.",
            },
            {
                "role": "user",
                "content": "Question: {{item.question}}\nAnswer: {{item.output}}\nExpected: {{item.expected_answer}}",
            },
        ],
    },
)

job = evaluator.submit(
    metric=metric,
    dataset=[
        {
            "question": "What is the capital of France?",
            "output": "Paris",
            "expected_answer": "Paris",
        },
    ],
)
job.wait_until_done()
result = job.get_result()
```

## Runtime Parameters

Use `RunConfigOnlineModel` for model-target evaluations:

```python
from nemo_evaluator_sdk import (
    RunConfigOnlineModel,
    InferenceParams,
    ReasoningParams,
)

params = RunConfigOnlineModel(
    parallelism=4,
    request_timeout=60,
    max_retries=2,
    ignore_request_failure=False,
    inference=InferenceParams(temperature=0.2, max_tokens=256),
    reasoning=ReasoningParams(end_token="</think>"),
)
```

Use plain `RunConfig` for offline evaluations where the dataset already contains the output to score.

## Model References

You can supply the evaluation target two ways. Which one is valid depends on whether you submit the evaluation as a durable platform job or run it locally with `nemo_evaluator_sdk.Evaluator`.

### Inline `Model`

An inline `Model` carries the resolved endpoint details. Always pass an inline `Model` as the `target` (or as a judge/embeddings field on the metric). If your deployment stores platform model entities, resolve the entity into endpoint details before constructing the `Model`:

```python
from nemo_evaluator_sdk import Model, RunConfigOnlineModel, SecretRef

model_entity = client.models.retrieve("my-model", workspace="default")
model = Model(
    url=client.models.get_model_entity_route_openai_url(model_entity),
    name="my-model",
    api_key_secret=SecretRef(root="nvidia-api-key"),
)

job = evaluator.submit(
    metric=metric,
    dataset=dataset,
    config=RunConfigOnlineModel(parallelism=4),
    target=model,
    prompt_template={
        "messages": [
            {
                "role": "user",
                "content": "Answer this question concisely: {{item.question}}",
            },
        ],
    },
)
job.wait_until_done()
result = job.get_result()
```

### `ModelRef` (supported by `evaluator.submit(...)`)

Durable remote `evaluator.submit(...)` jobs additionally accept a `ModelRef` target. A `ModelRef` names a platform model entity (`workspace/model-name`) and is resolved by the evaluator backend when the job runs, so you do not have to resolve the endpoint yourself. Use this for platform-managed model routing. A `ModelRef` target generates outputs online, so it requires an online run config (`RunConfigOnlineModel`):

```python
from nemo_evaluator_sdk import ModelRef, RunConfigOnlineModel

job = evaluator.submit(
    metric=metric,
    dataset=dataset,
    config=RunConfigOnlineModel(),
    target=ModelRef(root="default/my-model"),
    prompt_template={
        "messages": [
            {
                "role": "user",
                "content": "Answer this question concisely: {{item.question}}",
            },
        ],
    },
)
```

`evaluator.submit(...)` accepts either a `Model` or a `ModelRef`. `nemo_evaluator_sdk.Evaluator` resolves no platform entities, so pass it an inline `Model`. See the [Define and Run Custom Python Metrics](/documentation/evaluate-models/tutorials/define-and-run-custom-python-metrics) tutorial for an end-to-end `ModelRef` + `FilesetRef` submit example.

For evaluating agentic systems, use an `Agent` request target instead of a `Model`. See [Agent Configuration](/documentation/evaluate-models/metrics/agent-configuration).