> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Targets and Runners

> Reference for what an agent-eval run can point at — a Model, a deployed Agent over HTTP, or an AgentTaskRunner (a callable, Harbor, or your own) — with each target's key fields and when to use it.

`AgentEvaluator().run(target=...)` accepts one of three kinds of target. Whatever you pick, it produces
**trials**, and trials are scored the same way — so the same tasks and metrics work against any target
(see [Agent Evaluation](/documentation/evaluate-models/agent-eval) for the model).

## At a glance

| Target                    | What it is                            | Extra dependencies          | How-to                                                                                         |
| ------------------------- | ------------------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------- |
| `Model`                   | a chat/completions LLM endpoint       | an inference endpoint + key | —                                                                                              |
| `GenericAgent`            | any HTTP JSON endpoint                | none                        | [Evaluate a Deployed Agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) |
| `NemoAgentToolkitAgent`   | a NeMo Agent Toolkit endpoint         | a running NAT workflow      | [Evaluate a Deployed Agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) |
| `CallableAgentTaskRunner` | an in-process async function          | none                        | [Quickstart](/documentation/evaluate-models/agent-eval/quickstart)                             |
| `HarborAgentTaskRunner`   | a Harbor task suite                   | `harbor` + Docker           | [Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner)                   |
| *your* `AgentTaskRunner`  | anything that turns tasks into trials | up to you                   | *(this page)*                                                                                  |

The union is `AgentEvalTarget = Model | Agent | AgentTaskRunner`, where `Agent = GenericAgent |
NemoAgentToolkitAgent`.

## `Model`

A chat/completions endpoint evaluated directly on your tasks — a useful **baseline** (how well does a
bare model do before you wrap it in an agent?). The evaluator prompts it with each task's
`instruction`.

| Field            | Required | Notes                                                                                                                                    |
| ---------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `url`            | yes      | endpoint URL (e.g. `.../v1/chat/completions` or `.../v1/completions`)                                                                    |
| `name`           | yes      | model identifier, stamped on trials                                                                                                      |
| `format`         | no       | `ModelFormat.NVIDIA_NIM` (default), `ModelFormat.OPEN_AI`, or `ModelFormat.LLAMA_STACK` — serialized as `nim` / `openai` / `llama_stack` |
| `api_key_secret` | no       | credential reference — `workspace/secret_name` or `secret_name`                                                                          |

```python
from nemo_evaluator_sdk.enums import ModelFormat
from nemo_evaluator_sdk.values import Model

target = Model(url="https://integrate.api.nvidia.com/v1/chat/completions", name="meta/llama-3.1-8b-instruct",
               format=ModelFormat.OPEN_AI, api_key_secret="NVIDIA_API_KEY")
```

For a local `run()`, `api_key_secret` names an **environment variable** in your process; for a submitted
job it names a **platform secret** in the workspace.

## `Agent` (HTTP)

A deployed agent reachable over HTTP. Two variants, both authenticated with **`api_key_secret`** — the
same credential reference `Model` uses: for a local `run()` it names an environment variable, for a
submitted job a platform secret. Its value is sent as a bearer token on each request.

### `GenericAgent`

Any JSON endpoint. You define the request with a Jinja `body` (rendered against the task inputs) and
pull the answer out with JSONPath. Full walkthrough:
[Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent).

| Field                  | Required | Notes                                                                                                                                                                                    |
| ---------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                  | yes      | endpoint the evaluator POSTs to                                                                                                                                                          |
| `name`                 | yes      | agent identifier, stamped on trials                                                                                                                                                      |
| `format`               | no       | `AgentFormat.GENERIC` (the default and only value)                                                                                                                                       |
| `body`                 | yes      | Jinja template for the request payload, rendered against task inputs (e.g. `{{ instruction }}`)                                                                                          |
| `response_path`        | yes      | JSONPath selecting the answer from the response                                                                                                                                          |
| `trajectory_path`      | no       | JSONPath selecting a trajectory to score                                                                                                                                                 |
| `api_key_secret`       | no       | credential reference (env var locally, platform secret for a job); its value is sent as a bearer token                                                                                   |
| `stream`               | no       | read JSON SSE `data:` frames instead of a single JSON body (default `false`)                                                                                                             |
| `response_aggregation` | no       | how streamed `data:` frames combine: `last` keeps the final matched value (default; snapshot-per-frame endpoints), `concat` joins matched string values in order (token-delta endpoints) |

### `NemoAgentToolkitAgent`

A [NeMo Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html) endpoint. It
speaks NAT's fixed request/response protocol, so you don't hand-write a `body` — point it at the
workflow's URL.

| Field            | Required | Notes                                                                                                                                                                      |
| ---------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`            | yes      | the NAT workflow endpoint                                                                                                                                                  |
| `name`           | yes      | agent identifier, stamped on trials                                                                                                                                        |
| `format`         | no       | `AgentFormat.NEMO_AGENT_TOOLKIT` (the default and only value)                                                                                                              |
| `nat`            | no       | `NatAgentConfig` — endpoint / query-param / response-path / aggregation overrides; defaults target `/generate/full` and `concat` token-delta frames into the full response |
| `api_key_secret` | no       | credential reference (env var locally, platform secret for a job); its value is sent as a bearer token                                                                     |

## `AgentTaskRunner` (callable, Harbor, or your own)

The most general target: anything implementing the one-method protocol.

```python
from collections.abc import Sequence

from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial


class AgentTaskRunner:
    async def run_tasks(
        self, tasks: Sequence[AgentEvalTask], config: AgentEvalRunConfig | None = None
    ) -> Sequence[AgentEvalTrial]: ...
```

The SDK ships two runners you'll usually reach for first:

* **`CallableAgentTaskRunner`** wraps an `async def agent(task) -> str | AgentOutput | TrialDraft`. The
  smallest possible target — no Docker, no HTTP. See the
  [Quickstart](/documentation/evaluate-models/agent-eval/quickstart). Return a `TrialDraft` to attach
  a trajectory or other evidence (see [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component)).
* **`HarborAgentTaskRunner`** runs a [Harbor](https://www.harborframework.com) task suite in Docker and
  scores its verifier reward. See [Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner).

Write your own when your agent doesn't fit those — a bespoke harness, a queue, a replay of stored runs.
Return one `AgentEvalTrial` per task; the evaluator scores them exactly like any other target:

```python
from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial, AgentEvalTrialStatus, AgentOutput


class EchoRunner:
    async def run_tasks(self, tasks, config=None):
        return [
            AgentEvalTrial(
                id=f"{task.id}:trial",
                task_id=task.id,
                status=AgentEvalTrialStatus.COMPLETED,
                output=AgentOutput(output_text=task.inputs["instruction"]),
            )
            for task in tasks
        ]
```

## Choosing a target

* Just trying the flow, or you already have the agent in Python → **`CallableAgentTaskRunner`**.
* The agent is deployed behind HTTP → **`GenericAgent`** (any endpoint) or **`NemoAgentToolkitAgent`**
  (a NAT workflow).
* You want a model baseline, no agent → **`Model`**.
* You have Harbor task datasets → **`HarborAgentTaskRunner`**.
* None of the above fits → implement **`AgentTaskRunner`**.

## Related

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)

#### [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component)