Targets and Runners

AgentEvaluator().run(target=...) accepts one of three kinds of target. Whatever you pick, it produces trials, and trials are scored the same way — so the same tasks and metrics work against any target (see Agent Evaluation for the model).

At a glance

The union is AgentEvalTarget = Model | Agent | AgentTaskRunner, where Agent = GenericAgent | NemoAgentToolkitAgent.

Model

A chat/completions endpoint evaluated directly on your tasks — a useful baseline (how well does a bare model do before you wrap it in an agent?). The evaluator prompts it with each task’s instruction.

1
from nemo_evaluator_sdk.enums import ModelFormat
2
from nemo_evaluator_sdk.values import Model
3
4
target = Model(url="https://integrate.api.nvidia.com/v1/chat/completions", name="meta/llama-3.1-8b-instruct",
5
               format=ModelFormat.OPEN_AI, api_key_secret="NVIDIA_API_KEY")

For a local run(), api_key_secret names an environment variable in your process; for a submitted job it names a platform secret in the workspace.

Agent (HTTP)

A deployed agent reachable over HTTP. Two variants, both authenticated with api_key_secret — the same credential reference Model uses: for a local run() it names an environment variable, for a submitted job a platform secret. Its value is sent as a bearer token on each request.

GenericAgent

Any JSON endpoint. You define the request with a Jinja body (rendered against the task inputs) and pull the answer out with JSONPath. Full walkthrough: Evaluate a Deployed Agent over HTTP.

NemoAgentToolkitAgent

A NeMo Agent Toolkit endpoint. It speaks NAT’s fixed request/response protocol, so you don’t hand-write a body — point it at the workflow’s URL.

AgentTaskRunner (callable, Harbor, or your own)

The most general target: anything implementing the one-method protocol.

1
from collections.abc import Sequence
2
3
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
4
from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial
5
6
7
class AgentTaskRunner:
8
    async def run_tasks(
9
        self, tasks: Sequence[AgentEvalTask], config: AgentEvalRunConfig | None = None
10
    ) -> Sequence[AgentEvalTrial]: ...

The SDK ships two runners you’ll usually reach for first:

• CallableAgentTaskRunner wraps an async def agent(task) -> str | AgentOutput | TrialDraft. The smallest possible target — no Docker, no HTTP. See the Quickstart. Return a TrialDraft to attach a trajectory or other evidence (see Score by Component).
• HarborAgentTaskRunner runs a Harbor task suite in Docker and scores its verifier reward. See Harbor Task Suite.

Write your own when your agent doesn’t fit those — a bespoke harness, a queue, a replay of stored runs. Return one AgentEvalTrial per task; the evaluator scores them exactly like any other target:

1
from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial, AgentEvalTrialStatus, AgentOutput
2
3
4
class EchoRunner:
5
    async def run_tasks(self, tasks, config=None):
6
        return [
7
            AgentEvalTrial(
8
                id=f"{task.id}:trial",
9
                task_id=task.id,
10
                status=AgentEvalTrialStatus.COMPLETED,
11
                output=AgentOutput(output_text=task.inputs["instruction"]),
12
            )
13
            for task in tasks
14
        ]

Choosing a target

• Just trying the flow, or you already have the agent in Python → CallableAgentTaskRunner.
• The agent is deployed behind HTTP → GenericAgent (any endpoint) or NemoAgentToolkitAgent (a NAT workflow).
• You want a model baseline, no agent → Model.
• You have Harbor task datasets → HarborAgentTaskRunner.
• None of the above fits → implement AgentTaskRunner.

Related