Agent Configuration

Online evaluations can target an agent instead of a model. An agent is an HTTP endpoint that accepts a request and returns a response, optionally with a trajectory of intermediate steps. Use agents when you want to evaluate an agentic system end to end rather than a standalone LLM endpoint.

Provide a GenericAgent or NemoAgentToolkitAgent as target=.... Agent is a union type alias used in SDK signatures, not an instantiable class; instantiate one of the concrete classes instead. The metric, prompt template, target, dataset rows, and runtime parameters are all passed through the Evaluator plugin SDK call.

Agent Formats

Two agent formats are supported:

Format	Value	Description
Generic	`generic`	Configurable HTTP POST with a Jinja-templated request body and JSONPath extraction for response and trajectory.
NeMo Agent Toolkit	`nemo_agent_toolkit`	Fixed protocol for NeMo Agent Toolkit endpoints.

Initialize the SDK

1 import os
2 
3 from nemo_evaluator.sdk import Evaluator
4 from nemo_platform import NeMoPlatform
5 
6 client = NeMoPlatform(
7     base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
8     workspace="default",
9 )
10 evaluator: Evaluator = client.evaluator  # this object is an Evaluator resource

Managing Secrets for Agent Endpoints

If your agent endpoint requires authentication, configure api_key_secret on the Agent.

For evaluator.submit(...) jobs, api_key_secret must name a NeMo platform secret in the target workspace. See Model API Authentication.

For remote evaluator.submit(...) jobs, create the secret in the platform workspace before submitting the job:

1 client.secrets.create(
2     name="my-agent-api-key",
3     value=os.environ["MY_AGENT_API_KEY"],
4 )

The secret name may be a workspace-local name such as "my-agent-api-key" or a full reference such as "my-workspace/my-agent-api-key" for remote jobs.

Generic Agent

A generic agent is any HTTP endpoint that:

Accepts a POST request with Content-Type: application/json.
Returns a JSON response containing the answer and, optionally, a trajectory.

You control the request shape with body and extract values from the response with JSONPath expressions.

Generic Agent Fields

Field	Required	Type	Description
`url`	Yes	string	Base URL of the agent endpoint.
`name`	Yes	string	Agent name or identifier.
`format`	No	string	`generic` (default) or `nemo_agent_toolkit`.
`api_key_secret`	No	`SecretRef`	API key reference. See Model API Authentication.
`body`	Yes	dict	Jinja template for the request payload. Use `{{ prompt }}`, `{{ messages }}`, or fields from the rendered prompt context.
`response_path`	Yes	string	JSONPath expression to extract the response text.
`trajectory_path`	No	string	JSONPath expression to extract the trajectory.

Run a Generic Agent Evaluation

1 from nemo_evaluator_sdk import ExactMatchMetric, GenericAgent, RunConfigOnline, SecretRef
2 from nemo_evaluator_sdk.enums import AgentFormat
3 
4 metric = ExactMatchMetric(reference="{{item.expected_answer}}")
5 agent = GenericAgent(
6     url="https://my-agent.example.com/invoke",
7     name="qa-agent",
8     format=AgentFormat.GENERIC,
9     api_key_secret=SecretRef(root="my-agent-api-key"),
10     body={"question": "{{ prompt }}"},
11     response_path="$.answer",
12     trajectory_path="$.reasoning_steps",
13 )
14 
15 generic_job = evaluator.submit(
16     metric=metric,
17     dataset=[
18         {"question": "What is the capital of France?", "expected_answer": "Paris"},
19     ],
20     config=RunConfigOnline(parallelism=4, request_timeout=60, max_retries=2),
21     target=agent,
22     prompt_template="Question: {{item.question}}\nAnswer:",
23 )
24 generic_job.wait_until_done()
25 result = generic_job.get_result()
26 for score in result.aggregate_scores.scores:
27     print(f"{score.name}: mean={score.mean}")

Use evaluator.submit(...) with the same argument shape when you want a durable remote job, but set api_key_secret to a platform secret name for the target workspace.

Example Generic Agent Endpoint

Your agent endpoint might look like this:

1 from fastapi import FastAPI
2 from pydantic import BaseModel
3 
4 app = FastAPI()
5 
6 class AgentRequest(BaseModel):
7     question: str
8 
9 class AgentResponse(BaseModel):
10     answer: str
11     reasoning_steps: list[dict]
12 
13 @app.post("/invoke")
14 async def invoke(request: AgentRequest) -> AgentResponse:
15     return AgentResponse(
16         answer="Paris",
17         reasoning_steps=[
18             {"step": "search", "result": "Found relevant documents"},
19             {"step": "synthesize", "result": "Generated answer from context"},
20         ],
21     )

NeMo Agent Toolkit Agent

Use the nemo_agent_toolkit format when evaluating agents built with the NeMo Agent Toolkit. This format uses the NAT streaming protocol:

Sends a POST to {url}/generate/full?filter_steps=none with {"input_message": "<text>"}.
Reads the SSE (Server-Sent Events) stream.
Extracts the value from each SSE data: chunk. NAT emits token-level deltas, so the values are concatenated in order to reconstruct the complete response (concat aggregation).
Returns the reconstructed response.

NeMo Agent Toolkit Fields

Field	Required	Type	Description
`url`	Yes	string	Base URL of the agent endpoint.
`name`	Yes	string	Agent name or identifier.
`format`	Yes	string	Set to `nemo_agent_toolkit`.
`api_key_secret`	No	`SecretRef`	API key reference. See Model API Authentication.

Run a NAT Agent Evaluation

Create the platform secret used by the NAT endpoint:

1 client.secrets.create(
2     name="my-nat-api-key",
3     value=os.environ["MY_NAT_API_KEY"],
4 )

1 from nemo_evaluator_sdk import ExactMatchMetric, NemoAgentToolkitAgent, RunConfigOnline, SecretRef
2 from nemo_evaluator_sdk.enums import AgentFormat
3 
4 metric = ExactMatchMetric(reference="{{item.expected_answer}}")
5 agent = NemoAgentToolkitAgent(
6     url="https://my-nat-agent.example.com",
7     name="nat-research-agent",
8     format=AgentFormat.NEMO_AGENT_TOOLKIT,
9     api_key_secret=SecretRef(root="my-nat-api-key"),
10 )
11 
12 nat_job = evaluator.submit(
13     metric=metric,
14     dataset=[
15         {"question": "What is the capital of France?", "expected_answer": "Paris"},
16     ],
17     config=RunConfigOnline(parallelism=4),
18     target=agent,
19     prompt_template={
20         "messages": [
21             {"role": "user", "content": "{{item.question}}"},
22         ],
23     },
24 )
25 nat_job.wait_until_done()
26 result = nat_job.get_result()

Clean Up

Delete the jobs and secrets created by these examples:

1 client.jobs.delete(generic_job.name, workspace="default")
2 client.jobs.delete(nat_job.name, workspace="default")
3 client.secrets.delete("my-agent-api-key", workspace="default")
4 client.secrets.delete("my-nat-api-key", workspace="default")

Model vs Agent: When to Use Which

Use Case	Use `Model`	Use `Agent`
Evaluate a standalone LLM endpoint	x
Evaluate an agentic system with tool use and multi-step reasoning		x
Evaluate a NeMo Agent Toolkit workflow		x
Evaluate a custom HTTP endpoint with non-standard response format		x
Use a standard chat completions API	x

Online evaluations accept either a model or an agent as the request target, never both.

Model Configuration - Inline model targets for LLM endpoints.
Agentic Evaluation Metrics - Metrics for evaluating agent tool calling, goal accuracy, and trajectory.
LLM-as-a-Judge - Custom judge-based evaluation with flexible scoring criteria.
Bring Your Own Metric - Integrate custom evaluation endpoints.