Manage Tasks & Tasksets

A task is a stored, reusable definition of an agent-eval unit of work: an intent (what the agent should do), the inputs it receives, and the metrics that score it. A taskset is a named grouping of tasks. Both are first-class entities in the Evaluator plugin, addressed by workspace/name and managed through the nemo_platform SDK.

Use stored tasks and tasksets when you want to define an evaluation unit once and reference it across runs, share it across a team, or assemble suites — rather than re-declaring the intent, inputs, and metrics inline every time.

Concepts

Both are addressed by workspace/name. Names are unique within a workspace, limited to 255 characters, and must match ^[\w\-\.]+$.

Initialize the SDK

1
import os
2
3
from nemo_platform import NeMoPlatform
4
5
6
client = NeMoPlatform(
7
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
8
    workspace="default",
9
)
10
11
tasks = client.evaluator.tasks        # EvaluatorTasksResource
12
tasksets = client.evaluator.tasksets  # EvaluatorTasksetsResource

Manage Tasks

A task scores its output with metrics. Store the metric first, then reference it from the task by workspace/name. See Manage Metrics for the metric classes and options.

1
from nemo_evaluator_sdk import ExactMatchMetric
2
3
# Store a metric the task will reference.
4
client.evaluator.metrics.create(
5
    "answer-exact-match",
6
    metric=ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}"),
7
)

Send the task as a TaskInput (the authorable subset of a task) and address it by name on create. Reference the stored metric with a MetricRef (workspace/name, or a bare name resolved against the task’s workspace). The service returns the stored Task.

1
from nemo_evaluator.api.schemas import MetadataItem, MetricRef, TaskInput, TaskInputs
2
3
task = TaskInput(
4
    intent="Answer the user's geography question with the capital city.",
5
    inputs=TaskInputs(instruction="What is the capital of France?"),
6
    metrics=[MetricRef("default/answer-exact-match")],
7
    metadata=[MetadataItem(key="suite", value="geography")],
8
)
9
10
stored = tasks.create("capital-of-france", task=task)
11
print(stored.id, stored.metrics)

TaskInput fields

Retrieve, list, and delete

1
# Retrieve one task by name
2
task = tasks.retrieve("capital-of-france")
3
4
# List tasks in the workspace (paginated)
5
page = tasks.list(page=1, page_size=100, sort="-created_at")
6
for item in page.data:
7
    print(item.name, item.intent)
8
9
# Delete a task
10
tasks.delete("capital-of-france")

sort accepts name, created_at, or updated_at, each optionally prefixed with - for descending order.

Manage Tasksets

A taskset references existing tasks by workspace/name. All referenced tasks must already exist when the taskset is created; a missing or duplicate reference is rejected.

1
from nemo_evaluator.api.schemas import TaskRef, TasksetInput
2
3
taskset = TasksetInput(
4
    description="Geography questions for smoke-testing the agent.",
5
    tasks=[
6
        TaskRef("default/capital-of-france"),
7
        TaskRef("default/capital-of-japan"),
8
    ],
9
)
10
11
stored = tasksets.create("geography-suite", taskset=taskset)
12
print(stored.tasks)

TasksetInput fields

Retrieve, list, and delete

1
taskset = tasksets.retrieve("geography-suite")
2
3
page = tasksets.list(page=1, page_size=100, sort="name")
4
for item in page.data:
5
    print(item.name, len(item.tasks))
6
7
tasksets.delete("geography-suite")

Deleting a taskset does not delete its member tasks — a taskset only holds references.

Run an evaluation over a taskset

An agent evaluation is submitted with an AgentEvalInputSpec, whose tasks field is either an inline list of tasks or a reference to a stored taskset. Referencing a taskset lets you keep the task definitions in one place and evaluate the whole set by name, instead of inlining every task on each run.

1
from nemo_evaluator.api.schemas import TasksetRef
2
from nemo_evaluator.jobs.agent_spec import AgentEvalInputSpec, ModelTarget
3
from nemo_evaluator_sdk.values import Model
4
from nemo_evaluator_sdk.enums import ModelFormat
5
6
# Instead of inlining AgentEvalTaskInput objects, point `tasks` at a stored taskset.
7
input_spec = AgentEvalInputSpec(
8
    tasks=TasksetRef("default/geography-suite"),
9
    target=ModelTarget(
10
        model=Model(url="https://integrate.api.nvidia.com/v1", name="meta/llama-3.3-70b-instruct", format=ModelFormat.OPEN_AI),
11
    ),
12
)

When the job runs, the taskset reference is resolved: its member tasks are loaded, and each task’s stored metric references are hydrated into runnable metrics — exactly as if you had inlined them. The same spec is submitted as the agent-evaluate job input; see Agent Evaluation for the full run, target, and results flow.

The inline form remains available for one-off tasks — swap tasks=TasksetRef(...) for tasks=[AgentEvalTaskInput(...), ...].

Async usage

AsyncNeMoPlatform exposes the same surface; await each call.

1
import asyncio
2
3
from nemo_platform import AsyncNeMoPlatform
4
5
6
async def main() -> None:
7
    client = AsyncNeMoPlatform(base_url="http://localhost:8080", workspace="default")
8
    page = await client.evaluator.tasks.list()
9
    for item in page.data:
10
        print(item.name)
11
12
13
asyncio.run(main())

Workspaces and projects

Every method accepts an optional workspace argument that overrides the client’s default workspace. On create, an optional project argument associates the task or taskset with a project. When you omit workspace, the client’s configured workspace is used.

REST API

The SDK resources are a thin client over the Evaluator plugin REST API, mounted under /apis/evaluator/v2/workspaces/{workspace}:

Creating a name that already exists returns 409. An invalid metric reference (task) or a missing or duplicate task reference (taskset) returns 422. Retrieving or deleting a name that does not exist returns 404.

Related Topics

• Manage Metrics - Define and reuse the metrics that score a task
• SDK Resources - Run and submit evaluations through the Evaluator plugin
• Agent Evaluation - How agent-eval tasks are executed and scored