Manage Tasks & Tasksets

A task is a stored, reusable definition of an agent-eval unit of work: an intent (what the agent should do), the inputs it receives, and the metrics that score it. A taskset is a named grouping of tasks. Both are first-class entities in the Evaluator plugin, addressed by workspace/name and managed through the nemo_platform SDK.

Use stored tasks and tasksets when you want to define an evaluation unit once and reference it across runs, share it across a team, or assemble suites — rather than re-declaring the intent, inputs, and metrics inline every time.

Concepts

Both are addressed by workspace/name. Names are unique within a workspace, limited to 255 characters, and must match ^[\w\-\.]+$.

Every stored task and taskset is versioned. Creating one publishes revision 1; replacing its content publishes the next revision. Earlier revisions stay readable for as long as the task or taskset exists — deleting it removes its revisions with it — which is what lets an evaluation be re-run against exactly the content it ran against the first time.

Initialize the SDK

1
import os
2
3
from nemo_platform import NeMoPlatform
4
5
6
client = NeMoPlatform(
7
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
8
    workspace="default",
9
)
10
11
tasks = client.evaluator.tasks        # EvaluatorTasksResource
12
tasksets = client.evaluator.tasksets  # EvaluatorTasksetsResource

Manage Tasks

A task scores its output with metrics. Store the metric first, then reference it from the task by workspace/name. See Manage Metrics for the metric classes and options.

1
from nemo_evaluator_sdk import ExactMatchMetric
2
3
# Store a metric the task will reference.
4
client.evaluator.metrics.create(
5
    "answer-exact-match",
6
    metric=ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}"),
7
)

Send the task as a TaskInput (the authorable subset of a task) and address it by name on create. Reference the stored metric with a MetricRef (workspace/name, or a bare name resolved against the task’s workspace). The service returns the stored Task.

1
from nemo_evaluator.api.schemas import EvaluatorTaskDefinition, MetadataItem, MetricRef, TaskInput, TaskInputs
2
3
task = TaskInput(
4
    spec=EvaluatorTaskDefinition(
5
        kind="evaluator",
6
        intent="Answer the user's geography question with the capital city.",
7
        inputs=TaskInputs(instruction="What is the capital of France?"),
8
        metrics=[MetricRef("default/answer-exact-match")],
9
    ),
10
    metadata=[MetadataItem(key="suite", value="geography")],
11
)
12
13
stored = tasks.create("capital-of-france", task=task)
14
print(stored.id, stored.spec.metrics)

TaskInput fields

Task kinds

A task is an evaluation unit; its kind says which runner executes it. There are two:

• evaluator — the task’s content is fields you author, scored by platform metrics.
• harbor — the task’s content is a packaged directory of files, scored by Harbor’s own reward.

Both are stored as the same record type, so a taskset can group them and you manage every evaluation unit in one place regardless of which runner executes it.

EvaluatorTaskDefinition (kind="evaluator"):

HarborTaskDefinition (kind="harbor"):

Retrieve, list, and delete

1
# Retrieve one task by name (its current content)
2
task = tasks.retrieve("capital-of-france")
3
print(task.spec.kind, task.revision, task.tags)  # e.g. evaluator 1 {'latest': 1}
4
5
# List tasks in the workspace (paginated)
6
page = tasks.list(page=1, page_size=100, sort="-created_at")
7
for item in page.data:
8
    print(item.name, item.spec.kind)
9
10
# Delete a task (this also removes all of its revisions)
11
tasks.delete("capital-of-france")

sort accepts name, created_at, or updated_at, each optionally prefixed with - for descending order.

Revisions

Publish a new revision

Use replace to publish new content. It creates the task if it does not exist, so a publisher needs no existence check.

1
revised_task = TaskInput(
2
    spec=EvaluatorTaskDefinition(
3
        kind="evaluator",
4
        intent="Answer the user's geography question with the capital city.",
5
        inputs=TaskInputs(instruction="Name the capital city of France."),
6
        metrics=[MetricRef("default/answer-exact-match")],
7
    ),
8
    metadata=[MetadataItem(key="suite", value="geography")],
9
)
10
11
updated = tasks.replace("capital-of-france", task=revised_task)
12
print(updated.revision)  # 2

Submitting content identical to the current revision publishes nothing and returns the existing revision — but any tags in the body are still applied, which is how you tag a revision after the fact.

List revisions

Each entry carries the content_hash used to pin a reference.

1
page = tasks.list_revisions("capital-of-france")
2
for revision in page.data:      # newest first
3
    print(revision.revision, revision.content_hash, revision.tags)

Read a specific revision

Pass a content digest or a tag. This returns the content as published, not the current content.

1
# Revisions come back newest-first and paginated, so index by ordinal rather than by position —
2
# `data[-1]` is only the oldest entry on the page you happen to have fetched.
3
page = tasks.list_revisions("capital-of-france")
4
digest = next(revision.content_hash for revision in page.data if revision.revision == 1)
5
6
original = tasks.retrieve("capital-of-france", revision=digest)  # revision 1, as published
7
current = tasks.retrieve("capital-of-france")                    # revision 2, the current content
8
9
assert original.revision == 1 and current.revision == 2
10
assert original.spec.inputs.instruction != current.spec.inputs.instruction

Tag a revision

A tag is a mutable pointer to a revision — useful for marking one as reviewed or approved after it has been evaluated. Read it back with tag=, the counterpart to revision=:

1
tasks.tag("capital-of-france", tag="blessed", revision=digest)
2
3
blessed = tasks.retrieve("capital-of-france", tag="blessed")

revision= takes a content digest and tag= takes a tag name. They select the same thing two ways, so pass one or the other — passing both raises ValueError.

Manage Tasksets

A taskset references existing tasks by workspace/name. All referenced tasks must already exist when the taskset is created; a missing or duplicate reference is rejected.

Member references are resolved to an exact revision when the taskset is stored. You may submit a bare name (capital-of-france), a tag (capital-of-france#latest), or a digest — what gets stored is always workspace/name#<digest>. This is why a stored taskset keeps naming the same content even after a member task publishes something new, and it is what makes a suite reproducible.

1
from nemo_evaluator.api.schemas import TaskRef, TasksetInput
2
3
taskset = TasksetInput(
4
    description="Geography questions for smoke-testing the agent.",
5
    tasks=[
6
        TaskRef("default/capital-of-france"),
7
        TaskRef("default/capital-of-japan"),
8
    ],
9
)
10
11
stored = tasksets.create("geography-suite", taskset=taskset)
12
print(stored.tasks)
13
# ['default/capital-of-france#a1b2...', 'default/capital-of-japan#c3d4...']

TasksetInput fields

Retrieve, list, and delete

1
taskset = tasksets.retrieve("geography-suite")
2
3
page = tasksets.list(page=1, page_size=100, sort="name")
4
for item in page.data:
5
    print(item.name, len(item.tasks))
6
7
tasksets.delete("geography-suite")

Tasksets carry the same revision surface as tasks — replace, list_revisions, tag, and retrieve(revision=...) / retrieve(tag=...):

1
# Re-resolving membership after a member task published new content cuts a new revision.
2
tasksets.replace("geography-suite", taskset=taskset)
3
4
for revision in tasksets.list_revisions("geography-suite").data:
5
    print(revision.revision, revision.content_hash)

Deleting a taskset does not delete its member tasks — a taskset only holds references.

Run an evaluation over a taskset

An agent evaluation is submitted with an AgentEvalInputSpec, whose tasks field is either an inline list of tasks or a reference to a stored taskset. Referencing a taskset lets you keep the task definitions in one place and evaluate the whole set by name, instead of inlining every task on each run.

1
from nemo_evaluator.api.schemas import TasksetRef
2
from nemo_evaluator.jobs.agent_spec import AgentEvalInputSpec, ModelTarget
3
from nemo_evaluator_sdk.values import Model
4
from nemo_evaluator_sdk.enums import ModelFormat
5
6
# Instead of inlining AgentEvalTaskInput objects, point `tasks` at a stored taskset.
7
input_spec = AgentEvalInputSpec(
8
    tasks=TasksetRef("default/geography-suite"),
9
    target=ModelTarget(
10
        model=Model(url="https://integrate.api.nvidia.com/v1", name="meta/llama-3.3-70b-instruct", format=ModelFormat.OPEN_AI),
11
    ),
12
)

When the job runs, the taskset reference is resolved like this:

• The taskset revision the ref names is loaded — the current one unless the ref pins a revision (see Pin the taskset itself).
• Each member of that revision is loaded at the revision pinned in the taskset, not the task’s current tip.
• Metric references on those members are hydrated into runnable metrics, the same as for inline tasks.
• Re-running the same taskset therefore evaluates the same content, even if a member has been republished since.

# What is stored on the taskset (digests fixed at create/replace time)
geography-suite@1
  └─ tasks:
       - default/capital-of-france#aaa111...   ← revision 1 content
       - default/capital-of-japan#bbb222...
# Later: the member task publishes new content
capital-of-france tip → revision 2 (#ccc333...)
# Re-grade still uses the taskset pins, not the tip
JobRun A  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...
JobRun B  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...   ← same content

To evaluate the new task content, publish a new taskset revision (for example tasksets.replace(...)) so membership is re-resolved to the newer digests.

Submit this spec as the agent-evaluate job input; see Agent Evaluation for the full run, target, and results flow.

The inline form remains available for one-off tasks — swap tasks=TasksetRef(...) for tasks=[AgentEvalTaskInput(...), ...].

Pin the taskset itself

A bare TasksetRef expands the taskset’s current revision, so it follows the suite forward every time the taskset is republished. Add a #<tag-or-digest> fragment to pin the grouping too:

1
# The digest lives on the revision, not on the taskset record — revisions come back newest first.
2
current = tasksets.list_revisions("geography-suite").data[0]
3
4
# Follows the suite forward — the next `replace` is picked up on the next run.
5
tracking = TasksetRef("default/geography-suite")
6
7
# Frozen: this exact membership and content, regardless of later `replace` calls.
8
pinned = TasksetRef(f"default/geography-suite#{current.content_hash}")
9
10
# A tag works the same way, and can be moved deliberately when you bless a new suite.
11
blessed = TasksetRef("default/geography-suite#blessed")

What each form is stable against:

Every taskset revision pins its members by digest, so neither form is disturbed when a member task publishes new content on its own — that is the guarantee stored membership buys you.

The two differ on replace. A bare ref tracks the taskset’s own revisions, and members are re-resolved on every write — so a replace can change both which tasks are named and the content they resolve to, even when the submitted member names were identical. Pin the taskset when a benchmark number has to stay comparable across that.

A fragment that no longer resolves fails the evaluation rather than falling back to the current revision.

Async usage

AsyncNeMoPlatform exposes the same surface; await each call.

1
import asyncio
2
3
from nemo_platform import AsyncNeMoPlatform
4
5
6
async def main() -> None:
7
    client = AsyncNeMoPlatform(base_url="http://localhost:8080", workspace="default")
8
    page = await client.evaluator.tasks.list()
9
    for item in page.data:
10
        print(item.name)
11
12
13
asyncio.run(main())

Workspaces and projects

Every method accepts an optional workspace argument that overrides the client’s default workspace. On create, an optional project argument associates the task or taskset with a project. When you omit workspace, the client’s configured workspace is used.

REST API

The SDK resources are a thin client over the Evaluator plugin REST API, mounted under /apis/evaluator/v2/workspaces/{workspace}:

Edge Cases

PUT distinguishes its two outcomes by status: 201 when a new revision was published, and 200 when the submitted content was already the current revision and nothing was cut. The rest:

Related Topics

• Manage Metrics - Define and reuse the metrics that score a task
• SDK Resources - Run and submit evaluations through the Evaluator plugin
• Agent Evaluation - How agent-eval tasks are executed and scored