Manage Tasks & Tasksets

A task is a stored, reusable definition of an agent-eval unit of work: an intent (what the agent should do), the inputs it receives, and the metrics that score it. A taskset is a named grouping of tasks. Both are first-class entities in the Evals plugin, addressed by workspace/name and managed through the nemo_helix SDK.

Use stored tasks and tasksets when you want to define an evaluation unit once and reference it across runs, share it across a team, or assemble suites — rather than re-declaring the intent, inputs, and metrics inline every time.

Concepts

Both are addressed by workspace/name. Names are unique within a workspace, limited to 255 characters, and must match ^[\w\-\.]+$.

Every stored task and taskset is versioned. Creating one publishes revision 1; replacing its content publishes the next revision. Earlier revisions stay readable for as long as the task or taskset exists — deleting it removes its revisions with it — which is what lets an evaluation be re-run against exactly the content it ran against the first time.

Initialize the SDK

from nemo_evals.sdk import Evaluator
from nemo_helix_plugin.client.client import NemoClient
client = NemoClient.from_config()
evaluator = Evaluator.from_client(client)
tasks = evaluator.tasks        # EvaluatorTasksResource
tasksets = evaluator.tasksets  # EvaluatorTasksetsResource

Manage Tasks

A task scores its output with metrics. Store the metric first, then reference it from the task by workspace/name. See Manage Metrics for the metric classes and options.

from nhx_evals_sdk import ExactMatchMetric
# Store a metric the task will reference.
evaluator.metrics.create(
    "answer-exact-match",
    metric=ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}"),
)

Send the task as a TaskInput (the authorable subset of a task) and address it by name on create. Reference the stored metric with a MetricRef (workspace/name, or a bare name resolved against the task’s workspace). The service returns the stored Task.

from nemo_evals.api.schemas import EvaluatorTaskDefinition, MetadataItem, MetricRef, TaskInput, TaskInputs
france_task = TaskInput(
    spec=EvaluatorTaskDefinition(
        kind="evaluator",
        intent="Answer the user's geography question with the capital city.",
        inputs=TaskInputs(instruction="What is the capital of France?"),
        reference={"expected": "Paris"},
        metrics=[MetricRef("default/answer-exact-match")],
    ),
    metadata=[MetadataItem(key="suite", value="geography")],
)
japan_task = TaskInput(
    spec=EvaluatorTaskDefinition(
        kind="evaluator",
        intent="Answer the user's geography question with the capital city.",
        inputs=TaskInputs(instruction="What is the capital of Japan?"),
        reference={"expected": "Tokyo"},
        metrics=[MetricRef("default/answer-exact-match")],
    ),
    metadata=[MetadataItem(key="suite", value="geography")],
)
stored_france = tasks.create("capital-of-france", task=france_task)
stored_japan = tasks.create("capital-of-japan", task=japan_task)
print(stored_france.id, stored_japan.id, stored_france.spec.metrics)

TaskInput fields

Task kinds

A task is an evaluation unit; its kind says which runner executes it. There are two:

• evaluator — the task’s content is fields you author, scored by platform metrics.
• harbor — the task’s content is a self-contained Fileset archive, scored by Harbor’s own reward.

Both are stored as the same record type, so a taskset can group them and you manage every evaluation unit in one place regardless of which runner executes it.

EvaluatorTaskDefinition (kind="evaluator"):

HarborTaskDefinition (kind="harbor"):

Harbor scoring always includes HarborRewardMetric; do not include it again in metrics. Custom metrics and views are part of the task revision, so changing them creates a new revision while reusing the archive. See custom Harbor scoring for a registration and submission example.

See Upload and Register Harbor Dataset for details about managing Harbor datasets in NeMo Helix.

Retrieve and list

# Retrieve one task by name (its current content)
task = tasks.retrieve("capital-of-france")
print(task.spec.kind, task.revision, task.tags)  # e.g. evaluator 1 {'latest': 1}
# List tasks in the workspace (paginated)
page = tasks.list(page=1, page_size=100, sort="-created_at")
for item in page.data:
    print(item.name, item.spec.kind)

sort accepts name, created_at, or updated_at, each optionally prefixed with - for descending order.

Revisions

Publish a new revision

Use replace to publish new content. It creates the task if it does not exist, so a publisher needs no existence check.

revised_task = TaskInput(
    spec=EvaluatorTaskDefinition(
        kind="evaluator",
        intent="Answer the user's geography question with the capital city.",
        inputs=TaskInputs(instruction="Name the capital city of France."),
        metrics=[MetricRef("default/answer-exact-match")],
    ),
    metadata=[MetadataItem(key="suite", value="geography")],
)
updated = tasks.replace("capital-of-france", task=revised_task)
print(updated.revision)  # 2

Submitting content identical to the current revision publishes nothing and returns the existing revision — but any tags in the body are still applied, which is how you tag a revision after the fact.

List revisions

Each entry carries the content_hash used to pin a reference.

page = tasks.list_revisions("capital-of-france")
for revision in page.data:      # newest first
    print(revision.revision, revision.content_hash, revision.tags)

Read a specific revision

Pass a content digest or a tag. This returns the content as published, not the current content.

# Revisions come back newest-first and paginated, so index by ordinal rather than by position —
# `data[-1]` is only the oldest entry on the page you happen to have fetched.
page = tasks.list_revisions("capital-of-france")
digest = next(revision.content_hash for revision in page.data if revision.revision == 1)
original = tasks.retrieve("capital-of-france", revision=digest)  # revision 1, as published
current = tasks.retrieve("capital-of-france")                    # revision 2, the current content
assert original.revision == 1 and current.revision == 2
assert original.spec.inputs.instruction != current.spec.inputs.instruction

Tag a revision

A tag is a mutable pointer to a revision — useful for marking one as reviewed or approved after it has been evaluated. Read it back with tag=, the counterpart to revision=:

tasks.tag("capital-of-france", tag="blessed", revision=digest)
blessed = tasks.retrieve("capital-of-france", tag="blessed")

revision= takes a content digest and tag= takes a tag name. They select the same thing two ways, so pass one or the other — passing both raises ValueError.

Manage Tasksets

A taskset references existing tasks by workspace/name. All referenced tasks must already exist when the taskset is created; a missing or duplicate reference is rejected.

Member references are resolved to an exact revision when the taskset is stored. You may submit a bare name (capital-of-france), a tag (capital-of-france#latest), or a digest — what gets stored is always workspace/name#<digest>. This is why a stored taskset keeps naming the same content even after a member task publishes something new, and it is what makes a suite reproducible.

from nemo_evals.api.schemas import TaskRef, TasksetInput
taskset_input = TasksetInput(
    description="Geography questions for validating the agent.",
    tasks=[
        TaskRef("default/capital-of-france"),
        TaskRef("default/capital-of-japan"),
    ],
)
stored_taskset = tasksets.create("geography-suite", taskset=taskset_input)
print(stored_taskset.tasks)
# ['default/capital-of-france#a1b2...', 'default/capital-of-japan#c3d4...']

TasksetInput fields

Retrieve and list

stored_taskset = tasksets.retrieve("geography-suite")
page = tasksets.list(page=1, page_size=100, sort="name")
for item in page.data:
    print(item.name, len(item.tasks))

Tasksets carry the same revision surface as tasks — replace, list_revisions, tag, and retrieve(revision=...) / retrieve(tag=...):

# Re-resolving membership after a member task published new content cuts a new revision.
tasksets.replace("geography-suite", taskset=taskset_input)
for revision in tasksets.list_revisions("geography-suite").data:
    print(revision.revision, revision.content_hash)

Deleting a taskset does not delete its member tasks — a taskset only holds references.

Filter tasks and tasksets

Both list() methods take filter arguments, so you can find tasks and tasksets by what they contain rather than paging through everything in a workspace. Arguments combine with AND.

# Tasks annotated with suite=geography
page = tasks.list(metadata={"suite": "geography"})
# Evaluator tasks that use a stored metric, and carry the "blessed" tag
page = tasks.list(kind="evaluator", metric="answer-exact-match", tag="blessed")
# Case-insensitive substring search on an evaluator task's intent
page = tasks.list(intent_contains="capital city")
# Tasksets that include capital-of-france at any revision
page = tasksets.list(task="capital-of-france")
# Case-insensitive substring search on a taskset's description
page = tasksets.list(description_contains="geography")

The async resources take the same arguments.

Keep these rules in mind:

• Metadata matches one annotation. metadata={"suite": "geography"} matches only an annotation whose key is suite and whose value is geography. A key and value on different annotations don’t match.
• Bare references use the request’s workspace. metric="answer-exact-match" and task="capital-of-france" match default/... when listing in default, as references do everywhere else.
• Members are matched by name or by digest, never by tag. Because stored taskset members are pinned to a digest, a #tag fragment can never match and returns 400. Use workspace/name to match any revision.

Filter over REST

The REST endpoints accept the same filters through the standard filter parameter. See Filtering for the bracket, text, and JSON forms.

Bracket values arrive as strings. To match a non-string metadata value, such as a number or a boolean, use a JSON filter such as ?filter={"metadata.verified": true}. The SDK always sends JSON.

Run an evaluation over a taskset

An agent evaluation is submitted with an AgentEvalInputSpec, whose tasks field is either an inline list of tasks or a reference to a stored taskset. Referencing a taskset lets you keep the task definitions in one place and evaluate the whole set by name, instead of inlining every task on each run.

from nemo_evals.api.schemas import TasksetRef
from nemo_evals.jobs.agent_spec import AgentEvalInputSpec, ModelTarget
from nhx_evals_sdk.values import Model
# Instead of inlining AgentEvalTaskInput objects, point `tasks` at a stored taskset.
input_spec = AgentEvalInputSpec(
    tasks=TasksetRef("default/geography-suite"),
    target=ModelTarget(
        model=Model(url="https://integrate.api.nvidia.com/v1", name="nvidia/nemotron-3.5-lightning-30b-a3b"),
    ),
)

When the job runs, the taskset reference is resolved like this:

• The taskset revision the ref names is loaded — the current one unless the ref pins a revision (see Pin the taskset itself).
• Each member of that revision is loaded at the revision pinned in the taskset, not the task’s current tip.
• Metric references on those members are hydrated into runnable metrics, the same as for inline tasks.
• Re-running the same taskset therefore evaluates the same content, even if a member has been republished since.

# What is stored on the taskset (digests fixed at create/replace time)
geography-suite@1
  └─ tasks:
       - default/capital-of-france#aaa111...   ← revision 1 content
       - default/capital-of-japan#bbb222...
# Later: the member task publishes new content
capital-of-france tip → revision 2 (#ccc333...)
# Re-grade still uses the taskset pins, not the tip
JobRun A  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...
JobRun B  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...   ← same content

To evaluate the new task content, publish a new taskset revision (for example tasksets.replace(...)) so membership is re-resolved to the newer digests.

Submit this spec as the agent-evaluate job input; see Agent Evaluation for the full run, target, and results flow.

The inline form remains available for one-off tasks — swap tasks=TasksetRef(...) for tasks=[AgentEvalTaskInput(...), ...].

Pin the taskset itself

A bare TasksetRef expands the taskset’s current revision, so it follows the suite forward every time the taskset is republished. Add a #<tag-or-digest> fragment to pin the grouping too:

# The digest lives on the revision, not on the taskset record — revisions come back newest first.
current = tasksets.list_revisions("geography-suite").data[0]
# Follows the suite forward — the next `replace` is picked up on the next run.
tracking = TasksetRef("default/geography-suite")
# Frozen: this exact membership and content, regardless of later `replace` calls.
pinned = TasksetRef(f"default/geography-suite#{current.content_hash}")
# A tag works the same way, and can be moved deliberately when you bless a new suite.
blessed = TasksetRef("default/geography-suite#blessed")

What each form is stable against:

Every taskset revision pins its members by digest, so neither form is disturbed when a member task publishes new content on its own — that is the guarantee stored membership buys you.

The two differ on replace. A bare ref tracks the taskset’s own revisions, and members are re-resolved on every write — so a replace can change both which tasks are named and the content they resolve to, even when the submitted member names were identical. Pin the taskset when a benchmark number has to stay comparable across that.

A fragment that no longer resolves fails the evaluation rather than falling back to the current revision.

Clean Up

Delete the taskset before its member tasks, then remove the metric used by the tasks:

tasksets.delete("geography-suite")
tasks.delete("capital-of-france")
tasks.delete("capital-of-japan")
evaluator.metrics.delete("answer-exact-match")

Async usage

AsyncEvaluator exposes the same surface; await each call.

import asyncio
from nemo_evals.sdk import AsyncEvaluator
from nemo_helix_plugin.client.client import AsyncNemoClient
async def main() -> None:
    async with AsyncNemoClient(base_url="http://localhost:8080", workspace="default") as async_client:
        evaluator = AsyncEvaluator.from_client(async_client)
        page = await evaluator.tasks.list()
    for item in page.data:
        print(item.name)
asyncio.run(main())

Workspaces and projects

Every method accepts an optional workspace argument that overrides the client’s default workspace. On create, an optional project argument associates the task or taskset with a project. When you omit workspace, the client’s configured workspace is used.

REST API

The SDK resources are a thin client over the Evals plugin REST API, mounted under /apis/evals/v2/workspaces/{workspace}:

Edge Cases

PUT distinguishes its two outcomes by status: 201 when a new revision was published, and 200 when the submitted content was already the current revision and nothing was cut. The rest:

Related Topics

• Manage Metrics - Define and reuse the metrics that score a task
• SDK Resources - Run and submit evaluations through the Evals plugin
• Agent Evaluation - How agent-eval tasks are executed and scored