> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server. # Manage Tasks & Tasksets A **task** is a stored, reusable definition of an agent-eval unit of work: an intent (what the agent should do), the inputs it receives, and the metrics that score it. A **taskset** is a named grouping of tasks. Both are first-class entities in the Evaluator plugin, addressed by `workspace/name` and managed through the `nemo_platform` SDK. Use stored tasks and tasksets when you want to define an evaluation unit once and reference it across runs, share it across a team, or assemble suites — rather than re-declaring the intent, inputs, and metrics inline every time. ## Concepts | Concept | What it is | Members | | ------------ | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Task** | A reusable agent-eval unit: `intent`, `inputs`, and the `metrics` that score it. | References the metrics that score it. | | **Taskset** | A flexible grouping of tasks with a description and metadata. | References member tasks by `workspace/name`, each pinned to an exact revision. Membership is a **set** — order is not significant and duplicate references are rejected. | | **Revision** | An immutable published snapshot of a task's or taskset's content, addressed by a content digest. | Belongs to the task or taskset it snapshots. | Both are addressed by `workspace/name`. Names are unique within a workspace, limited to 255 characters, and must match `^[\w\-\.]+$`. Every stored task and taskset is **versioned**. Creating one publishes revision 1; replacing its content publishes the next revision. Earlier revisions stay readable for as long as the task or taskset exists — deleting it removes its revisions with it — which is what lets an evaluation be re-run against exactly the content it ran against the first time. Publishing is **idempotent**. Replacing a task with content identical to its current revision publishes nothing and returns the existing revision — so a pipeline can re-submit the same definition freely without accumulating versions. ## Initialize the SDK ```python import os from nemo_platform import NeMoPlatform client = NeMoPlatform( base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), workspace="default", ) tasks = client.evaluator.tasks # EvaluatorTasksResource tasksets = client.evaluator.tasksets # EvaluatorTasksetsResource ``` ## Manage Tasks A task scores its output with metrics. Store the metric first, then reference it from the task by `workspace/name`. See [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) for the metric classes and options. ```python from nemo_evaluator_sdk import ExactMatchMetric # Store a metric the task will reference. client.evaluator.metrics.create( "answer-exact-match", metric=ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}"), ) ``` Send the task as a `TaskInput` (the authorable subset of a task) and address it by name on create. Reference the stored metric with a `MetricRef` (`workspace/name`, or a bare `name` resolved against the task's workspace). The service returns the stored `Task`. ```python from nemo_evaluator.api.schemas import EvaluatorTaskDefinition, MetadataItem, MetricRef, TaskInput, TaskInputs france_task = TaskInput( spec=EvaluatorTaskDefinition( kind="evaluator", intent="Answer the user's geography question with the capital city.", inputs=TaskInputs(instruction="What is the capital of France?"), reference={"expected": "Paris"}, metrics=[MetricRef("default/answer-exact-match")], ), metadata=[MetadataItem(key="suite", value="geography")], ) japan_task = TaskInput( spec=EvaluatorTaskDefinition( kind="evaluator", intent="Answer the user's geography question with the capital city.", inputs=TaskInputs(instruction="What is the capital of Japan?"), reference={"expected": "Tokyo"}, metrics=[MetricRef("default/answer-exact-match")], ), metadata=[MetadataItem(key="suite", value="geography")], ) stored_france = tasks.create("capital-of-france", task=france_task) stored_japan = tasks.create("capital-of-japan", task=japan_task) print(stored_france.id, stored_japan.id, stored_france.spec.metrics) ``` ### `TaskInput` fields | Field | Type | Required | Description | | ---------- | -------------------- | -------- | --------------------------------------------------------------------------------------------- | | `spec` | `TaskDefinition` | Yes | The task's content, discriminated by `kind` — see below. | | `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. | | `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. | ### Task kinds A task is an evaluation unit; its `kind` says which runner executes it. There are two: * `evaluator` — the task's content is fields you author, scored by platform metrics. * `harbor` — the task's content is a packaged directory of files, scored by Harbor's own reward. Both are stored as the same record type, so a taskset can group them and you manage every evaluation unit in one place regardless of which runner executes it. `EvaluatorTaskDefinition` (`kind="evaluator"`): | Field | Type | Required | Description | | ----------- | ------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `intent` | `str` | Yes | Human-readable description of the desired agent behavior. | | `inputs` | `TaskInputs` | No | The task's recognized input fields. `instruction` is the agent's prompt; it falls back to `intent` when unset. | | `reference` | `dict[str, Any]` | No | Grader-only ground truth (held-out tests, expected outputs, rubric data). Surfaced to metrics but never seeded into the agent's workspace or shown to the agent. Held out from the *agent*, not from the API. | | `metrics` | `list[MetricRefOrInline]` | No | The metrics that score the task, as `MetricRef` references (`workspace/name`) to stored metrics. Pre-built inline metric bundles (`MetricInline`) are also accepted and are normalized to stored metrics on create. | | `views` | `dict[str, SemanticView]` | No | Optional reporting views mapping metric outputs into named semantic scores. | `HarborTaskDefinition` (`kind="harbor"`): | Field | Type | Required | Description | | ---------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `archive_ref` | `str` | Yes | Files reference to the task's packaged directory (`workspace/fileset#path`). One fileset per task, so a task shared by several tasksets is stored once. | | `archive_digest` | `str` | Yes | Content hash Harbor computed over the task directory. | | `instruction` | `str` | No | The task's instruction text, when it has one. | | `config` | `dict` | No | Harbor's own task configuration (verifier, agent, environment, steps), stored as published. | Storing a Harbor task is supported; **running one from storage is not yet**. A taskset may group both kinds, but expanding a `harbor` member is rejected with `422` before the run starts, whatever target you submit against. Harbor evaluations continue to run through the existing dataset-driven path. A stored task holds **metric references only**. Any inline metric bundle you pass on create is stored as a content-addressed *derived* metric, and the task record is normalized to reference it. This is why `stored.spec.metrics` always comes back as a list of `MetricRef` references. ### Retrieve and list ```python # Retrieve one task by name (its current content) task = tasks.retrieve("capital-of-france") print(task.spec.kind, task.revision, task.tags) # e.g. evaluator 1 {'latest': 1} # List tasks in the workspace (paginated) page = tasks.list(page=1, page_size=100, sort="-created_at") for item in page.data: print(item.name, item.spec.kind) ``` `sort` accepts `name`, `created_at`, or `updated_at`, each optionally prefixed with `-` for descending order. ## Revisions ### Publish a new revision Use `replace` to publish new content. It creates the task if it does not exist, so a publisher needs no existence check. ```python revised_task = TaskInput( spec=EvaluatorTaskDefinition( kind="evaluator", intent="Answer the user's geography question with the capital city.", inputs=TaskInputs(instruction="Name the capital city of France."), metrics=[MetricRef("default/answer-exact-match")], ), metadata=[MetadataItem(key="suite", value="geography")], ) updated = tasks.replace("capital-of-france", task=revised_task) print(updated.revision) # 2 ``` Submitting content identical to the current revision publishes nothing and returns the existing revision — but any tags in the body are still applied, which is how you tag a revision after the fact. ### List revisions Each entry carries the `content_hash` used to pin a reference. ```python page = tasks.list_revisions("capital-of-france") for revision in page.data: # newest first print(revision.revision, revision.content_hash, revision.tags) ``` ### Read a specific revision Pass a content digest or a tag. This returns the content **as published**, not the current content. ```python # Revisions come back newest-first and paginated, so index by ordinal rather than by position — # `data[-1]` is only the oldest entry on the page you happen to have fetched. page = tasks.list_revisions("capital-of-france") digest = next(revision.content_hash for revision in page.data if revision.revision == 1) original = tasks.retrieve("capital-of-france", revision=digest) # revision 1, as published current = tasks.retrieve("capital-of-france") # revision 2, the current content assert original.revision == 1 and current.revision == 2 assert original.spec.inputs.instruction != current.spec.inputs.instruction ``` ### Tag a revision A tag is a mutable pointer to a revision — useful for marking one as reviewed or approved after it has been evaluated. Read it back with `tag=`, the counterpart to `revision=`: ```python tasks.tag("capital-of-france", tag="blessed", revision=digest) blessed = tasks.retrieve("capital-of-france", tag="blessed") ``` `revision=` takes a content digest and `tag=` takes a tag name. They select the same thing two ways, so pass one or the other — passing both raises `ValueError`. A tag names exactly one revision. Re-tagging moves the pointer rather than adding a second one, so `retrieve(tag=...)` always resolves to a single revision — there is no way for two revisions to share a tag. `latest` is managed automatically and always names the most recently published revision; it cannot be moved by hand. A tag name may not be empty, and may not look like a content digest (64 hexadecimal characters) — such a tag could be stored but never resolved, because a digest-shaped reference is looked up as a digest rather than as a tag. ## Manage Tasksets A taskset references existing tasks by `workspace/name`. All referenced tasks must already exist when the taskset is created; a missing or duplicate reference is rejected. Member references are **resolved to an exact revision when the taskset is stored**. You may submit a bare name (`capital-of-france`), a tag (`capital-of-france#latest`), or a digest — what gets stored is always `workspace/name#`. This is why a stored taskset keeps naming the same content even after a member task publishes something new, and it is what makes a suite reproducible. ```python from nemo_evaluator.api.schemas import TaskRef, TasksetInput taskset_input = TasksetInput( description="Geography questions for validating the agent.", tasks=[ TaskRef("default/capital-of-france"), TaskRef("default/capital-of-japan"), ], ) stored_taskset = tasksets.create("geography-suite", taskset=taskset_input) print(stored_taskset.tasks) # ['default/capital-of-france#a1b2...', 'default/capital-of-japan#c3d4...'] ``` ### `TasksetInput` fields | Field | Type | Required | Description | | ------------- | -------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `description` | `str` | No | Human-readable description of the grouping. | | `tasks` | `list[TaskRef]` | No | References to member tasks (`workspace/name`, or bare `name` within the same workspace), optionally pinned with `#`. Each is resolved to an exact digest when stored. Set semantics — duplicates rejected. | | `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. | | `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. | ### Retrieve and list ```python stored_taskset = tasksets.retrieve("geography-suite") page = tasksets.list(page=1, page_size=100, sort="name") for item in page.data: print(item.name, len(item.tasks)) ``` Tasksets carry the same revision surface as tasks — `replace`, `list_revisions`, `tag`, and `retrieve(revision=...)` / `retrieve(tag=...)`: ```python # Re-resolving membership after a member task published new content cuts a new revision. tasksets.replace("geography-suite", taskset=taskset_input) for revision in tasksets.list_revisions("geography-suite").data: print(revision.revision, revision.content_hash) ``` Re-submitting the *same* member names can still publish a new revision. Members are re-resolved on every write, so if a member task published in the meantime the grouping now names different content and genuinely differs. A taskset's identity is the exact revisions it names, not the names alone. Member *order*, by contrast, is not part of that identity: membership is a set, so it is stored in a canonical order and reordering the same members publishes nothing. Deleting a taskset does not delete its member tasks — a taskset only holds references. ## Run an evaluation over a taskset An agent evaluation is submitted with an `AgentEvalInputSpec`, whose `tasks` field is either an inline list of tasks or a **reference to a stored taskset**. Referencing a taskset lets you keep the task definitions in one place and evaluate the whole set by name, instead of inlining every task on each run. ```python from nemo_evaluator.api.schemas import TasksetRef from nemo_evaluator.jobs.agent_spec import AgentEvalInputSpec, ModelTarget from nemo_evaluator_sdk.values import Model # Instead of inlining AgentEvalTaskInput objects, point `tasks` at a stored taskset. input_spec = AgentEvalInputSpec( tasks=TasksetRef("default/geography-suite"), target=ModelTarget( model=Model(url="https://integrate.api.nvidia.com/v1", name="nvidia/nemotron-3.5-lightning-30b-a3b"), ), ) ``` When the job runs, the taskset reference is resolved like this: * The **taskset revision** the ref names is loaded — the current one unless the ref pins a revision (see [Pin the taskset itself](#pin-the-taskset-itself)). * Each member of that revision is loaded at the **revision pinned in the taskset**, not the task's current tip. * Metric references on those members are hydrated into runnable metrics, the same as for inline tasks. * Re-running the same taskset therefore evaluates the same content, even if a member has been republished since. ```text # What is stored on the taskset (digests fixed at create/replace time) geography-suite@1 └─ tasks: - default/capital-of-france#aaa111... ← revision 1 content - default/capital-of-japan#bbb222... # Later: the member task publishes new content capital-of-france tip → revision 2 (#ccc333...) # Re-grade still uses the taskset pins, not the tip JobRun A ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222... JobRun B ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222... ← same content ``` To evaluate the *new* task content, publish a new taskset revision (for example `tasksets.replace(...)`) so membership is re-resolved to the newer digests. Submit this spec as the agent-evaluate job input; see [Agent Evaluation](/documentation/evaluate-models/agent-eval) for the full run, target, and results flow. The inline form remains available for one-off tasks — swap `tasks=TasksetRef(...)` for `tasks=[AgentEvalTaskInput(...), ...]`. ### Pin the taskset itself A bare `TasksetRef` expands the taskset's current revision, so it follows the suite forward every time the taskset is republished. Add a `#` fragment to pin the grouping too: ```python # The digest lives on the revision, not on the taskset record — revisions come back newest first. current = tasksets.list_revisions("geography-suite").data[0] # Follows the suite forward — the next `replace` is picked up on the next run. tracking = TasksetRef("default/geography-suite") # Frozen: this exact membership and content, regardless of later `replace` calls. pinned = TasksetRef(f"default/geography-suite#{current.content_hash}") # A tag works the same way, and can be moved deliberately when you bless a new suite. blessed = TasksetRef("default/geography-suite#blessed") ``` What each form is stable against: | | A member task republishes | The taskset is replaced | | ------------------------------ | ------------------------- | ------------------------ | | `TasksetRef("suite")` | unaffected | follows the new revision | | `TasksetRef("suite#")` | unaffected | unaffected | Every taskset revision pins its members by digest, so **neither** form is disturbed when a member task publishes new content on its own — that is the guarantee stored membership buys you. The two differ on `replace`. A bare ref tracks the taskset's own revisions, and members are re-resolved on every write — so a `replace` can change *both* which tasks are named *and* the content they resolve to, even when the submitted member names were identical. Pin the taskset when a benchmark number has to stay comparable across that. A fragment that no longer resolves fails the evaluation rather than falling back to the current revision. A member's grader-only `reference` (held-out ground truth) is loaded from the pinned revision along with the rest of its content, so a taskset-driven run grades against the ground truth that revision fixed. Because `reference` is covered by the revision digest, changing it publishes a new revision — a pin fixes the grading, not just the prompt. ## Clean Up Delete the taskset before its member tasks, then remove the metric used by the tasks: ```python tasksets.delete("geography-suite") tasks.delete("capital-of-france") tasks.delete("capital-of-japan") client.evaluator.metrics.delete("answer-exact-match") ``` ## Async usage `AsyncNeMoPlatform` exposes the same surface; await each call. ```python import asyncio from nemo_platform import AsyncNeMoPlatform async def main() -> None: client = AsyncNeMoPlatform(base_url="http://localhost:8080", workspace="default") page = await client.evaluator.tasks.list() for item in page.data: print(item.name) asyncio.run(main()) ``` ## Workspaces and projects Every method accepts an optional `workspace` argument that overrides the client's default workspace. On create, an optional `project` argument associates the task or taskset with a project. When you omit `workspace`, the client's configured workspace is used. ## REST API The SDK resources are a thin client over the Evaluator plugin REST API, mounted under `/apis/evaluator/v2/workspaces/{workspace}`: | Method | Path | Description | | -------- | --------------------------------------- | ----------------------------------------------------------------- | | `GET` | `/tasks` | List tasks (paginated). | | `POST` | `/tasks/{name}` | Create a task and publish revision 1. | | `PUT` | `/tasks/{name}` | Replace a task's content and publish; creates it if absent. | | `GET` | `/tasks/{name}` | Retrieve a task's current content. | | `GET` | `/tasks/{name}/revisions` | List published revisions (paginated, newest first). | | `GET` | `/tasks/{name}/revisions/{revision}` | Retrieve content as of a digest or tag. | | `PUT` | `/tasks/{name}/tags/{tag}?revision=` | Point a tag at an existing revision. | | `DELETE` | `/tasks/{name}` | Delete a task and all of its revisions. | | `GET` | `/tasksets` | List tasksets (paginated). | | `POST` | `/tasksets/{name}` | Create a taskset and publish revision 1. | | `PUT` | `/tasksets/{name}` | Replace a taskset's membership and publish; creates it if absent. | | `GET` | `/tasksets/{name}` | Retrieve a taskset's current membership. | | `GET` | `/tasksets/{name}/revisions` | List published revisions (paginated, newest first). | | `GET` | `/tasksets/{name}/revisions/{revision}` | Retrieve membership as of a digest or tag. | | `PUT` | `/tasksets/{name}/tags/{tag}?revision=` | Point a tag at an existing revision. | | `DELETE` | `/tasksets/{name}` | Delete a taskset and all of its revisions. | ### Edge Cases `PUT` distinguishes its two outcomes by status: **`201`** when a new revision was published, and **`200`** when the submitted content was already the current revision and nothing was cut. The rest: | Case | Status | Behavior | | ------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------- | | `PUT` with new content | `201` | A revision is published and `latest` moves to it. | | `PUT` with content identical to the current revision | `200` | Nothing is published; any tags in the body are still applied. | | `PUT` with taskset members reordered | `200` | Membership is a set, so a reordering is not a content change. | | `PUT` reverting to *older* content | `201` | The record genuinely changed, so it publishes a new ordinal rather than reusing the old one. | | `PUT` re-resolving a member that has since republished | `201` | A taskset's identity is the exact revisions it names. | | Two identical `PUT`s racing | `201` + `200` | One publishes; the other adopts its revision rather than cutting a duplicate. | | `POST` on an existing name | `409` | Create is strict; use `PUT` to upsert. | | `PUT` losing a race with a concurrent write | `409` | Retry against the current state. | | Invalid metric reference (task) | `422` | Rejected at validation. | | Missing or duplicate task reference (taskset) | `422` | Members must exist, and must resolve to distinct tasks. | | Reserved or malformed tag name | `422` | `latest` cannot be moved by hand; a digest-shaped tag is refused. | | Retrieving or deleting an unknown name or revision | `404` | Applies to both records and revisions. | | `TasksetRef` pinning a revision that no longer resolves | job fails | Expansion refuses rather than falling back to the current revision. | | `DELETE` on a task a taskset pins | `204` | Not prevented; the taskset's reference dangles and fails on read. | ## Related Topics * [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) - Define and reuse the metrics that score a task * [SDK Resources](/documentation/evaluate-models/sdk-resources) - Run and submit evaluations through the Evaluator plugin * [Agent Evaluation](/documentation/evaluate-models/agent-eval) - How agent-eval tasks are executed and scored