> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server. # Manage Tasks & Tasksets A **task** is a stored, reusable definition of an agent-eval unit of work: an intent (what the agent should do), the inputs it receives, and the metrics that score it. A **taskset** is a named grouping of tasks. Both are first-class entities in the Evaluator plugin, addressed by `workspace/name` and managed through the `nemo_platform` SDK. Use stored tasks and tasksets when you want to define an evaluation unit once and reference it across runs, share it across a team, or assemble suites — rather than re-declaring the intent, inputs, and metrics inline every time. ## Concepts | Concept | What it is | Members | | ----------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | **Task** | A reusable agent-eval unit: `intent`, `inputs`, and the `metrics` that score it. | References the metrics that score it. | | **Taskset** | A flexible grouping of tasks with a description and metadata. | References member tasks by `workspace/name`. Membership is a **set** — order is not significant and duplicate references are rejected. | Both are addressed by `workspace/name`. Names are unique within a workspace, limited to 255 characters, and must match `^[\w\-\.]+$`. Tasks and tasksets support **create, retrieve, list, and delete** — there is no update. To change a stored task or taskset, delete it and create a new one, or store a new version under a different name. ## Initialize the SDK ```python import os from nemo_platform import NeMoPlatform client = NeMoPlatform( base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), workspace="default", ) tasks = client.evaluator.tasks # EvaluatorTasksResource tasksets = client.evaluator.tasksets # EvaluatorTasksetsResource ``` ## Manage Tasks A task scores its output with metrics. Store the metric first, then reference it from the task by `workspace/name`. See [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) for the metric classes and options. ```python from nemo_evaluator_sdk import ExactMatchMetric # Store a metric the task will reference. client.evaluator.metrics.create( "answer-exact-match", metric=ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}"), ) ``` Send the task as a `TaskInput` (the authorable subset of a task) and address it by name on create. Reference the stored metric with a `MetricRef` (`workspace/name`, or a bare `name` resolved against the task's workspace). The service returns the stored `Task`. ```python from nemo_evaluator.api.schemas import MetadataItem, MetricRef, TaskInput, TaskInputs task = TaskInput( intent="Answer the user's geography question with the capital city.", inputs=TaskInputs(instruction="What is the capital of France?"), metrics=[MetricRef("default/answer-exact-match")], metadata=[MetadataItem(key="suite", value="geography")], ) stored = tasks.create("capital-of-france", task=task) print(stored.id, stored.metrics) ``` ### `TaskInput` fields | Field | Type | Required | Description | | ---------- | ------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `intent` | `str` | Yes | Human-readable description of the desired agent behavior. | | `inputs` | `TaskInputs` | No | The task's recognized input fields. `instruction` is the agent's prompt; it falls back to `intent` when unset. | | `metrics` | `list[MetricRefOrInline]` | No | The metrics that score the task, as `MetricRef` references (`workspace/name`) to stored metrics. Pre-built inline metric bundles (`MetricInline`) are also accepted and are normalized to stored metrics on create. | | `views` | `dict[str, SemanticView]` | No | Optional reporting views mapping metric outputs into named semantic scores. | | `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. | A stored task holds **metric references only**. Any inline metric bundle you pass on create is stored as a content-addressed *derived* metric, and the task record is normalized to reference it. This is why `stored.metrics` always comes back as a list of `MetricRef` references. ### Retrieve, list, and delete ```python # Retrieve one task by name task = tasks.retrieve("capital-of-france") # List tasks in the workspace (paginated) page = tasks.list(page=1, page_size=100, sort="-created_at") for item in page.data: print(item.name, item.intent) # Delete a task tasks.delete("capital-of-france") ``` `sort` accepts `name`, `created_at`, or `updated_at`, each optionally prefixed with `-` for descending order. ## Manage Tasksets A taskset references existing tasks by `workspace/name`. All referenced tasks must already exist when the taskset is created; a missing or duplicate reference is rejected. ```python from nemo_evaluator.api.schemas import TaskRef, TasksetInput taskset = TasksetInput( description="Geography questions for smoke-testing the agent.", tasks=[ TaskRef("default/capital-of-france"), TaskRef("default/capital-of-japan"), ], ) stored = tasksets.create("geography-suite", taskset=taskset) print(stored.tasks) ``` ### `TasksetInput` fields | Field | Type | Required | Description | | ------------- | -------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- | | `description` | `str` | No | Human-readable description of the grouping. | | `tasks` | `list[TaskRef]` | No | References to member tasks (`workspace/name`, or bare `name` within the same workspace). Set semantics — duplicates rejected. | | `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. | ### Retrieve, list, and delete ```python taskset = tasksets.retrieve("geography-suite") page = tasksets.list(page=1, page_size=100, sort="name") for item in page.data: print(item.name, len(item.tasks)) tasksets.delete("geography-suite") ``` Deleting a taskset does not delete its member tasks — a taskset only holds references. ## Run an evaluation over a taskset An agent evaluation is submitted with an `AgentEvalInputSpec`, whose `tasks` field is either an inline list of tasks or a **reference to a stored taskset**. Referencing a taskset lets you keep the task definitions in one place and evaluate the whole set by name, instead of inlining every task on each run. ```python from nemo_evaluator.api.schemas import TasksetRef from nemo_evaluator.jobs.agent_spec import AgentEvalInputSpec, ModelTarget from nemo_evaluator_sdk.values import Model from nemo_evaluator_sdk.enums import ModelFormat # Instead of inlining AgentEvalTaskInput objects, point `tasks` at a stored taskset. input_spec = AgentEvalInputSpec( tasks=TasksetRef("default/geography-suite"), target=ModelTarget( model=Model(url="https://integrate.api.nvidia.com/v1", name="meta/llama-3.3-70b-instruct", format=ModelFormat.OPEN_AI), ), ) ``` When the job runs, the taskset reference is resolved: its member tasks are loaded, and each task's stored metric references are hydrated into runnable metrics — exactly as if you had inlined them. The same spec is submitted as the agent-evaluate job input; see [Agent Evaluation](/documentation/evaluate-models/agent-eval) for the full run, target, and results flow. The inline form remains available for one-off tasks — swap `tasks=TasksetRef(...)` for `tasks=[AgentEvalTaskInput(...), ...]`. Stored tasks carry no grader-only `reference` (held-out ground truth): that field lives only on inline `AgentEvalTaskInput`. Taskset-driven tasks therefore run with an empty `reference`, so use a taskset when your metrics score the agent's output directly rather than against per-task held-out data. ## Async usage `AsyncNeMoPlatform` exposes the same surface; await each call. ```python import asyncio from nemo_platform import AsyncNeMoPlatform async def main() -> None: client = AsyncNeMoPlatform(base_url="http://localhost:8080", workspace="default") page = await client.evaluator.tasks.list() for item in page.data: print(item.name) asyncio.run(main()) ``` ## Workspaces and projects Every method accepts an optional `workspace` argument that overrides the client's default workspace. On create, an optional `project` argument associates the task or taskset with a project. When you omit `workspace`, the client's configured workspace is used. ## REST API The SDK resources are a thin client over the Evaluator plugin REST API, mounted under `/apis/evaluator/v2/workspaces/{workspace}`: | Method | Path | Description | | -------- | ------------------ | -------------------------- | | `GET` | `/tasks` | List tasks (paginated). | | `POST` | `/tasks/{name}` | Create a task. | | `GET` | `/tasks/{name}` | Retrieve a task. | | `DELETE` | `/tasks/{name}` | Delete a task. | | `GET` | `/tasksets` | List tasksets (paginated). | | `POST` | `/tasksets/{name}` | Create a taskset. | | `GET` | `/tasksets/{name}` | Retrieve a taskset. | | `DELETE` | `/tasksets/{name}` | Delete a taskset. | Creating a name that already exists returns `409`. An invalid metric reference (task) or a missing or duplicate task reference (taskset) returns `422`. Retrieving or deleting a name that does not exist returns `404`. ## Related Topics * [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) - Define and reuse the metrics that score a task * [SDK Resources](/documentation/evaluate-models/sdk-resources) - Run and submit evaluations through the Evaluator plugin * [Agent Evaluation](/documentation/evaluate-models/agent-eval) - How agent-eval tasks are executed and scored