> This page is for version 0.10. > For other versions, use one of these documentation indexes: > - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt > - 0.10: https://docs.nvidia.com/nemo-helix/v0.10/llms.txt > - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt > - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt > - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt > - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt > - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server. # Execute Agents as Jobs An **execute-agent job** (`agents.execute`) runs an agent once, to completion, as a scheduled platform job. You give it an agent and a prompt; the platform schedules a container, runs the agent until it finishes or times out, saves everything the run produced, and exits. This is the bounded counterpart to [Deploy Agents](/nemo-helix/v0.10/documentation/agents/deploy-agents). A deployment is a service that stays up waiting for requests. An execute job is a single unit of work with a beginning and an end — you are asking an agent to do one thing, not to be available. ## When to Use a Job Instead of a Deployment | | Agent deployment | Execute-agent job | | -------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | Shape | Long-running service | One bounded run | | Invoked by | HTTP request through the Agents gateway | Job submission | | Lifetime | Until you `undeploy` | Until the agent finishes or the timeout fires | | Result | An HTTP response | Durable job results you fetch later | | Cost when idle | Holds resources | None — nothing runs between jobs | | Good for | Chat, interactive sessions, anything latency-sensitive with many callers | Batch work, scheduled analysis, long autonomous tasks, one-off tasks over a set of files | Reach for a job when the work is long enough that holding an HTTP connection open is the wrong shape, when the run produces files rather than a reply, or when the agent only needs to exist for the duration of one task. The CLI and Python SDK connect to the deployment in your current CLI context. To connect to a local or remote deployment, see [Connect with CLI Contexts](/nemo-helix/v0.10/documentation/reference/cli-reference/connect-to-deployments). #### CLI ```bash # Confirm the deployment and workspace in the current context nemo config view ``` #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient client = NemoClient.from_config() ``` > **Note** > > These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first. To check, run `nemo config view`. --- ## How It Works 1. **Submission resolves and snapshots everything.** When the job is created, the platform resolves the agent reference, resolves and merges the [agent environment](/nemo-helix/v0.10/documentation/agents/deploy-agents#agent-environments), and validates any working-directory references. A bad agent name, a missing secret, or a nonexistent fileset fails your create request rather than the run. The resolved config, compute, and secret references are snapshotted onto the job, so a later edit to the underlying entities does not change what an already-submitted job runs. 2. **The job runs the agent once.** The scheduled step stages the working directory, invokes the agent through Fabric with your prompt, and waits up to `timeout_seconds` for a result. 3. **Everything is saved as job results.** The input working directory, the output working directory, the run's artifacts, and Fabric's own run record are all saved and downloadable after the job finishes — including when it fails. ## Submit a Job The examples below use the calculator agent that ships with the source checkout. Register it first if you have not already: ```bash nemo agents register \ --name calculator-agent \ --agent-config plugins/nemo-agents/examples/nemo-agent-config/calculator-agent/agent.yaml ``` #### CLI ```bash nemo agents execute \ --agent calculator-agent \ --input "What is 12 multiplied by 8?" ``` The command prints the created job record, including the generated `name` you use to track it. #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient from nemo_agents_plugin.sdk import AgentsResource client = NemoClient.from_config() agents = AgentsResource(client) job = agents.jobs.execute.create( spec={ "agent": "calculator-agent", "input": "What is 12 multiplied by 8?", }, ) print(job["name"], job["status"]) ``` ### Job Spec Fields | Field | Required | Description | | ----------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `agent` | Yes | An Agent entity name, a `workspace/name` ref, or an [inline agent definition](#inline-agents). | | `input` | Yes, unless the agent declares a [`default_input`](#declare-how-an-agent-runs) | The prompt handed to the agent. | | `context` | No | [Per-run parameters](#run-context) handed to the agent alongside the prompt. Must match the agent's [`context_schema`](#declare-how-an-agent-runs) when it declares one. | | `environment` | No | A `workspace/name` ref to a stored `AgentEnvironment`, or an inline environment. Supplies secrets, environment variables, tool settings, and compute. | | `workdir` | No | The [working directory](#working-directories) the agent starts from. | | `timeout_seconds` | No | Maximum time to wait for a result. Defaults to 3600 (one hour). | | `auto_telemetry` | No | Automatically configures supported ATIF telemetry to export to Intake. Defaults to `true`. | | `image` | No | The [container image](#execution-image) to run the agent in. Defaults to the platform's task image. | | `post_run_action` | Depends | A [post-run action](#post-run-actions) contributed by an installed plugin. Required only when the agent declares several actions and not `agents.noop`. | The CLI generates one flag per spec field (`--agent`, `--input`, `--environment`, `--workdir.base-workdir`, `--timeout-seconds`, `--auto-telemetry`, `--image`, `--post-run-action.kind`). Fields that do not reduce to a single flag — an inline agent, an inline environment, `context`, `workdir.artifact_mounts`, `post_run_action.config` — are supplied with `--spec` (a JSON string) or `--spec-file` (a YAML or JSON file). Individual flags override values from the supplied spec. ### Execution Image The job runs the agent inside a container image. That image must contain the Fabric adapter the agent's harness names, so the image you need depends on the harness: * **A bundled harness** (`claude`, `codex`, or `deepagents`) needs no image at all. Its harness and adapter ship in the platform's task image, which is what the job uses when you omit `image`. * **Hermes** needs a separately packaged execution image because its harness cannot share the platform environment. * **Your own Fabric adapter** needs an image that contains it. Build one with `nemo agents package` and pass it as `image`; the packaged image carries both the adapter and the execute task module. Resolution is, in order: the request's `image`; the `agents.jobs.default_image` platform setting; the execution profile's `default_task_image`; the platform CPU tasks image. Setting `agents.jobs.default_image` changes the default for every execute job on the install, and is unrelated to `agents.deployments.default_image`, which is the fallback for `AgentDeployment` images only. > **Note** > > A custom image is pulled with the credentials the jobs runtime already has: > the Docker daemon's for `docker` mode, and the platform's `image_pull_secrets` > merged with the execution profile's for `k8s`. There is no per-job pull > credential, so a private image has to be reachable with those. ## Track a Job and Fetch Its Results An execute job is an ordinary platform job. `nemo agents execute` submits it; from there it is tracked and read back like any other job, by name (e.g. `"calc-run-1"` below), using the generic `nemo jobs` commands. The Python SDK does mirror the whole lifecycle under `agents.jobs.execute`. #### CLI ```bash # Block until the job reaches a terminal status, streaming logs nemo jobs watch calc-run-1 # Or poll it yourself nemo jobs get-status calc-run-1 nemo jobs tail calc-run-1 -n 200 # List what the run produced, then download one result nemo jobs results list calc-run-1 nemo jobs results download output_artifacts --job calc-run-1 -o output_artifacts.tar.gz ``` #### Python SDK ```python import json import tarfile from io import BytesIO job = agents.jobs.execute.get("calc-run-1") print(job["status"]) results = agents.jobs.execute.list_results("calc-run-1") for result in results["data"]: print(result["name"]) # Directory results arrive as a gzipped tarball... workdir = agents.jobs.execute.download_result("output_workdir", job="calc-run-1") with tarfile.open(fileobj=BytesIO(workdir), mode="r:gz") as tar: print(tar.getnames()) # ...and the run record as JSON. run = json.loads(agents.jobs.execute.download_result("fabric_run_result", job="calc-run-1")) print(run["status"], run["response"]) ``` `download_result` returns bytes rather than writing a file, so results can be read in memory. The same calls exist on `AsyncAgentsResource`. ### List an Agent's Runs Each execute job records the `workspace/name` reference of the agent it ran as `spec.agent_ref`. Use it as a filter to list an agent's runs. Cross-workspace agent references are supported. Jobs that ran an [inline agent](#inline-agents) are not runs of any persisted agent and thus never match. #### CLI ```bash nemo jobs list --filter.source nemo-agents-plugin-execute \ --filter '{"spec.agent_ref": {"$in": ["default/calc", "research/calc"]}}' ``` #### Python SDK ```python runs = agents.jobs.execute.list(agents=["default/calc", "research/calc"]) ``` ### Result Names Results download as gzipped tarballs of a directory, or as a single JSON file. | Result | Saved | Contains | | ------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------- | | `input_workdir` | Always | The working directory as the agent found it, after staging. The record of what the run was given. | | `output_workdir` | Always | The working directory as the agent left it — every file it created or modified. | | `output_artifacts` | Always | Fabric's per-run artifacts, including the agent process's captured stdout and stderr. | | `fabric_run_result` | On a completed run | Fabric's run record: status, response, runtime and invocation IDs. | | `fabric_error` | When the invocation itself raised | The error type, message, and whether it was a timeout. | A [post-run action](#post-run-actions) can save results of its own alongside these. > **Note** > > `output_workdir` and `output_artifacts` are saved on failure too, on a > best-effort basis. When a run fails, that may be where the explanation is — > the job also logs the tail of the agent's stderr so the initial diagnosis needs no > downloads. ## Working Directories An agent that reads and writes files needs somewhere to do it. `workdir` builds that directory from the [Files service](/nemo-helix/v0.10/documentation/get-started/core-concepts) before the agent starts. Stored-agent skills come from the agent's ethos fileset, staged at the Fabric base before this directory is filled in. | Field | Description | | ----------------- | ------------------------------------------------------------------------------------------ | | `base_workdir` | A Files reference (`workspace/fileset` or `fileset#path/`) copied in as the starting tree. | | `artifact_mounts` | Individual fileset artifacts layered on top, each at a relative `mount_path`. | Mounts are applied after the base tree, so a mount at the same path wins. Mount paths must be relative and must not overlap each other. ```yaml # run.yaml agent: research-agent input: Review the config and summarize the risks. workdir: base_workdir: default/project-source artifact_mounts: - ref: default/config-artifact#config.yaml mount_path: app/config.yaml - ref: default/notes-artifact#notes.txt mount_path: notes/notes.txt ``` ```bash nemo agents execute --spec-file run.yaml ``` Whatever the agent leaves behind in that directory comes back as `output_workdir`. ## Environments, Secrets, and Compute Execute jobs use the same environment model as deployments. Pass `--environment default/research` and the environment's spec is merged into the agent config, its secret references become secret-backed environment variables on the job step, and its compute spec sizes the container. ```bash nemo agents execute \ --agent research-agent \ --environment default/research \ --input "Summarize this repo's dependencies and risks." ``` Compute is expressed the Kubernetes way on the compute spec. Execute jobs support `cpu`, `memory`, and `nvidia.com/gpu`; any other resource key is rejected at submission rather than silently dropped. See [Agent Environments](/nemo-helix/v0.10/documentation/agents/deploy-agents#agent-environments) for how to create the environment, environment spec, and compute spec. ## Inline Agents `agent` also accepts a full agent definition instead of a reference. The config is validated at submission and snapshotted onto the job, but it is never stored as an Agent entity. This suits an agent composed per request — models chosen by the caller, harness settings scoped to a single run — where there would be nothing to keep in sync afterward. ```json { "agent": { "config_format": "nemo-agents-spec-v1", "config": { "config_format": "nemo-agents-spec-v1", "name": "one-off-analyst", "default_harness": "deepagents", "harnesses": {"deepagents": {"kind": "deepagents", "settings": {"deepagents": {}}}}, "models": { "default": { "provider": "nvidia", "model": "nvidia-nemotron-3-5-lightning-30b-a3b", "api_key_env": "NVIDIA_API_KEY" } } } }, "input": "Summarize yesterday's failures." } ``` Supply it with `--spec`/`--spec-file` from the CLI, or as the `spec` body from the SDK. ## Run Context `context` carries per-run parameters to the agent as the top level of the Fabric request context, so one agent definition can serve many runs. The platform adds its own fields under the reserved `nemo_helix` key: `job_id`, `job_workspace`, and `agent` (the `workspace/name` that ran). A request whose `context` sets `nemo_helix` is rejected. ```json { "agent": "research/analyst", "input": "Analyze recent runs.", "context": {"since": "2026-10-01T00:00:00Z"} } ``` The agent receives: ```json { "since": "2026-10-01T00:00:00Z", "nemo_helix": {"job_id": "job-…", "job_workspace": "research", "agent": "research/analyst"} } ``` `context` is stored on the job spec in plaintext. Use it for parameters, not secrets; bind secrets through an [environment](#environments-secrets-and-compute). When the agent declares a [`context_schema`](#declare-how-an-agent-runs), the job's `context` is validated against it before the platform adds `nemo_helix`. A context that does not match fails the create request. ## Declare How an Agent Runs An Agent entity can declare the runs it accepts, so a caller learns about a mistake when it creates the job rather than when the agent misreads it. Set these fields under `helix_registration` in the `agent.yaml` you pass to `nemo agents register`, or as top-level fields of the register request when you use the API: | Field | Default | Description | | ------------------ | ------------------ | ------------------------------------------------------------------------------------------------------- | | `context_schema` | None (any context) | A JSON Schema the job's `context` must match. Also accepted on an [inline agent](#inline-agents). | | `default_input` | None | The prompt a job uses when it omits `input`. | | `post_run_actions` | `["agents.noop"]` | The [post-run action](#post-run-actions) kinds a job may choose. Must name at least one installed kind. | #### agent.yaml ```yaml config_format: nemo-agents-spec-v1 name: analyst helix_registration: context_schema: type: object properties: since: {type: string, format: date-time} required: [since] additionalProperties: false default_input: Analyze recent runs. post_run_actions: [insights.analysis] # ...harnesses, models, and the rest of the agent config ``` #### API ```json { "name": "analyst", "config": {"...": "..."}, "context_schema": { "type": "object", "properties": {"since": {"type": "string", "format": "date-time"}}, "required": ["since"], "additionalProperties": false }, "default_input": "Analyze recent runs.", "post_run_actions": ["insights.analysis"] } ``` A `helix_registration` block inside the request's `config` is rejected: over the API, these are fields of the agent, not of its config. With this agent, a job can omit `input`, must pass `context.since` as an RFC 3339 timestamp, and is rejected if it misspells a key such as `sinse`. Registration checks that `context_schema` is a valid JSON Schema (Draft 2020-12 unless it names another dialect in `$schema`) and that every kind in `post_run_actions` is installed. A `context_schema` cannot use references (`$ref`, `$dynamicRef`, or `$recursiveRef`); write each property's constraints in place. A job that omits `post_run_action` gets one from the agent's declaration: | Declared `post_run_actions` | Job omits `post_run_action` | Job names a kind | | -------------------------------- | ----------------------------------- | ----------------------- | | Includes `agents.noop` | Runs `agents.noop` | Must be a declared kind | | Exactly one kind | Runs that kind with an empty config | Must be that kind | | Several kinds, not `agents.noop` | Rejected: the job must choose | Must be a declared kind | | Inline agent (no declaration) | Runs `agents.noop` | Any installed kind | > **Note** > > An Agent registered without `post_run_actions`, including one registered before > the field existed, allows only `agents.noop`. A job for that agent that names > any other action is rejected. Add the actions its jobs need under > `helix_registration.post_run_actions` in its `agent.yaml`, then run > `nemo agents delete` and `nemo agents register` again. For a deployed agent, > `nemo agents redeploy` does both and redeploys it. ## Post-Run Actions A post-run action is plugin-owned work that turns the agent's output into something the plugin owns — for example, the Compass plugin's analysis runs attach an action that saves an `analysis-report` result and files the insights it found. * **When it runs.** After the agent invocation succeeds, inside the job's execute step. It does not run when the invocation fails, and it never runs for `nemo agents invoke` or a deployed agent; only execute jobs have one. * **Failure.** If the action raises, the job fails, even though the agent itself succeeded. `fabric_run_result` and the working-directory results are saved before the action starts, so they are still available. * **Results.** Anything the action saves appears in the job's results next to `fabric_run_result`. An action is named by `post_run_action.kind` and configured by `post_run_action.config`. An Agent entity limits the kinds its jobs may choose to its [`post_run_actions`](#declare-how-an-agent-runs), and decides what an omitted `post_run_action` means. An inline agent declares nothing, so its job may name any installed kind, and omitting `post_run_action` runs `agents.noop`, which does nothing. Only kinds installed on the platform are accepted, and the config is validated against the action's own schema; an unknown kind or a malformed config fails the create request rather than the run. The plugin that provides the action must also be installed in the job's [execution image](#execution-image); the job checks this when it starts, and fails before invoking the agent if the action is missing or that version of it rejects the config. If you are using a feature built on execute jobs, its own commands set this for you. List the actions installed on the platform: ```bash nemo agents post-run-actions list ``` The same list is available from `GET /apis/agents/v2/post-run-actions`, which returns each action's `kind`, `title`, and `description`. ### Write a Post-Run Action A plugin contributes an action by registering a class under the `nemo.agents.post_run_actions` entry-point group. The entry-point name is the action's `kind`: ```toml [project.entry-points."nemo.agents.post_run_actions"] "my-plugin.publish" = "my_plugin.actions:PublishAction" ``` The class implements the `PostRunAction` protocol from `nemo_helix_plugin.agents.post_run_actions`, which does not depend on the Agents plugin: ```python from typing import ClassVar from nemo_helix_plugin.agents.post_run_actions import PostRunActionContext from pydantic import BaseModel, ConfigDict class PublishActionConfig(BaseModel): model_config = ConfigDict(extra="forbid") destination: str class PublishAction: title: ClassVar[str] = "Publish" description: ClassVar[str] = "Publish the agent's answer to a destination." config_model: ClassVar[type[PublishActionConfig]] = PublishActionConfig def after_invoke(self, context: PostRunActionContext) -> None: config = self.config_model.model_validate(context.config) ... ``` `PostRunActionContext` carries the job context (`ctx`, for saving results), the action's `config`, the `agent_name`, the resolved `agent_config` the job ran (as snapshotted when the job was created), and Fabric's `fabric_result`. ## Requirements and Limits * **Config format.** Only `nemo-agents-spec-v1` agents can run as execute jobs. Legacy NAT workflow configs are rejected; use `nemo agents evaluate` or a deployment for those. * **Fabric environment.** The merged agent config must select the `local` Fabric environment provider. Because an environment spec can override the provider, this is checked after the merge. * **Timeout.** `timeout_seconds` bounds the wait for a Fabric result and defaults to one hour. A run that exceeds it fails with a saved `fabric_error` rather than hanging. * **Reserved environment variables.** A secret cannot be bound to an environment variable name the platform injects itself (the `NEMO_JOB_*` family, `NHX_BASE_URL`, `AGENT_CONFIG_PATH`, and similar). The collision is rejected at submission. ## Clean Up Job records and their results persist until you remove them. ```bash nemo jobs cancel calc-run-1 # stop a running job nemo jobs delete calc-run-1 # remove the record and its results ```