Deploy Agents

Deploy a registered agent as a running service and invoke it through the Agents gateway. An agent can run in one of three modes — as a local subprocess (the default), or as a durable container on Docker or Kubernetes.

$
# Configure CLI (if not already done)
$
nemo config set --base-url "$NMP_BASE_URL" --workspace default

Deployment Modes

nemo agents deploy --mode <mode> selects the runtime backend for a deployment. The default is subprocess; docker and k8s run the agent as a durable container through the deployments plugin.

In every mode the agent is reached the same way — through the Agents gateway, which resolves the agent’s active deployment and proxies the request, so clients do not need to know which mode the agent runs in.

Subprocess Mode (Default)

The simplest path: the platform launches a nat start fastapi process for the agent on its own host, assigns a port, watches its health, and tears it down on nemo agents undeploy. No image or executor configuration is required.

$
# Register the agent from a NAT workflow YAML
$
nemo agents create --name calc-agent \
>
  --agent-config ./calculator-agent.yml
$
$
# Deploy as a local subprocess (waits for "running" by default)
$
nemo agents deploy --agent calc-agent
$
$
# Invoke through the Agents gateway
$
nemo agents invoke --agent calc-agent --input "What is 12 multiplied by 8?"

Container Modes (Docker / Kubernetes)

Container modes give an agent a durable deployment that survives a platform restart. Instead of a local process, the platform compiles the agent into a generic deployment and hands it to the deployments plugin, which runs it on the configured executor (Docker or Kubernetes) and projects the running container’s address back onto the agent deployment. The Agents gateway then routes to that projected address.

Prerequisites

1. A container image for the agent. Container modes run a NAT runtime image that has nat on its PATH plus your agent’s dependencies (tools, custom components). Build one with nemo agents package. Image building requires the container extra — install it with pip install 'nemo-agents-plugin[container]' if nemo agents package reports that python-on-whales is missing.

The examples below use the calculator agent that ships with the source checkout. From the repository root, its project directory is plugins/nemo-agents/examples/calculator-agent/ — the workflow YAML lives at src/calculator_agent/calculator-agent.yml and the project’s pyproject.toml sits at the directory root. cd into that project directory first so the paths below resolve:

$
# From the repo root, switch into the example's project directory
>
cd plugins/nemo-agents/examples/calculator-agent
>
>
# Render a Dockerfile, build the image, and tag it locally.
>
# --pyproject builds in "project mode" so the agent's custom components
$
# (here, the `calculator` function group) are installed into the image.
$
# Omit --pyproject only for a single-file agent that has no local package.
$
nemo agents package \
>
  --agent src/calculator_agent/calculator-agent.yml \
>
  --pyproject pyproject.toml \
>
  --nat-version 1.8.0 \
>
  --tag calc-agent:local
$
$
# For k8s, publish to a registry your cluster can pull from
$
nemo agents package \
>
  --agent src/calculator_agent/calculator-agent.yml \
>
  --pyproject pyproject.toml \
>
  --nat-version 1.8.0 \
>
  --tag calc-agent:1.0 \
>
  --publish --registry <your-registry>

2. A configured deployments executor. The platform operator defines named executors in the platform configuration and points the agents plugin at them. A minimal Docker + Kubernetes configuration:

1
agents:
2
  deployments:
3
    # Names below must match deployments.executors[].name
4
    docker_executor: local-docker
5
    k8s_executor: k8s-local
6
    # Optional: the image used when --image is omitted
7
    default_image: ""
8
    # Container port the NAT server binds (and the readiness probe target)
9
    container_port: 8000
10
11
deployments:
12
  default_executor: local-docker
13
  executors:
14
    - name: local-docker
15
      backend: docker
16
      config:
17
        # Set false when running locally-built (not registry) images
18
        pull_images: false
19
    - name: k8s-local
20
      backend: k8s
21
      config:
22
        # Omit kubeconfig_path to use in-cluster ServiceAccount auth
23
        default_namespace: default

Deploy on Docker

$
# Run from the example's project directory (see Prerequisites)
>
nemo agents create --name calc-agent \
>
  --agent-config src/calculator_agent/calculator-agent.yml
>
>
# --mode docker compiles to the deployments plugin's docker executor
$
nemo agents deploy --agent calc-agent \
>
  --mode docker \
>
  --image calc-agent:local
$
$
# Reached through the Agents gateway, exactly like subprocess mode
$
nemo agents invoke --agent calc-agent --input "What is 12 multiplied by 8?"

Deploy on Kubernetes

Deployment is identical apart from --mode k8s. The deployments plugin creates a Kubernetes Deployment and a ClusterIP Service, and projects the Service’s in-cluster DNS address (<service>.<namespace>.svc.cluster.local:<port>) onto the agent deployment. Because the Agents gateway runs in-cluster, it routes to that address directly.

$
# Run from the example's project directory (see Prerequisites)
>
nemo agents create --name calc-agent \
>
  --agent-config src/calculator_agent/calculator-agent.yml
>
>
nemo agents deploy --agent calc-agent \
>
  --mode k8s \
>
  --image <registry>/calc-agent:1.0
>
>
nemo agents invoke --agent calc-agent --input "What is 12 multiplied by 8?"

Model Access from a Deployed Agent

Regardless of mode, model traffic from inside the agent routes back through the Inference Gateway. The platform injects the gateway URL when it deploys the agent, and the gateway resolves model entity names to upstream providers and supplies their credentials. Two conventions apply to the agent’s workflow YAML:

• Leave base_url and api_key unset on openai and nim LLMs — the deployment injects the gateway URL and the gateway looks up upstream credentials.
• Reference models by their Inference Gateway entity name, with slashes and dots converted to hyphens (meta/llama-3.1-8b-instruct becomes default/meta-llama-3-1-8b-instruct). Use ${NEMO_DEFAULT_MODEL} in agent config files to defer to the SDK or CLI context’s configured default model at agent registration time. If you call the REST API directly, replace the placeholder with an explicit VirtualModel name before creating the agent.

To make an external model available to the agent, register a provider first — see Deploy Models for NVIDIA Build, OpenAI, and Anthropic examples.

Docker mode on Linux

Docker Desktop (macOS/Windows) handles this automatically. On Linux, where host.docker.internal does not resolve inside containers, point the platform at the Docker bridge address (172.17.0.1 by default) instead, or nemo agents invoke fails with openai.APIConnectionError.

Set both in config.yaml, then start the platform bound to all interfaces:

1
platform:
2
  base_url: http://172.17.0.1:8080
3
agents:
4
  deployments:
5
    gateway_url_override: http://172.17.0.1:8080

$
uv run nemo services run --host 0.0.0.0 --port 8080

If your Docker bridge uses a non-default subnet, substitute its gateway address (docker network inspect bridge --format '{{ (index .IPAM.Config 0).Gateway }}').

Inspect a Deployment

$
# Block until the deployment is running or failed
$
nemo agents deployments wait --agent calc-agent
$
$
# List / inspect deployments
$
nemo agents deployments list
$
nemo agents deployments get <deployment-name>

For a container-mode deployment, the deployment reports deployment_mode (docker or k8s), a status of running once ready, and an endpoints list carrying the container’s routable address. Subprocess deployments carry a loopback endpoint instead. The Agents gateway uses whichever the deployment’s mode provides, so invocation is identical across modes.

Deployment Cleanup

$
# Stop the running deployment (removes the process/container/k8s objects)
$
nemo agents undeploy <deployment-name>
$
$
# Remove the agent entity
$
nemo agents delete <agent-name>