> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Run the Agent Optimizer

Use the Agent Optimizer to analyze a deployed agent and act on improvement
suggestions. The optimizer inspects the agent's config, the workspace model
catalog, any prior optimizer snapshots, and optional evaluation baselines,
then writes suggestions you can review from the CLI or hand off to a coding
agent.

This page covers the main path: establish a baseline, generate optimization
suggestions, apply a candidate change to a sibling agent, and review the
evaluation result before promotion.

## What the Optimizer Checks

| Suggestion type     | Signal                                                                                                        | Result                                                                                                                     |
| ------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Model optimization  | An agent uses a single frontier model where a smaller model or route split may preserve quality at lower cost | Suggests a model swap or Switchyard random-routing virtual model                                                           |
| Skill optimization  | The agent uses skills and has an evaluation suite                                                             | Suggests running `nemo agents optimize-skills` to improve skill files and keep changes that pass evaluation                |
| Prompt optimization | The agent has an optimization config and baseline dataset                                                     | Suggests staging with `nemo agents optimize prepare-fileset`, then running `nemo agents optimize` for Fabric-backed tuning |
| New model scan      | Difference between the current model list and the previous optimizer snapshot                                 | Suggests evaluating or auditing newly available models                                                                     |

Optimizer state is stored in the `nemo-agent-optimizer` fileset:

* `optimizer_suggestions.jsonl`: one suggestion per line, including applied state.
* `optimizer_snapshot.json`: model and agent names from the latest run.

Security-oriented suggestions such as missing guardrails, PII exposure, or
leaked secrets are covered in [Secure Agents](/documentation/agents/secure-agents).

## Prerequisites

Before running the optimizer, make sure you have:

1. Local services running (`nemo services run`).
2. The agents plugin installed. For local development from this repository:
   ```bash
   uv sync --package nemo-agents-plugin
   source .venv/bin/activate   # puts `nemo` on PATH
   ```
3. A workspace with at least one model provider and discovered model entities.
4. At least one deployed platform-managed agent.
5. An evaluation baseline before promoting a candidate agent.

If you need a demo agent, start the platform and create the ReAct example:

```bash
nemo services run
```

In another terminal:

```bash
export NMP_BASE_URL=http://127.0.0.1:8080
cd plugins/nemo-agents

printf '%s' "$NVIDIA_API_KEY" | nemo secrets create ngc-api-key --from-file -
nemo inference providers create nvidia-build \
  --host-url https://integrate.api.nvidia.com \
  --api-key-secret-name ngc-api-key
nemo wait inference provider nvidia-build

nemo agents create \
  --name react-agent \
  --agent-config examples/react-agent/react-agent.yml
nemo agents deploy --agent react-agent
nemo agents deployments wait --agent react-agent
```

The example agent uses `nvidia-nemotron-3-nano-30b-a3b`, so it can produce a
model optimization suggestion when the workspace model catalog contains a
smaller compatible model.

## Optimize with Switchyard Routing

Switchyard is the inference middleware that lets a virtual model split traffic
across multiple backend models. The common optimization pattern is to create a
[virtual model](/documentation/models-and-inference) with a strong model and a weaker,
cheaper model, then evaluate whether the route split preserves application
quality.

Run `nemo models list` first and replace the placeholders below with model
entity names from your workspace that use the `OPENAI_CHAT` backend format.

#### CLI

The command below creates a virtual model that sends 80% of traffic to the
strong model and 20% to the weak one.

```bash
nemo inference virtual-models create routed-agent-model \
  --workspace default \
  --models '[
    {"model":"default/<strong-model-entity>","backend_format":"OPENAI_CHAT"},
    {"model":"default/<weak-model-entity>","backend_format":"OPENAI_CHAT"}
  ]' \
  --request-middleware '[{
    "name":"nemo-switchyard",
    "config_type":"random_routing",
    "config":{
      "strong":{"model":"default/<strong-model-entity>"},
      "weak":{"model":"default/<weak-model-entity>"},
      "strong_probability":0.8,
      "enable_stats":false
    }
  }]'
```

Before wiring the virtual model to an agent, smoke-test the route by
making several minimal chat-completions calls and checking the returned
model name. The observed split should roughly match `strong_probability`.

#### Skill

Ask your coding agent:

> Optimize my deployed agent.

The `agents-optimize` skill picks a deployed agent, establishes an
evaluation baseline, runs the analysis steps below, and surfaces
suggestions for you to apply.

Verify the skill is installed:

```bash
nemo skills show agents-optimize
```

What it does under the hood:

* Lists deployed agents and prompts you to choose one.
* Inspects the agent's `llms[*].model_name` and looks for cheaper compatible
  models in the workspace catalog.
* Creates a Switchyard `random_routing` virtual model with an 80% strong /
  20% weak split and smoke-tests the route before wiring it to a sibling
  agent.
* Suggests skill optimization, prompt tuning, and new-model evaluations
  where the agent qualifies.
* Persists suggestions to the `nemo-agent-optimizer` fileset.

#### Python SDK

```python
import os
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

client.inference.virtual_models.create(
    name="routed-agent-model",
    workspace="default",
    models=[
        {"model": "default/<strong-model-entity>", "backend_format": "OPENAI_CHAT"},
        {"model": "default/<weak-model-entity>", "backend_format": "OPENAI_CHAT"},
    ],
    request_middleware=[{
        "name": "nemo-switchyard",
        "config_type": "random_routing",
        "config": {
            "strong": {"model": "default/<strong-model-entity>"},
            "weak": {"model": "default/<weak-model-entity>"},
            "strong_probability": 0.8,
            "enable_stats": False,
        },
    }],
)
```

## Optimize Skills

Skill optimization applies when the agent depends on local skill files and has
an evaluation suite. The loop runs evaluations, analyzes failures, lets the
coding agent edit only the configured skills directory, reruns verification,
and keeps the change only when the evaluation result improves.

#### CLI

```bash
nemo agents optimize-skills --spec-file .agent-improver.yml
```

Set `open_pr: true` in the YAML when you want the loop to prepare a
reviewable branch.

A sample `.agent-improver.yml` is in
`plugins/nemo-agents/examples/agent-improver.example.yml`.

#### Skill

Ask your coding agent:

> Optimize the skills used by my agent and keep the changes that improve evaluation scores.

The `agents-optimize` skill drives the skill-optimization loop when the
selected agent has skills and an evaluation suite. Verify it is installed:

```bash
nemo skills show agents-optimize
```

What it does under the hood:

* Confirms the agent uses skills (a `--skills-path`, a `.agent-improver.yml`,
  or skill files referenced from the config).
* Runs `nemo agents optimize-skills` against the configured skills directory.
* Re-runs evaluation and keeps the change only when scores improve.
* Persists outcomes to the `nemo-agent-optimizer` fileset.

#### Python SDK

```python
import yaml
from pathlib import Path

from nemo_agents_plugin.jobs.optimize_skills import OptimizeSkillsJob
from nemo_platform_plugin.scheduler import NemoJobScheduler

spec = yaml.safe_load(Path(".agent-improver.yml").read_text())
NemoJobScheduler().run_local(
    OptimizeSkillsJob,
    spec,
    workspace="default",
)
```

## Inspect Saved Results

Use the Files service to inspect what the optimizer saved:

```bash
nemo files list nemo-agent-optimizer

nemo files download nemo-agent-optimizer \
  --remote-path optimizer_suggestions.jsonl \
  -o optimizer_suggestions.jsonl

nemo files download nemo-agent-optimizer \
  --remote-path optimizer_snapshot.json \
  -o optimizer_snapshot.json
```

Telemetry is optional. If agents use the `nemo_files` telemetry exporter, trace
files are written to `nemo-agent-telemetry`, and the optimizer samples the
largest JSONL file:

```bash
nemo files list nemo-agent-telemetry
```

## Run Prompt and Parameter Tuning

The `nemo agents optimize` command submits Fabric-backed numeric
optimization through `agents.optimize` (implementation in
`nemo-optimization`). Input must be a Fabric-native agent package
(`schema_version: fabric.agent/v1alpha1`). The golden-path harness is
Hermes (`nvidia.fabric.hermes`); see
`plugins/nemo-optimization/examples/hermes-optimize/` (install steps live in
that README).

After `uv sync --package nemo-agents-plugin` (and activating `.venv`), invoke
`nemo` directly.

For CLI platform runs, keep the config and everything it references in one
directory — an *optimize bundle* — stage that bundle with `prepare-fileset`,
and pass `--optimize-config` as a path relative to the staged fileset root.
For local Python SDK runs, `optimize_config` must be an **absolute** host path;
paths *inside* the YAML (`dataset`, `eval.fabric.base_dir`, hook and MCP configs)
resolve against your current working directory.

```bash
export BUNDLE="$(pwd)/plugins/nemo-optimization/examples/hermes-optimize"
cd "$BUNDLE"
```

### Chat-only Hermes (no MCP)

#### CLI

```bash
nemo agents optimize prepare-fileset \
  --source "$BUNDLE" \
  --optimize-config optimize-chatonly.yaml \
  --fileset hermes-optimize-chatonly \
  --workspace default
nemo agents optimize \
  --optimize-config-fileset default/hermes-optimize-chatonly \
  --optimize-config optimize-chatonly.yaml \
  --workspace default
```

#### Skill

Ask your coding agent:

> Run prompt tuning on my deployed agent against this optimization config.

The `agents-optimize` skill suggests `nemo agents optimize` when the agent
has an optimization config and a baseline dataset. Verify it is
installed:

```bash
nemo skills show agents-optimize
```

What it does under the hood:

* Confirms the agent has a Fabric-native optimization YAML.
* Stages the optimize bundle with `nemo agents optimize prepare-fileset`,
  then submits it with `nemo agents optimize`.
* Compares results against the evaluation baseline and surfaces deltas
  for review.

#### Python SDK

```python
import os
from pathlib import Path

from nemo_optimization.jobs.optimize import OptimizeJob
from nemo_platform import NeMoPlatform
from nemo_platform_plugin.scheduler import NemoJobScheduler

WORKSPACE = "default"
bundle = Path("plugins/nemo-optimization/examples/hermes-optimize").resolve()
optimize_config = bundle / "optimize-chatonly.yaml"
os.chdir(bundle)  # dataset / base_dir are relative to the bundle

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace=WORKSPACE,
)

result = NemoJobScheduler().run_local(
    OptimizeJob,
    {
        "optimize_config": str(optimize_config),
        "workspace": WORKSPACE,
    },
    workspace=WORKSPACE,
    sdk=client,
)
print(result)
```

### MCP Hermes (phishing analyzer)

Point `PHISHING_AGENT_SRC` / `PHISHING_MCP_BIN` at an
`email-phishing-analyzer-harnesses` checkout (its own `.venv` after
`uv sync`). Do **not** pip-install that agent into the platform venv.
Full setup steps are in `plugins/nemo-optimization/examples/hermes-optimize/README.md`.

#### CLI

```bash
export PHISHING_AGENT_ROOT="${PHISHING_AGENT_ROOT:-$HOME/work/email-phishing-analyzer-harnesses}"
export PHISHING_AGENT_SRC="$PHISHING_AGENT_ROOT/src"
export PHISHING_MCP_BIN="$PHISHING_AGENT_ROOT/.venv/bin/email-phishing-analyzer-mcp"

# These environment variables are templated into optimize-mcp.yaml.

nemo agents optimize prepare-fileset \
  --source "$BUNDLE" \
  --optimize-config optimize-mcp.yaml \
  --fileset hermes-optimize-mcp \
  --workspace default
nemo agents optimize \
  --optimize-config-fileset default/hermes-optimize-mcp \
  --optimize-config optimize-mcp.yaml \
  --workspace default
```

#### Python SDK

```python
import os
from pathlib import Path

from nemo_optimization.jobs.optimize import OptimizeJob
from nemo_platform import NeMoPlatform
from nemo_platform_plugin.scheduler import NemoJobScheduler

WORKSPACE = "default"
agent_root = Path(
    os.environ.get(
        "PHISHING_AGENT_ROOT",
        Path.home() / "work/email-phishing-analyzer-harnesses",
    )
)
os.environ.setdefault("PHISHING_AGENT_SRC", str(agent_root / "src"))
os.environ.setdefault(
    "PHISHING_MCP_BIN",
    str(agent_root / ".venv/bin/email-phishing-analyzer-mcp"),
)

bundle = Path("plugins/nemo-optimization/examples/hermes-optimize").resolve()
optimize_config = bundle / "optimize-mcp.yaml"
os.chdir(bundle)  # dataset / base_dir are relative to the bundle

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace=WORKSPACE,
)

result = NemoJobScheduler().run_local(
    OptimizeJob,
    {
        "optimize_config": str(optimize_config),
        "workspace": WORKSPACE,
    },
    workspace=WORKSPACE,
    sdk=client,
)
print(result)
```

When `--agent` is a platform-managed agent name, the job fetches the stored
Fabric agent config, overlays the optimization settings, runs Inference Gateway
model preflight, and dispatches to the Tune backend. `--agent` must be a
workspace agent name (`hermes-optimize-chatonly` or
`default/hermes-optimize-chatonly`). Endpoint URLs and other URI forms
(`http://...`, `https://...`, `file://...`) are rejected -- raw HTTP endpoint
optimize mode was removed. Use a platform-managed agent reference or an inline
Fabric agent package in `--optimize-config`.

## Run a Study on the Platform

`nemo agents optimize` hands the study to the platform's Jobs service, which
runs it on the platform host or cluster and therefore **cannot read your
filesystem**. The command accepts no host paths: it takes a fileset holding the
whole optimize bundle, plus a config path relative to that fileset's root.

### 1. Stage the bundle

```bash
nemo agents optimize prepare-fileset \
  --source "$BUNDLE" \
  --optimize-config optimize-chatonly.yaml \
  --fileset hermes-optimize-chatonly \
  --workspace default
```

`prepare-fileset` validates before it uploads, and prints the matching
`nemo agents optimize` command on success. It checks that:

* the YAML parses and enables an optimizer with a non-empty search space;
* there is an Agent under Test — either an inline
  `schema_version: fabric.agent/v1alpha1` package or a resolvable `--agent`;
* every path the config references (dataset, `eval.fabric.base_dir`,
  `eval.run_hook.path` / `agent_src`, MCP `config_paths`) is relative and
  present under `--source`;
* no absolute host path would be shipped to the worker.

Use `--dry-run` to validate without uploading, and `--no-check-models` to skip
resolving the config's models against the platform.

The upload is recursive. Delete local run output (`artifacts/`, `.tmp/`) from
the bundle before staging.

### 2. Run

```bash
nemo agents optimize \
  --optimize-config-fileset default/hermes-optimize-chatonly \
  --optimize-config optimize-chatonly.yaml \
  --workspace default
```

`--optimize-config-fileset` is required: passing an absolute host path fails at
compile time with a pointer back to `prepare-fileset`. Add
`--output <fileset-or-dir>` to publish the study's artifacts (optimized config,
trials dataframe, ATIF evidence) somewhere addressable when it finishes.

### Where the study runs

`OptimizeJob` picks its executor from the execution profiles the platform
actually registered, for the requested profile (currently always `default`):

| Registered profile                 | Executor                                                             | What the deployment must provide                                                                                                |
| ---------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `subprocess`                       | Host subprocess running `python -m nemo_optimization.tasks.optimize` | The jobs host venv must carry the optimization and Fabric harness dependencies (and the Docker socket, if trials need it)       |
| `cpu` (docker or `kubernetes_job`) | The `nmp-cpu-tasks` image, same entry point                          | The task image must carry those dependencies; `nemo-optimization-plugin` is in the `cpu-tasks` dependency group for this reason |

Subprocess wins when both are registered, because a study drives Fabric trials
that often need the host's Docker daemon and harness adapters. If neither is
registered under the profile, compile fails and lists what is available.

## Troubleshooting

**No suggestions appear.** Confirm the workspace has agents, model entities, and a model catalog entry smaller than the agent's current model. New-model suggestions require a previous optimizer snapshot, so they do not appear on the first run.

**The model evaluation fails.** Confirm the judge model in the eval config is available through the workspace Inference Gateway. You can replace the eval files in `<agent-name>-eval` with your own evaluation config and dataset.

**Data safety suggestions do not appear.** Telemetry is optional. The optimizer only scans `nemo-agent-telemetry` when that fileset exists and contains JSONL trace files.

## Next steps

* [Agent overview](/documentation/agents): review how platform-managed agents are registered, deployed, invoked, evaluated, and optimized.
* [Agent evaluation](/documentation/evaluate-models/metrics/agent-configuration): configure agents as online evaluation targets and choose the right agent response mapping.
* [CLI reference](/documentation/reference/cli-reference/full-cli-reference): look up complete command options and global CLI flags for scripted workflows.