Run the Agent Optimizer

Use the Agent Optimizer to analyze a deployed agent and act on improvement suggestions. The optimizer inspects the agent’s config, the workspace model catalog, any prior optimizer snapshots, and optional evaluation baselines, then writes suggestions you can review from the CLI or hand off to a coding agent.

This page covers the main path: establish a baseline, generate optimization suggestions, apply a candidate change to a sibling agent, and review the evaluation result before promotion.

What the Optimizer Checks

Optimizer state is stored in the nemo-agent-optimizer fileset:

• optimizer_suggestions.jsonl: one suggestion per line, including applied state.
• optimizer_snapshot.json: model and agent names from the latest run.

Suggestions about missing guardrails are covered in Add Guardrails to an Agent. PII exposure and leaked secrets are covered in Scan Trace Data.

Prerequisites

Before running the optimizer, make sure you have:

1. Local services running (nemo services run).
2. The agents plugin installed. For local development from this repository:

   uv sync --package nemo-agents-plugin
   source .venv/bin/activate   # puts `nemo` on PATH
3. A workspace with at least one model provider and discovered model entities.
4. At least one deployed platform-managed agent.
5. An evaluation baseline before promoting a candidate agent.

If you need a demo agent, start the platform and create the ReAct example:

nemo services run

If nemo setup already selected a default model, skip the NEMO_DEFAULT_MODEL export below—the example uses the model saved in your active context. Otherwise, choose a model from nemo models list and set NEMO_DEFAULT_MODEL. An exported value overrides the context value.

In another terminal:

export NHX_BASE_URL=http://127.0.0.1:8080
cd plugins/nemo-agents
printf '%s' "$NVIDIA_API_KEY" | nemo secrets create ngc-api-key --from-file -
nemo inference providers create nvidia-build \
  --host-url https://integrate.api.nvidia.com \
  --api-key-secret-name ngc-api-key
nemo wait inference provider nvidia-build
# Choose a model entity shown by `nemo models list`.
export NEMO_DEFAULT_MODEL=nvidia-nemotron-3-5-lightning-30b-a3b
nemo agents create \
  --name react-agent \
  --agent-config examples/react-agent/react-agent.yml
nemo agents deploy --agent react-agent
nemo agents deployments wait --agent react-agent

The example agent reads NEMO_DEFAULT_MODEL, so it can produce a model optimization suggestion when the workspace model catalog contains a smaller compatible model.

Optimize with Switchyard Routing

Switchyard is the inference middleware that lets a virtual model split traffic across multiple backend models. The common optimization pattern is to create a virtual model with a strong model and a weaker, cheaper model, then evaluate whether the route split preserves application quality.

Run nemo models list first and replace the placeholders below with model entity names from your workspace that use the OPENAI_CHAT backend format.

nemo inference virtual-models create routed-agent-model \
  --workspace default \
  --models '[
    {"model":"default/<strong-model-entity>","backend_format":"OPENAI_CHAT"},
    {"model":"default/<weak-model-entity>","backend_format":"OPENAI_CHAT"}
  ]' \
  --request-middleware '[{
    "name":"nemo-switchyard",
    "config_type":"random_routing",
    "config":{
      "strong":{"model":"default/<strong-model-entity>"},
      "weak":{"model":"default/<weak-model-entity>"},
      "strong_probability":0.8,
      "enable_stats":false
    }
  }]'

Optimize Skills

Skill optimization applies when the agent depends on local skill files and has an evaluation suite. The loop runs evaluations, analyzes failures, lets the coding agent edit only the configured skills directory, reruns verification, and keeps the change only when the evaluation result improves.

nemo agents optimize-skills --spec-file .agent-improver.yml

Inspect Saved Results

Use the Files service to inspect what the optimizer saved:

nemo files list nemo-agent-optimizer
nemo files download nemo-agent-optimizer \
  --remote-path optimizer_suggestions.jsonl \
  -o optimizer_suggestions.jsonl
nemo files download nemo-agent-optimizer \
  --remote-path optimizer_snapshot.json \
  -o optimizer_snapshot.json

Telemetry is optional. If agents use the nemo_files telemetry exporter, trace files are written to nemo-agent-telemetry, and the optimizer samples the largest JSONL file:

nemo files list nemo-agent-telemetry

Run Prompt and Parameter Tuning

The nemo agents optimize run-strategy command submits Fabric-backed numeric optimization. nemo agents optimize is a shared command group, one per installed optimization strategy; --strategy legacy selects the Fabric-backed numeric study implemented in nemo-optimization. Run nemo agents optimize list-strategies to see what this environment has. Input must be a Fabric-native agent package (schema_version: fabric.agent/v1alpha1). The golden-path harness is Hermes (nvidia.fabric.hermes); see plugins/nemo-optimization/examples/hermes-optimize/ (install steps live in that README).

After uv sync --package nemo-agents-plugin (and activating .venv), invoke nemo directly.

For CLI platform runs, keep the config and everything it references in one directory — an optimize bundle — stage that bundle with prepare-fileset, and pass --optimize-config as a path relative to the staged fileset root. For local Python SDK runs, optimize_config must be an absolute host path; paths inside the YAML (dataset, eval.fabric.base_dir, hook and MCP configs) resolve against your current working directory.

export BUNDLE="$(pwd)/plugins/nemo-optimization/examples/hermes-optimize"
cd "$BUNDLE"

Chat-only Hermes (no MCP)

nemo agents optimize prepare-fileset \
  --source "$BUNDLE" \
  --optimize-config optimize-chatonly.yaml \
  --fileset hermes-optimize-chatonly \
  --workspace default
nemo agents optimize run-strategy \
  --strategy legacy \
  --optimize-config-fileset default/hermes-optimize-chatonly \
  --optimize-config optimize-chatonly.yaml \
  --workspace default

MCP Hermes (phishing analyzer)

The agent calls the email-phishing-analyzer MCP server from the email-phishing-analyzer-harnesses checkout, declared statically like any production MCP server: its console script as url and its key in env (PHISHING_MCP_BIN, PHISHING_ANALYZER_API_KEY; the analyzer calls integrate.api.nvidia.com while the coordinator uses inference-api.nvidia.com). Trials are scored on the judged classification, on the analyzer being called exactly once, and on the email reaching the tool verbatim; the last two are read from the ATIF trajectory. optimize-mcp-mock.yaml runs the same study against a fixture shipped in the bundle that replays recorded analyzer results, so it needs no checkout or analyzer credential. Full steps are in plugins/nemo-optimization/examples/hermes-optimize/README.md.

nemo agents optimize prepare-fileset \
  --source "$BUNDLE" \
  --optimize-config optimize-mcp.yaml \
  --fileset hermes-optimize-mcp \
  --workspace default
nemo agents optimize run-strategy \
  --strategy legacy \
  --optimize-config-fileset default/hermes-optimize-mcp \
  --optimize-config optimize-mcp.yaml \
  --workspace default

When --agent is a platform-managed agent name, the job fetches the stored Fabric agent config, overlays the optimization settings, runs Inference Gateway model preflight, and dispatches to the Tune backend. --agent must be a workspace agent name (hermes-optimize-chatonly or default/hermes-optimize-chatonly). Endpoint URLs and other URI forms (http://..., https://..., file://...) are rejected — raw HTTP endpoint optimize mode was removed. Use a platform-managed agent reference or an inline Fabric agent package in --optimize-config.

Run a Study on the Platform

nemo agents optimize run-strategy hands the study to the platform’s Jobs service, which runs it on the platform host or cluster and therefore cannot read your filesystem. The command accepts no host paths: it takes a fileset holding the whole optimize bundle, plus a config path relative to that fileset’s root.

1. Stage the bundle

nemo agents optimize prepare-fileset \
  --source "$BUNDLE" \
  --optimize-config optimize-chatonly.yaml \
  --fileset hermes-optimize-chatonly \
  --workspace default

prepare-fileset validates before it uploads, and prints the matching nemo agents optimize run-strategy command on success. It checks that:

• the YAML parses and enables an optimizer with a non-empty search space;
• there is an Agent under Test — either an inline schema_version: fabric.agent/v1alpha1 package or a resolvable --agent;
• every path the config references (dataset, eval.fabric.base_dir, stdio MCP server scripts in mcp.servers.*.args) is relative and present under --source;
• no absolute host path would be shipped to the worker.

Use --dry-run to validate without uploading, and --no-check-models to skip resolving the config’s models against the platform.

2. Run

nemo agents optimize run-strategy \
  --strategy legacy \
  --optimize-config-fileset default/hermes-optimize-chatonly \
  --optimize-config optimize-chatonly.yaml \
  --workspace default

--optimize-config-fileset is required: passing an absolute host path fails at compile time with a pointer back to prepare-fileset. Add --output <fileset-or-dir> to publish the study’s artifacts (optimized config, trials dataframe, ATIF evidence) somewhere addressable when it finishes.

Where the study runs

run-strategy does no optimization itself: it resolves --strategy to the installed strategy job and delegates compile to it, so the executor and entry point below are the legacy strategy’s own. That job picks its executor from the execution profiles the platform actually registered, for the requested profile (currently always default):

Subprocess wins when both are registered, because a study drives Fabric trials that often need the host’s Docker daemon and harness adapters. If neither is registered under the profile, compile fails and lists what is available.

Troubleshooting

No suggestions appear. Confirm the workspace has agents, model entities, and a model catalog entry smaller than the agent’s current model. New-model suggestions require a previous optimizer snapshot, so they do not appear on the first run.

The model evaluation fails. Confirm the judge model in the eval config is available through the workspace Inference Gateway. You can replace the eval files in <agent-name>-eval with your own evaluation config and dataset.

Data safety suggestions do not appear. Telemetry is optional. The optimizer only scans nemo-agent-telemetry when that fileset exists and contains JSONL trace files.

Next Steps

• Agent overview: review how platform-managed agents are registered, deployed, invoked, evaluated, and optimized.
• Agent evaluation: configure agents as online evaluation targets and choose the right agent response mapping.
• CLI reference: look up complete command options and global CLI flags for scripted workflows.