Insight-Driven Optimization

Insight-driven optimization automates the agent improvement loop. It turns production traces and evaluation results into autonomous experimentation that continuously improves your application agent. Where Intake gives you observability and Experiments gives you a comparison surface, the optimization agents are the layer that reasons over that telemetry and acts on it.

This is deliberately not an implementation of a single optimization technique. It is a common front door through which many optimization techniques can be distributed over time, all built on the shared flywheel components the platform already provides.

The system introduces one first-class entity and a family of agents that operate on it:

• The Insight is a named, persistent description of a recurring problem in the agent under test (AUT), backed by the traces that provide evidence. The Insight is the unit of action across the whole loop. An experiment or an evaluation-suite change can always be traced back to the Insight that motivated it.
• The Analyst turns raw traces into actionable Insights.
• The Experimenter turns a single Insight into an empirically validated pull request (PR).
• The Eval Author builds and extends evaluation suites in response to real-world usage.

Before You Start

Who This Is For

The optimization agents are for engineers who own and operate an agent and want to automate portions of the agent improvement loop.

Requirements

• A NeMo Platform deployment (local or remote) with Intake enabled, so the Analyst has traces to read.
• The optimization plugins installed in the same Python environment as the NeMo command-line interface (CLI). From a source checkout, uv sync installs all three through the default enabled-plugins group. The Experimenter and Eval Author plugins require Python earlier than 3.14.
• Access to the code base for the agent under test. The Experimenter needs a change surface (source, prompts, tool definitions, model selection, or runtime configuration) and a way to run or evaluate the agent reproducibly.
• Traces in Intake for the agent under test. The Analyst diagnoses only what it can observe, so an agent with no telemetry cannot be analyzed.
• A supported inference provider and default/fast model pair configured by nemo setup for the optimization agents. This access is separate from the model access that your agent under test needs at runtime.
• Train and validation datasets in a Harbor-compatible layout, plus a task template and an output directory for artifacts. This is what the Eval Author works from and what the Experimenter uses to validate changes.

To install only one side of the loop, use the convenience groups:

$
uv sync --only-group insights         # Analyst only
$
uv sync --only-group experimentalist  # Experimenter and Eval Author

Dependencies

The optimization agents build on the following NeMo Platform service dependencies:

• NeMo Intake is the first-party trace store that fuels the loop. The Analyst reads execution traces, evaluator scores, and user feedback from Intake.
• Harbor so the Experimenter can run candidates through an evaluation pipeline and construct Insight-specific suites.
• NeMo Experiments acts as the primary review and persistence surface for candidate evaluation runs.
• Platform entity store (Postgres) is the durable storage for Insights, analysis configs, run status, experiment runs, and candidates.

Compatibility and Access

• All Insights service endpoints are workspace-scoped under /apis/insights/v2/workspaces/{workspace}/ and depend on platform auth. The service checks workspace access before reading or writing.
• The plugins register themselves with the platform through CLI, service, controller, job, software development kit (SDK), and skill entry points. For a fully local platform, restart nemo services run after installation so the platform discovers them.

The Shared Profile

Both halves of the loop read a shared per-agent profile, optimizer.yaml, discovered by walking up from the current directory. The Analyst consumes only its analysis subset (agent, agent_spec, workspace); the Experimenter validates the full schema:

1
agent: research-agent
2
agent_spec: AGENT-SPEC.md        # optional; falls back to AGENT-SPEC.md, then README.md
3
workspace: default               # optional; defaults to "default"
4
agent_source: .                  # local dir or git URL with optional @ref
5
task_template: ./task-template
6
datasets:
7
  train: ./harbor_eval/dataset/train
8
  validation: ./harbor_eval/dataset/validation

Relative paths resolve against the profile. An adjacent .env is loaded when a profile is found, without replacing variables already set in the shell. NMP_BASE_URL is the base-URL environment variable for this workflow, and --base-url takes precedence over it.

How It Works

The Loop

The optimization agents automate the four steps of the agent iteration loop:

1. Observe. The AUT is exercised and emits traces (and optional evaluator scores or feedback) into Intake.
2. Diagnose. The Analyst crawls those traces, clusters failures, and writes or updates Insights.
3. Experiment. The Experimenter picks up a single Insight, hypothesizes root causes, and generates candidate changes.
4. Validate. Candidates are scored on a training split and validated against a held-out split. The winner becomes a draft PR, and the evaluation runs are recorded in NeMo Experiments.

When to Use It

Use the optimization agents when you want to:

• Diagnose real usage automatically. Convert a pile of traces and evaluator scores into a ranked, evidence-backed set of failure patterns.
• Turn a diagnosis into a validated change. Hand an Insight to the Experimenter and get back a candidate PR with evaluation results, rather than a hand-written ticket.
• Run the loop continuously. Opt an agent into periodic analysis so Insights stay current as new traffic arrives.
• Keep evaluation suites relevant. Grow datasets and metrics in response to the failures your agent actually exhibits.

Production traffic provides useful evidence. New agents can start with staged replays or generated cases and improve coverage as production traffic arrives.

Core Concepts and Data Model

The Insight

An Insight is the first-class entity of the loop. It is a persistent, named description of a recurring problem that is stored in the platform entity store with these fields:

The store assigns id, created_at, and updated_at. The Analyst aims for at least three representative traces as evidence before filing a new Insight, and when it finds more evidence for an existing Insight it appends trace refs rather than restating the problem.

Insight Persistence: Platform Versus Local File

When no profile is discovered, the Analyst reads and writes Insights through the Insights application programming interface (API). When a profile governs the run, the Analyst reads and writes the shared local file at <profile-dir>/.nemo-optimizer/insights.yaml instead, which is the same default the Experimenter reads. Pass --insights-file-output <path> to point at a different file explicitly. Trace and feedback reads still hit the live platform at --base-url either way, so the local file also covers deployments that host Intake data but do not have the Insights plugin installed. Each run merges into the file (de-duplicating trace refs) rather than overwriting it.

Telemetry Hierarchy

The loop inherits Intake’s model: a span is one timed operation (LLM call, tool call, and so on), a trace is one end-to-end run, and a session groups related traces. Insight evidence is cited at the trace level.

Experiment Entities

An optimization run is tracked as an ExperimentRun (agent, insight, configuration snapshot, status, rounds completed, winner, summary).

The Agents

Analyst

The Analyst reads telemetry from Intake and emits Insights. It runs as a single reasoning agent with a set of read-only tools over Intake:

• fetch_spans: Survey spans, either grouped (for example by session_id, to fan out across many runs) or flat (to drill into one session). Filters include agent, status, span kind, model, provider, tool name, dataset, and time range.
• get_span: Fetch a single span by ID.
• fetch_scores: Read evaluator results (verifier and judge outputs) attached to a span.
• fetch_annotations and get_annotation: Read feedback, labels, notes, and metadata. Negative feedback is the strongest starting signal.
• list_insights: Read existing Insights so findings are de-duplicated against what is already filed.

Its method is to survey sessions broadly, gather evidence (starting from negative feedback and error spans), cluster similar failures across many sessions, check for existing Insights, then emit a single result containing new Insights and evidence appended to existing ones. Give the Analyst an optional agent spec (--agent-spec AGENT-SPEC.md) so it can flag divergence from intended behavior.

Experimenter

The Experimenter turns an Insight into an empirically validated candidate. Internally it is an evolutionary optimization loop that runs in rounds:

1. Baseline: Build the baseline agent (agent-0) and evaluate it on the validation split; build an initial goal tree (a weighted capability rubric used for trajectory scoring).
2. Analyze: Read the target Insight and perform root cause analysis.
3. Propose: Generate a small number of candidate improvements targeting those root causes, each tagged with an optimization type.
4. Implement: A coding agent applies each proposed change to a copy of the agent and runs a basic integration test with a bounded repair loop.
5. Validate: Score new candidates on the held-out validation split, optionally adding a qualitative trajectory score against the goal tree.
6. Select and continue: Keep a diverse Pareto front of survivors and iterate until a budget or convergence condition is met, then pick the winner.

Insight Mode

The Experimenter starts from a single Insight. By default it reads the local .nemo-optimizer/insights.yaml beside the profile; --insight names another local file or a platform Insight ID. When a local file holds multiple Insights, --insight-id selects one by exact ID, exact title, or zero-based index. The agent referenced by the Insight is used unless --agent overrides it. The Eval Author step builds an Insight-specific evaluation suite before optimization begins, which requires a task template.

Dataset Mode

Pass --no-insight to bypass both an explicit Insight and the profile-local default and optimize directly against a dataset.

Train and Validation Isolation

The validation split is hidden during candidate generation. At run start the validation data is moved into a held-out directory, and the bash tool the coding and analysis agents use blocks reads of that path. Candidates therefore cannot be tuned against the data they are later scored on. Validation data is temporarily restored only when validation scoring runs.

Evaluation and Rewards

Candidates are scored with the Harbor evaluator. Each trial yields metrics from the verifier, and the aggregate reward is the mean across trials, with failed trials counted as zero. Metrics are normalized to [0.0, 1.0], where 1.0 is perfect. Live and production datasets often carry no verifiable reward, so validation on those relies on curated metrics and trajectory scoring rather than a ground-truth verifier.

Output

Output is written under <experiment-dir>/eval-and-optimize/: the run record, per-candidate agent code and metadata, per-round analysis and goal trees, and evaluator results. When configured with a Git source, the Experimenter can archive candidate branches and open a draft PR or merge request (MR) for the winning candidate against the baseline reference.

Eval Author

The Eval Author builds and maintains the evaluation suites that the loop uses. Given an Insight and its evidence traces, it creates an Insight-specific evaluation suite for validating candidates that address the Insight. The Experimenter invokes the Eval Author workflow in Insight mode and reads the eval_author section of the experiment configuration.

The plugin also exposes the canonical nemo agents eval-author command namespace with discover, audit, propose, run, and doctor verbs. These standalone verbs are currently scaffolding and exit with a nonzero status until their implementations are available.

Get Started

Set Up

Point the CLI at a platform and configure model access for the optimization agents. The examples use a local platform; replace the base URL for a remote deployment.

$
export NMP_BASE_URL=http://localhost:8080
$
export WORKSPACE=default
$
export NMP_STUDIO_URL="$NMP_BASE_URL/studio"
$
export NMP_ACCESS_TOKEN="$(nemo auth token)"
$
export AGENT=your-agent-name           # This agent must have traces in Intake.
$
$
nemo setup

During setup, register a supported provider and select the default and fast Platform Model Entities. The default model handles quality-critical work and the fast model handles latency-sensitive work; press Enter at the fast-model prompt to reuse the default. The Analyst, Eval Author, and Experimentalist all use this pair. Provider credentials are stored in Platform Secrets rather than plugin-specific environment variables.

Set NMP_STUDIO_URL explicitly if Studio uses a different origin or path. For an authenticated remote deployment, log in against that base URL before requesting the token. For local development with authentication disabled, omit the NMP_ACCESS_TOKEN command and the Authorization header in the API example.

Confirm the plugins are installed and discoverable:

$
nemo agents analyst --help
$
nemo agents experimentalist --help
$
nemo agents eval-author --help

From an agent directory with an optimizer.yaml profile, check that the effective inputs and credentials resolve:

$
nemo agents analyst doctor
$
nemo agents experimentalist doctor

Send Traces to Intake

The Analyst depends on Intake as its observability store. Your agent must have traces in Intake before it can generate Insights.

Generate Insights

Run the Analyst against the target agent’s traces:

$
nemo agents analyst run \
>
  --agent "$AGENT" \
>
  --workspace "$WORKSPACE" \
>
  --base-url "$NMP_BASE_URL"

Useful flags:

• --agent-spec AGENT-SPEC.md: Append a spec so the Analyst can flag divergence from intended behavior.
• --insights-file-output tmp/insights.yaml: Read and write Insights from a specific local YAML file. Trace reads still hit --base-url.
• --verbose or -v: Stream the Analyst’s tool calls and reasoning to standard error.

Confirm It Worked

The persistence target depends on whether the Analyst discovered an optimizer.yaml profile.

For a profile-based run, confirm that the local Insights file exists and contains the generated records:

$
test -s .nemo-optimizer/insights.yaml
$
sed -n '1,120p' .nemo-optimizer/insights.yaml

If no profile was discovered and you did not pass --insights-file-output, list the Insights stored through the API:

$
curl --fail-with-body \
>
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>
  "$NMP_BASE_URL/apis/insights/v2/workspaces/$WORKSPACE/insights?agent=$AGENT&page=1&page_size=20"

API-stored Insights also appear in Studio at $NMP_STUDIO_URL/workspaces/$WORKSPACE/optimizer. Profile-based Insights do not appear there because they are stored in the local YAML file.

A successful result contains at least one Insight for the agent. Each Insight has a clear title, an actionable description, and trace_refs that point to Intake traces.

Run an Experiment

Convert Evaluations to Harbor

The Experimenter validates its work by running evaluations against your agent. It only keeps empirically validated optimizations. The only supported evaluation backend today is Harbor.

Before you can run the Experimenter, express your agent’s evaluation as Harbor tasks. This is more than just a scorer: it is a container to run the agent, an input, and a verifier that scores the results. A Harbor task directory looks like this:

<task-id>/
  task.toml          # resources, timeouts, and environment variable passthrough
  instruction.md     # the prompt given to the agent, including where to write output
  environment/
    Dockerfile       # a container that can run your agent (and the verifier)
  tests/
    test.sh          # verifier entry point that writes /logs/verifier/reward.json
    score.py         # one or more scorers, such as an LLM-based judge

The Experimenter relies on a classic train/test split for validation, so create two separate directories of Harbor tasks, one for train and one for validation.

Two things to know about the Harbor integration:

• The agent-in-container contract. Harbor runs your agent through WrappedAgent(BaseAgent). Its setup() uploads your agent directory to /app and runs uv sync on it. Its run() executes the agent with the task instruction as --prompt.
• How evaluation scores reach Intake. The Analyst and the Experimenter both read scores from Intake, but neither Harbor nor the verifier pushes scores there; the verifier’s reward is written to local disk by default. The Experimenter uploads scores to Intake after the Harbor run completes, and each verifier result is stored as an evaluation_results record attached to the trace in Intake. Pushing results to Intake requires a platform client, so pass --base-url and --workspace.

Start the Experimenter

Hand it a single Insight:

$
nemo agents experimentalist run \
>
  --insight insight-id \
>
  --agent ./path-to-agent \
>
  --train-dataset ./harbor_eval/dataset/train \
>
  --validation-dataset ./harbor_eval/dataset/validation \
>
  --task-template ./task-template \
>
  --config ./experiment-config.yaml \
>
  --experiment-dir tmp/experiment \
>
  --workspace "$WORKSPACE" \
>
  --base-url "$NMP_BASE_URL"

From an agent directory with an optimizer.yaml profile, most of those flags come from the profile:

$
nemo agents experimentalist run

The Experimenter’s own models come from the default/fast pair selected by nemo setup (refer to Models). Train and validation datasets are required, and they must be local paths for the Harbor evaluator. The loop runs a baseline evaluation, performs root cause analysis, implements changes, then validates and picks a winner.

Review the Results

View the experiment in Studio at $NMP_STUDIO_URL/workspaces/$WORKSPACE/experiment. There you can see the originating Insight and compare evaluation runs across all the candidates.

The Experimenter also writes intermediate artifacts along the way. Those land under <experiment-dir>/eval-and-optimize/, which defaults to <profile-dir>/.nemo-optimizer/experiments/<timestamp> when a profile governs the run, and ./tmp otherwise:

• OPTIMIZATION.md: The report with per-agent breakdowns and round-by-round root causes.
• agents/agent-0: The baseline, with agents/agent-N as the candidates. Each holds full agent source.
• results/: The per-agent evaluation outputs.

Additional Workflows

Opt an Agent into Periodic Analysis

Instead of running nemo agents analyst run manually, opt an agent in and let the platform run the Analyst on a schedule:

$
nemo insights analysis enable  --agent "$AGENT" --workspace "$WORKSPACE"
$
nemo insights analysis status  --workspace "$WORKSPACE"          # omit --agent to list all
$
nemo insights analysis disable --agent "$AGENT" --workspace "$WORKSPACE"

A framework controller (insights-analysis) reconciles on a fixed 60-second loop, finds every agent opted in across workspaces, and submits one Analyst job per agent that is due and has enough new telemetry. Runs are incremental: each successful run records a cursor (last_successful_run_at), and subsequent runs only consider traces newer than that cursor.

enable captures the effective default/fast pair in the server-side analysis config. Re-run it after changing the pair with nemo setup; enabled records created before model-pair persistence must also be re-enabled.

Configure the Analysis Schedule

The global schedule lives in the Insights plugin configuration or environment. The timezone setting uses an Internet Assigned Numbers Authority (IANA) time zone name and defaults to Coordinated Universal Time (UTC).

1
insights:
2
  analyst:
3
    enabled: true            # master switch for the periodic controller
4
    frequency: daily         # daily (default) or weekly
5
    run_at_hour: 0           # local hour-of-day, 0-23
6
    run_on_weekday: monday   # only used when frequency: weekly
7
    timezone: UTC            # IANA name, such as America/Denver
8
    job_profile: default     # jobs execution profile for scheduled runs

run_at_hour is interpreted in timezone and converted to the server clock at evaluation time, so runs fire at the intended local hour even across daylight-saving transitions. The controller still reconciles every 60 seconds and submits a run after the scheduled daily or weekly window is reached.

Command Reference

The Analyst lives under nemo agents analyst, and the Experimenter lives under nemo agents experimentalist. The Eval Author command surface lives under nemo agents eval-author. These command namespaces share the optimizer.yaml profile, and each validates its own configuration. Scheduled analysis remains under nemo insights analysis.

nemo agents analyst run

Run the Analyst for one analysis of an agent’s traces.

nemo insights analysis enable | disable | status

Manage per-agent opt-in for periodic analysis. enable and disable require --agent; status takes an optional --agent (omit it to list all configs in the workspace). enable stores the active default/fast model pair for future scheduled jobs. All three accept --workspace and --base-url.

nemo agents analyst doctor

Check whether the current profile is ready for analysis. Exits non-zero when a required check fails.

nemo agents experimentalist run

Run the local Experimenter loop.

nemo agents experimentalist doctor

Diagnose the Experimenter setup: profile, configured models, Insight resolution, datasets, and the experiment plan.

nemo agents eval-author

Discover the current Eval Author command surface with --help. The discover, audit, propose, run, and doctor verbs are placeholders and exit with a nonzero status until their implementations are available.

Models

The optimization agents use the active Platform CLI context:

Each Model Entity owns its backend format and provider route. OpenAI-compatible and Anthropic entities are adapted to Nooa’s unified completion client through the Platform route; the agents never inspect provider model-name conventions or read provider credentials directly.

For OpenAI Chat Completions entities, the shared client disables reasoning while function tools are active and disables LiteLLM’s automatic Responses bridge. This keeps frontier models on the Model Entity’s declared Chat Completions contract without a model-name allowlist; LiteLLM drops the reasoning parameter for models that do not support it. Anthropic entities use their native Messages route.

Troubleshooting

Analyst Returns No Insights or Aborts on the Trace Floor

The agent has too few or no traces in Intake for the target workspace. Confirm that telemetry is flowing and that you are filtering by the right agent name.

Periodic Analysis Does Not Start

Check that the agent is enabled (nemo insights analysis status), that insights.analyst.enabled is true, that the scheduled window has passed in the configured timezone, and that at least 10 new sessions have landed since the last run. If status shows no model selection, re-run nemo insights analysis enable after nemo setup.

Insights Are Not in the Platform

A discovered optimizer.yaml profile routes Insights to .nemo-optimizer/insights.yaml instead of the Insights API, as does an explicit --insights-file-output. Check that file, or run without a profile to write through the API.

The Experiment Cannot Score or Rewards Are Zero

Confirm datasets are local Harbor-compatible paths. Live and production datasets may lack verifiable rewards by design; rely on curated metrics and trajectory scoring in that case.

No Pull Request Was Opened for the Winner

Publishing requires a git agent source (--agent <git-url>@<ref>) and publishing enabled in the run configuration. A local-directory source cannot open a PR.