DocumentationReferenceRelease Notes

v0.4.0

View as Markdown

NeMo Platform v0.4.0 introduces a research-preview optimization loop for improving agents from real telemetry. Agent owners can send traces to Intake, use agents to diagnose recurring issues, create targeted evaluation coverage, experiment against a local code base, and review validated candidate changes. The release also moves new agent workflows toward agent.yaml packages backed by Fabric while keeping NAT workflows available as a legacy path.

Highlights

  • Research-preview agent optimization. Analyst and Experimentalist workflows now support the observe, diagnose, experiment, and evaluate loop for agents under active development.
  • Trace-driven diagnosis. NeMo Intake ingests OTLP, chat-completions, and ATIF telemetry, stores traces in ClickHouse, and attaches annotations and evaluator results so agents and humans can investigate failures from the same evidence.
  • Validated candidate changes. Experimentalist can run against a local agent source tree or git source, evaluate candidates with Harbor, write optimization artifacts locally, and open a draft PR when a changed winner is found.
  • Fabric-first agents. New Platform-managed agents use the nemo-agents-spec-v1 agent.yaml contract. Fabric-backed agents can run through supported harnesses and route model traffic through the Inference Gateway; NAT remains supported for existing workflows.
  • Experiments for review. NeMo Experiments compares evaluation runs using cost, latency, token, and evaluator-score rollups computed from Intake telemetry, with Studio leaderboard and drill-down views behind the Experiments feature flag.
  • Customizer DPO. Customizer adds DPO training through the rl backend, powered by NeMo-RL and Ray on Kubernetes, with backend-specific hyperparameter guidance for Automodel, Unsloth, and RL jobs.

What’s included

Insight-Driven Optimization

  • nemo agents analyst scans Intake traces, evaluator scores, and reviewer annotations to produce evidence-backed Insights.
  • nemo agents experimentalist turns an Insight, or a dataset-only objective, into candidate code changes, train/validation evaluations, local artifacts, and an optional draft PR or MR.
  • optimizer.yaml provides the shared per-agent profile for the loop, including the agent name, workspace, agent source, AGENT-SPEC.md, datasets, task template, and experiment configuration.
  • AGENT-SPEC.md remains the durable Markdown contract for intended agent behavior and can be consumed by the Analyst and Experimentalist when present.
  • Benchmark and proof assets include Terminal-Bench 2.1, Tau3 Airline, Banking, Retail, and Telecom suites, plus a guided Tau3 example-agent walkthrough.

Intake and Experiments

  • Intake supports OTLP, chat-completions, and ATIF ingest paths for agent telemetry.
  • Traces, spans, sessions, annotations, and evaluator results are queryable by API and reviewable in Studio.
  • Local setup can automatically provision and reuse a local ClickHouse container for Intake unless an external ClickHouse URL is configured.
  • The Helm chart includes an embedded ClickHouse option for development and non-critical single-node installations, plus configuration for external ClickHouse in production-oriented deployments.
  • Experiments and Evaluations are durable entities whose leaderboard metrics are computed from retained Intake telemetry.
  • The nemo-experiments-upload skill helps coding agents publish evaluation runs and verify rollups.

Agents and Fabric

  • New agents use Platform-managed agent.yaml files with the nemo-agents-spec-v1 config format.
  • NeMo Fabric is the preferred runtime wrapper for new agents, with support for harness adapters, shared model bindings, skills, MCP servers, tool policy, and Relay/ATIF/ATOF telemetry configuration.
  • nemo agents create, deploy, invoke, run, and package support the Fabric-backed agent path.
  • Fabric agent artifacts can be staged into Docker and Kubernetes deployments.
  • Streaming responses are supported through Fabric-backed agents.
  • Legacy NAT workflow YAMLs continue to work for existing NAT-specific evaluation and deployment workflows.

Agent Evaluation

  • The evaluator SDK adds Fabric runtime support, Docker Compose sandboxing, native agent-eval aggregation, pass-at-k scoring, typed run metadata, and task/taskset revisions.
  • Agent evaluations can publish results to Intake so Experiments and Insights can consume the same run telemetry.

Customizer

  • The rl customization backend supports full-weight DPO jobs through NeMo-RL.
  • DPO schemas expose schedule, optimizer, batch, DPO-specific loss, gradient, activation-checkpointing, and parallelism controls.
  • Customizer skills and references now cover backend-specific job JSON for Automodel, Unsloth, and RL, including dataset formats, batch sizing, integrations, and troubleshooting.
  • Customizer adds local Optuna HPO workflows for tuning hyperparameters from a developer workstation.

Studio

  • Studio includes primary surfaces for Agents, Data Designer, Guardrails, Safe Synthesizer, Intake traces, Optimizer Insights, Experiments, and Jobs, with Customizer and Anonymizer surfaces available behind their feature flags.
  • Guardrails in Studio can create and delete configurations and manage guardrail test cases through the checks surface.
  • Data Designer adds more seed sources, AI-assisted job configuration, preview, and file transform workflows.
  • Safe Synthesizer reports include score-driven gauges.
  • Studio can load plugin web bundles through /apis/plugins and render them inside the Studio React tree with trusted bundle URL checks.
  • NeMo Studio Assistant adds a packaged agent, chat history, tool-call rendering, and reasoning display for Studio-assisted workflows.

Platform, CLI, and Deployment

  • Early access: Agent deployment to an OpenShell Gateway with customizable network policies (default-deny).
  • Deployment readiness now gates on workload reachability for container-backed deployments.
  • Optional scoped access keys are available for authenticated deployments.
  • The CLI adds machine-readable output controls, Intake and Experiments command exposure, anonymous usage telemetry, and a Telemetry and Privacy reference page.
  • Python 3.12 is now the minimum supported Python version for the source checkout and nemo-platform distribution.
  • Auditor adds a blocking submit option and an aggregated artifacts endpoint.

Install

For a fresh local checkout:

git clone https://github.com/NVIDIA-NeMo/nemo-platform.git
cd nemo-platform
make bootstrap
source .venv/bin/activate
nemo setup

See Setup for prerequisites and provider configuration.

For self-managed Kubernetes, start with Install NeMo Platform Helm Chart.

Upgrade from v0.3.x

From an existing local checkout:

git fetch
git checkout main
make bootstrap
source .venv/bin/activate
nemo setup

After setup, restart local services before using CLI, SDK, Studio, or plugin workflows against the upgraded checkout.

Compatibility

  • Python 3.12-3.13
  • macOS and Linux for local CLI, SDK, Studio, and hosted-provider workflows
  • Linux x86_64 for local NVIDIA GPU workloads
  • CUDA 13-capable NVIDIA drivers for local GPU workloads
  • Self-managed Kubernetes clusters deployed with Helm
  • Docker for local services, Docker-backed platform jobs, local ClickHouse, Docker Sandboxes, and local model-serving workflows
  • ClickHouse for Intake trace storage and Experiments rollups
  • NVIDIA GPU access for local training, model serving, and GPU-backed synthetic data workflows
  • Node 22.18.0+ for Studio assets
  • Platform API and nemo-platform Python SDK 0.4.0

Current constraints

  • Research-preview optimizer. Analyst and Experimentalist are early agentic workflows. They are useful for guided optimization, but quality and autonomy are still research-preview and require developer review.
  • Local-first issue-driven loop. The full Insight to Experimentalist loop runs from a local developer environment against local paths or a git source. A remote platform can provide Intake and entity APIs, but the issue-driven optimization agents do not yet run as a horizontally scaled cluster service.
  • Evaluation backend scope. The Experimentalist loop validates candidates with Harbor-compatible train and validation datasets. Other evaluator backends are not the validated path for this loop yet.
  • Optuna HPO runtime. Optuna-based hyperparameter optimization runs locally from a developer workstation. It is not available as a horizontally scaled cluster service in this release.
  • DPO runtime. Customizer rl DPO jobs run on Kubernetes/Ray and do not have a local Docker fallback. DPO is full-weight only; PEFT/adapter output is not supported for RL jobs.
  • Experiments feature flag. The Experiments Studio surface is gated by VITE_FF_EXPERIMENT, which is off by default.
  • Embedded ClickHouse scope. Helm’s embedded ClickHouse is intended for development, evaluation, and non-critical single-node deployments. Use an externally managed ClickHouse for production deployments that require high availability, backups, or larger scale.
  • Python support. Python 3.11 is no longer part of the supported source checkout or nemo-platform distribution matrix.

Known issues

  • Deep Agents trials can be scored as failures. Agent-eval trials run through the NeMo Fabric Deep Agents adapter can be recorded as failures even when the agent completed successfully. This will be fixed in the next release.