> This page is for NeMo Relay, version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v0.11.2) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt
> - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt
> - v0.11.2: https://docs.nvidia.com/nemo/relay/v0.11.2/llms.txt
> - v0.10.4: https://docs.nvidia.com/nemo/relay/v0.10.4/llms.txt
> - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt
> - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt
> - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.

# Connecting to Observability Platforms

> Export Relay telemetry from your existing deployment and verify delivery to a tracing platform.

Enterprise teams need to investigate agent activity in the observability platform
they already operate. This guide connects a
[NVIDIA NeMo Relay](https://github.com/NVIDIA/NeMo-Relay) capture path to that platform
and verifies that agent or turn activity, model calls, and tools arrive together.
It applies to managed coding agents, framework integrations, and instrumented applications.

The optional [NVIDIA Brev](https://brev.nvidia.com/) example uses
[Nous Research Hermes Agent](https://github.com/NousResearch/hermes-agent) with its
native Relay integration and [Arize Phoenix](https://arize.com/phoenix/).
Hermes reads synthetic incident requests, uses model and file tools to prioritize
them, and writes a handoff. These products are example choices; the export process
applies to any Relay capture path.

> **Try the complete example on Brev**
>
> [Run the complete example](#run-the-brev-example) to capture a Hermes agent task
> and verify its traces in Phoenix.
>
> **What is Brev?**
>
> [NVIDIA Brev](https://brev.nvidia.com/) provides cloud development machines you
> can access from your local tools, so you can try the example without installing
> the agent stack on your workstation.

## Before you start

| Requirement | Your deployment                                               | Brev example                                                                    |
| ----------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Relay       | A capture path that sees the required model and tool activity | Hermes native integration with Relay 0.11.0, managed by Hermes                  |
| Host        | A platform supported by your chosen integration               | CPU Linux VM, Python 3.14, Git, uv                                              |
| Access      | Trace-ingestion permission and any provider credentials       | NVIDIA API key and an accessible model; local Phoenix needs no API key          |
| Changes     | Exporter configuration in the process that owns capture       | Dedicated Hermes installation and home, exporter policy, and output directories |
| Cleanup     | Retain required audit evidence before removal                 | Stop Phoenix and remove only the example directories you created                |

> **Keep examples isolated**
>
> Use a fresh host without existing system-wide Relay policy. That policy also
> applies to Hermes, SDK, and framework processes, including explicit plugin files.

## 1. Choose the Relay capture path

Choose the process that observes your workload. Configure its exporter there.

#### [Managed coding agents](/nemo/relay/dev/daemon/about)

A daemon brokers supported harness connections. Per-user workers capture routed
model calls and supported hooks in processes running as those users; install
the exporter policy in the workers.

#### [Framework integration](/nemo/relay/dev/supported-integrations/about)

LangChain, LangGraph, and Deep Agents integrations observe supported model and
tool boundaries inside the application. Configure that application's Relay host.

#### [Instrumented application](/nemo/relay/dev/instrument-applications/about)

Python, Node.js, or Rust code creates agent scopes and wraps the model and tool
calls it owns.

#### [Hermes native integration](/nemo/relay/dev/getting-started/quick-start/hermes)

Hermes initializes Relay in process. Select its plugin configuration through
HERMES_NEMO_RELAY_PLUGINS_TOML; a separate Relay CLI gateway is unnecessary.
The Brev example uses this path.

For an existing deployment, finish the selected integration setup, then return
here once you can capture one agent or turn scope, one model call, and one tool call.
An exporter sends existing events; it does not add instrumentation. The
[support matrix](/nemo/relay/dev/reference/support-matrix) describes each integration's coverage.

## 2. Enable the plugin and select a destination

Place the following `plugins.toml` in the configuration loaded by that worker
or application. This minimal example sends OpenInference traces to local Phoenix
and records Agent Trajectory Observability Format (ATOF) events locally for
capture verification. It omits full prompt and tool payloads from exported traces:

```toml
version = 1

[[components]]
kind = "observability"
enabled = true

[components.config]
version = 4
enable_full_payloads = false

[components.config.atof]
enabled = true

[[components.config.atof.sinks]]
type = "file"
output_directory = "./logs"
filename = "events.jsonl"
mode = "append"

[components.config.opentelemetry]
enabled = true

[[components.config.opentelemetry.endpoints]]
type = "openinference"
endpoint = "http://127.0.0.1:6006/v1/traces"
transport = "http_binary"
service_name = "relay-support-triage"

[components.config.opentelemetry.endpoints.resource_attributes]
"openinference.project.name" = "relay-support-triage"
```

Choose a writable log directory private to the exporting user or application.
Local ATOF records can contain sensitive data even when full payload export is
disabled; apply your retention and access policy to `logs/events.jsonl`.

| Capture path             | Where to configure plugins                                                                                                    |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| Managed workers          | Administrator-owned `/etc/nemo-relay/plugins.toml` on Linux and macOS, or `%ProgramData%\nemo-relay\plugins.toml` on Windows. |
| Framework integration    | The plugin file selected when the integration initializes its Relay host.                                                     |
| Instrumented application | Pass the file path to the SDK's plugin activation API, as below.                                                              |
| Hermes                   | Set `HERMES_NEMO_RELAY_PLUGINS_TOML` to the plugin file before starting Hermes.                                               |

An explicit SDK plugin file adds a configuration layer; system plugin policy still
applies. Before upgrading an existing deployment to Relay 1.0, migrate worker
plugins to the current protocol and check native-plugin compatibility. See the
[migration guide](/nemo/relay/dev/reference/migration-guides#migrate-worker-plugins-to-nemo-relay-100).

For a directly instrumented Python application, import the observability
component and activate the host around application work:

```python
from nemo_relay import plugin
import nemo_relay.observability

async with plugin.activate(plugin.PluginConfig(), "plugins.toml"):
    await run_application()
```

Restart or reopen the affected process after changing its configuration.
Check activation diagnostics before testing delivery.
[Plugin initialization](/nemo/relay/dev/configure-plugins/initialize-plugins) explains the lifecycle.

Choose the destination recipe for the endpoint, projection, and authentication
changes. A projection maps Relay events to the destination's telemetry format.
Credentials belong in the exporting process's environment;
`header_env` maps header names to environment-variable names.

#### [Phoenix](/nemo/relay/dev/user-guides/connecting-to-observability-platforms/phoenix)

OpenInference projects with agent, model, and tool spans.

#### [LangSmith](/nemo/relay/dev/user-guides/connecting-to-observability-platforms/langsmith)

Send OpenInference traces to a project or agent environment.

#### [W\&B Weave](/nemo/relay/dev/user-guides/connecting-to-observability-platforms/weave)

Send GenAI spans to Agents and verify conversation grouping for multi-turn work.

#### [Datadog](/nemo/relay/dev/user-guides/connecting-to-observability-platforms/datadog)

Export OTLP through your Agent or Collector to the appropriate Datadog site.

## 3. Exercise a representative workload

Run work with a clear expected outcome. Keep the service name, run time, and
local event-log path so you can find the same activity in the destination. Include
an agent or turn scope (the parent work unit), a model call, and a tool call.
For a concrete workload, use the [Brev example below](#run-the-brev-example).
Approve payload fields and redaction policy before enabling payload capture.

## 4. Verify capture and destination delivery

Accept the integration when all these checks pass:

| Check               | Evidence                                                                                    |
| ------------------- | ------------------------------------------------------------------------------------------- |
| Correct destination | The intended project and service contain the run at its recorded time.                      |
| Captured activity   | Expected agent or turn, LLM, and tool spans exist for the chosen capture path.              |
| Correlation         | Observed calls share the trace and have the expected parents; exported parents are present. |
| Useful fields       | Model identity and usage appear when the capture path supplies them.                        |
| Privacy             | Restricted payloads are absent or redacted according to the approved policy.                |

Local ATOF records prove capture. Inspect the destination tree to prove delivery;
an HTTP success or a closed model scope alone is insufficient.

## Next steps

Apply the same exporter settings to your chosen capture path. Use
[exporter diagnostics](/nemo/relay/dev/configure-plugins/observability/configuration) to distinguish
capture gaps, authentication failures, and delivery problems.

---

## Run the Brev example

Example

Hermes Agent

Arize Phoenix

Follow this section from start to finish. It installs Hermes’s native integration
and supplies the exporter configuration; the earlier manual configuration
steps explain how to adapt your own deployment. Supporting links are optional.

### 1. Provision and connect to a VM

In the [Brev Console](https://brev.nvidia.com/), create a fresh instance using
these recommended starting resources. See [Brev provisioning](https://docs.nvidia.com/brev/cli/instance-creation)
for instance selection; available CPU types depend on your organization and provider.

| Setting          | Recommendation                                     |
| ---------------- | -------------------------------------------------- |
| Operating system | Ubuntu 22.04 or 24.04 LTS, x86_64                  |
| CPU and memory   | 4 vCPUs and 16 GB RAM                              |
| Storage          | At least 40 GB                                     |
| GPU              | None; inference runs on the model provider         |
| Access           | Administrator account with sudo and outbound HTTPS |

These are starting resources for the example, not Relay sizing requirements.
Use a separate VM from the other examples, because system-wide plugin policy
can affect other Relay processes on the same host.

On your **local computer**, install the Brev CLI if needed. On macOS:

```bash
brew install brevdev/homebrew-brev/brev
```

For other operating systems, use the [Brev CLI installation instructions](https://docs.nvidia.com/brev/getting-started/quickstart).
Connect to the instance you created:

```bash
brev login
brev refresh
brev list
brev shell YOUR_INSTANCE
```

Run the remaining setup in **Bash on the VM host**.

### 2. Install the runtime and run the workload

```bash
sudo apt-get update
sudo apt-get install -y ca-certificates curl git xz-utils
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
uv python install 3.14
python3.14 --version
uv --version
git --version
```

Install the pinned Hermes revision with its own installer, using a dedicated
home for the example:

```bash
HERMES_COMMIT=95fc3e668fc1e13ad05d22a4f02fccc6a6d6ce02
export HERMES_HOME="$HOME/.hermes-relay-observability"
curl -fsSL "https://raw.githubusercontent.com/NousResearch/hermes-agent/$HERMES_COMMIT/scripts/install.sh" -o /tmp/install-hermes-relay.sh
bash /tmp/install-hermes-relay.sh --commit "$HERMES_COMMIT" \
  --hermes-home "$HERMES_HOME" --skip-setup --skip-browser --skip-computer-use
export PATH="$HOME/.local/bin:$PATH"
hermes --version
hermes config set terminal.backend local
hermes config set terminal.cwd .
```

This [Hermes revision](https://github.com/NousResearch/hermes-agent/tree/95fc3e668fc1e13ad05d22a4f02fccc6a6d6ce02)
uses Python 3.14 and declares Relay `>=0.11,<0.12`. Keep its supported dependency;
do not install a different Relay SDK over it. The browser and computer-use
components are optional and are omitted from this local file task. The terminal
configuration keeps the task on the VM host.

Get the companion files:

```bash
git clone --depth 1 https://github.com/NVIDIA/NeMo-Relay.git relay-observability
cd relay-observability/docs/user-guides/_examples/observability
```

Inspect the exporter configuration before the first run:

```bash
python3.14 run.py --destination phoenix --output /tmp/relay-triage-config --prepare-only
HERMES_NEMO_RELAY_PLUGINS_TOML=/tmp/relay-triage-config/plugins.toml hermes doctor
```

In **NeMo Relay Plugins**, confirm that the selected file is listed, plugins are
enabled, and no Relay configuration errors appear. Other optional-service warnings
do not require enabling those services. The runner selects a new exporter file
for each actual run.

In a second terminal on the same host, start disposable Phoenix:

```bash
PHOENIX_HOST=127.0.0.1 uvx --from arize-phoenix==21.0.0 phoenix serve
```

Back in the example directory, select a tool-capable model from the
[NVIDIA API catalog](https://build.nvidia.com/explore/discover) and enter its API key:

```bash
export NVIDIA_BASE_URL=https://integrate.api.nvidia.com/v1
export NVIDIA_MODEL=YOUR_ACCESSIBLE_MODEL_ID
IFS= read -r -s -p 'NVIDIA API key: ' NVIDIA_API_KEY; printf '\n'
export NVIDIA_API_KEY
python3.14 run.py --provider nvidia --model "$NVIDIA_MODEL" \
  --destination phoenix --output /tmp/relay-triage --verify
```

Hermes uses its native Relay integration to capture its model and tool calls.
The runner selects the exporter file, validates the handoff and local parentage,
and checks Phoenix delivery. Failed runs are preserved for inspection.

To use another working Hermes provider, omit `--provider` and `--model` and keep
your existing provider configuration. Configure one with `hermes model` if needed.
The [Hermes Quick Start](/nemo/relay/dev/getting-started/quick-start/hermes) explains provider
setup and the NeMo Relay Plugins diagnostics in `hermes doctor`.
For a hosted destination, use that recipe's environment variables and
`--destination langsmith`, `weave`, or `datadog`; `--verify` is Phoenix-specific.
Use a fresh Hermes home for each destination or changed plugin policy.

In another terminal on your **local computer**, forward the Phoenix UI:

```bash
brev port-forward YOUR_INSTANCE --port 6006:6006
```

Open `http://localhost:6006` and find project and service `relay-support-triage`.
The runner should report successful workload and trace verification. In the trace
tree, confirm the agent, model, and tool spans have their expected parents.
Inspect `logs/events.jsonl` and `delivery.json` under the chosen output directory
if verification fails. Use a new output directory for every run.

#### Expected trace

Find the Hermes agent or session scope and follow its model and tool activity.
Tool names and call counts depend on the agent's choices:

| Activity          | Expected evidence                                                      |
| ----------------- | ---------------------------------------------------------------------- |
| Agent             | A closed Hermes agent scope anchors the run.                           |
| Read the queue    | A tool reads `requests.csv`.                                           |
| Model work        | One or more model calls interpret the incidents and choose actions.    |
| Write the handoff | A tool creates `handoff.md` with every incident ID.                    |
| Correlation       | Captured model and tool scopes connect to the agent in the same trace. |

### 3. Check synthetic redaction

Use only the supplied synthetic fixture. This second run explicitly enables
payloads and an email redaction profile. Use a separate Hermes home for this
policy so it starts an independent gateway and plugin lifecycle. Keep Phoenix
running while gateways finish exporting:

```bash
hermes gateway stop
export HERMES_HOME="$HOME/.hermes-relay-redacted"
hermes config set terminal.backend local
hermes config set terminal.cwd .
python3.14 run.py --provider nvidia --model "$NVIDIA_MODEL" \
  --destination phoenix --output /tmp/relay-triage-redacted --redaction-demo --verify
```

The runner checks that the synthetic contact is absent from emitted payloads.
The source file and provider conversation retain it; redaction affects telemetry.

### 4. Clean up

Retain the run evidence. Stop each example home’s gateway before stopping
Phoenix:

```bash
HERMES_HOME="$HOME/.hermes-relay-observability" hermes gateway stop
HERMES_HOME="$HOME/.hermes-relay-redacted" hermes gateway stop
```

Remove only the dedicated Hermes installation, example homes, and output
directories, or delete the disposable Brev instance.