Skip to navigation

Connecting to Observability Platforms

View as MarkdownOpen in Claude

Enterprise teams need to investigate agent activity in the observability platform they already operate. This guide connects a NVIDIA NeMo Relay capture path to that platform and verifies that agent or turn activity, model calls, and tools arrive together. It applies to managed coding agents, framework integrations, and instrumented applications.

The optional NVIDIA Brev example uses Nous Research Hermes Agent with its native Relay integration and Arize Phoenix. Hermes reads synthetic incident requests, uses model and file tools to prioritize them, and writes a handoff. These products are example choices; the export process applies to any Relay capture path.

Try the complete example on Brev

Run the complete example to capture a Hermes agent task and verify its traces in Phoenix.

What is Brev?

NVIDIA Brev provides cloud development machines you can access from your local tools, so you can try the example without installing the agent stack on your workstation.

Before you start

RequirementYour deploymentBrev example
RelayA capture path that sees the required model and tool activityHermes native integration with Relay 0.11.0, managed by Hermes
HostA platform supported by your chosen integrationCPU Linux VM, Python 3.14, Git, uv
AccessTrace-ingestion permission and any provider credentialsNVIDIA API key and an accessible model; local Phoenix needs no API key
ChangesExporter configuration in the process that owns captureDedicated Hermes installation and home, exporter policy, and output directories
CleanupRetain required audit evidence before removalStop Phoenix and remove only the example directories you created
Keep examples isolated

Use a fresh host without existing system-wide Relay policy. That policy also applies to Hermes, SDK, and framework processes, including explicit plugin files.

1. Choose the Relay capture path

Choose the process that observes your workload. Configure its exporter there.

For an existing deployment, finish the selected integration setup, then return here once you can capture one agent or turn scope, one model call, and one tool call. An exporter sends existing events; it does not add instrumentation. The support matrix describes each integration’s coverage.

2. Enable the plugin and select a destination

Place the following plugins.toml in the configuration loaded by that worker or application. This minimal example sends OpenInference traces to local Phoenix and records Agent Trajectory Observability Format (ATOF) events locally for capture verification. It omits full prompt and tool payloads from exported traces:

version = 1
[[components]]
kind = "observability"
enabled = true
[components.config]
version = 4
enable_full_payloads = false
[components.config.atof]
enabled = true
[[components.config.atof.sinks]]
type = "file"
output_directory = "./logs"
filename = "events.jsonl"
mode = "append"
[components.config.opentelemetry]
enabled = true
[[components.config.opentelemetry.endpoints]]
type = "openinference"
endpoint = "http://127.0.0.1:6006/v1/traces"
transport = "http_binary"
service_name = "relay-support-triage"
[components.config.opentelemetry.endpoints.resource_attributes]
"openinference.project.name" = "relay-support-triage"

Choose a writable log directory private to the exporting user or application. Local ATOF records can contain sensitive data even when full payload export is disabled; apply your retention and access policy to logs/events.jsonl.

Capture pathWhere to configure plugins
Managed workersAdministrator-owned /etc/nemo-relay/plugins.toml on Linux and macOS, or %ProgramData%\nemo-relay\plugins.toml on Windows.
Framework integrationThe plugin file selected when the integration initializes its Relay host.
Instrumented applicationPass the file path to the SDK’s plugin activation API, as below.
HermesSet HERMES_NEMO_RELAY_PLUGINS_TOML to the plugin file before starting Hermes.

An explicit SDK plugin file adds a configuration layer; system plugin policy still applies. Before upgrading an existing deployment to Relay 1.0, migrate worker plugins to the current protocol and check native-plugin compatibility. See the migration guide.

For a directly instrumented Python application, import the observability component and activate the host around application work:

from nemo_relay import plugin
import nemo_relay.observability
async with plugin.activate(plugin.PluginConfig(), "plugins.toml"):
await run_application()

Restart or reopen the affected process after changing its configuration. Check activation diagnostics before testing delivery. Plugin initialization explains the lifecycle.

Choose the destination recipe for the endpoint, projection, and authentication changes. A projection maps Relay events to the destination’s telemetry format. Credentials belong in the exporting process’s environment; header_env maps header names to environment-variable names.

3. Exercise a representative workload

Run work with a clear expected outcome. Keep the service name, run time, and local event-log path so you can find the same activity in the destination. Include an agent or turn scope (the parent work unit), a model call, and a tool call. For a concrete workload, use the Brev example below. Approve payload fields and redaction policy before enabling payload capture.

4. Verify capture and destination delivery

Accept the integration when all these checks pass:

CheckEvidence
Correct destinationThe intended project and service contain the run at its recorded time.
Captured activityExpected agent or turn, LLM, and tool spans exist for the chosen capture path.
CorrelationObserved calls share the trace and have the expected parents; exported parents are present.
Useful fieldsModel identity and usage appear when the capture path supplies them.
PrivacyRestricted payloads are absent or redacted according to the approved policy.

Local ATOF records prove capture. Inspect the destination tree to prove delivery; an HTTP success or a closed model scope alone is insufficient.

Next steps

Apply the same exporter settings to your chosen capture path. Use exporter diagnostics to distinguish capture gaps, authentication failures, and delivery problems.


Run the Brev example

ExampleHermes AgentArize Phoenix

Follow this section from start to finish. It installs Hermes’s native integration and supplies the exporter configuration; the earlier manual configuration steps explain how to adapt your own deployment. Supporting links are optional.

1. Provision and connect to a VM

In the Brev Console, create a fresh instance using these recommended starting resources. See Brev provisioning for instance selection; available CPU types depend on your organization and provider.

SettingRecommendation
Operating systemUbuntu 22.04 or 24.04 LTS, x86_64
CPU and memory4 vCPUs and 16 GB RAM
StorageAt least 40 GB
GPUNone; inference runs on the model provider
AccessAdministrator account with sudo and outbound HTTPS

These are starting resources for the example, not Relay sizing requirements. Use a separate VM from the other examples, because system-wide plugin policy can affect other Relay processes on the same host.

On your local computer, install the Brev CLI if needed. On macOS:

brew install brevdev/homebrew-brev/brev

For other operating systems, use the Brev CLI installation instructions. Connect to the instance you created:

brev login
brev refresh
brev list
brev shell YOUR_INSTANCE

Run the remaining setup in Bash on the VM host.

2. Install the runtime and run the workload

sudo apt-get update
sudo apt-get install -y ca-certificates curl git xz-utils
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
uv python install 3.14
python3.14 --version
uv --version
git --version

Install the pinned Hermes revision with its own installer, using a dedicated home for the example:

HERMES_COMMIT=95fc3e668fc1e13ad05d22a4f02fccc6a6d6ce02
export HERMES_HOME="$HOME/.hermes-relay-observability"
curl -fsSL "https://raw.githubusercontent.com/NousResearch/hermes-agent/$HERMES_COMMIT/scripts/install.sh" -o /tmp/install-hermes-relay.sh
bash /tmp/install-hermes-relay.sh --commit "$HERMES_COMMIT" \
--hermes-home "$HERMES_HOME" --skip-setup --skip-browser --skip-computer-use
export PATH="$HOME/.local/bin:$PATH"
hermes --version
hermes config set terminal.backend local
hermes config set terminal.cwd .

This Hermes revision uses Python 3.14 and declares Relay >=0.11,<0.12. Keep its supported dependency; do not install a different Relay SDK over it. The browser and computer-use components are optional and are omitted from this local file task. The terminal configuration keeps the task on the VM host.

Get the companion files:

git clone --depth 1 https://github.com/NVIDIA/NeMo-Relay.git relay-observability
cd relay-observability/docs/user-guides/_examples/observability

Inspect the exporter configuration before the first run:

python3.14 run.py --destination phoenix --output /tmp/relay-triage-config --prepare-only
HERMES_NEMO_RELAY_PLUGINS_TOML=/tmp/relay-triage-config/plugins.toml hermes doctor

In NeMo Relay Plugins, confirm that the selected file is listed, plugins are enabled, and no Relay configuration errors appear. Other optional-service warnings do not require enabling those services. The runner selects a new exporter file for each actual run.

In a second terminal on the same host, start disposable Phoenix:

PHOENIX_HOST=127.0.0.1 uvx --from arize-phoenix==21.0.0 phoenix serve

Back in the example directory, select a tool-capable model from the NVIDIA API catalog and enter its API key:

export NVIDIA_BASE_URL=https://integrate.api.nvidia.com/v1
export NVIDIA_MODEL=YOUR_ACCESSIBLE_MODEL_ID
IFS= read -r -s -p 'NVIDIA API key: ' NVIDIA_API_KEY; printf '\n'
export NVIDIA_API_KEY
python3.14 run.py --provider nvidia --model "$NVIDIA_MODEL" \
--destination phoenix --output /tmp/relay-triage --verify

Hermes uses its native Relay integration to capture its model and tool calls. The runner selects the exporter file, validates the handoff and local parentage, and checks Phoenix delivery. Failed runs are preserved for inspection.

To use another working Hermes provider, omit --provider and --model and keep your existing provider configuration. Configure one with hermes model if needed. The Hermes Quick Start explains provider setup and the NeMo Relay Plugins diagnostics in hermes doctor. For a hosted destination, use that recipe’s environment variables and --destination langsmith, weave, or datadog; --verify is Phoenix-specific. Use a fresh Hermes home for each destination or changed plugin policy.

In another terminal on your local computer, forward the Phoenix UI:

brev port-forward YOUR_INSTANCE --port 6006:6006

Open http://localhost:6006 and find project and service relay-support-triage. The runner should report successful workload and trace verification. In the trace tree, confirm the agent, model, and tool spans have their expected parents. Inspect logs/events.jsonl and delivery.json under the chosen output directory if verification fails. Use a new output directory for every run.

Expected trace

Find the Hermes agent or session scope and follow its model and tool activity. Tool names and call counts depend on the agent’s choices:

ActivityExpected evidence
AgentA closed Hermes agent scope anchors the run.
Read the queueA tool reads requests.csv.
Model workOne or more model calls interpret the incidents and choose actions.
Write the handoffA tool creates handoff.md with every incident ID.
CorrelationCaptured model and tool scopes connect to the agent in the same trace.

3. Check synthetic redaction

Use only the supplied synthetic fixture. This second run explicitly enables payloads and an email redaction profile. Use a separate Hermes home for this policy so it starts an independent gateway and plugin lifecycle. Keep Phoenix running while gateways finish exporting:

hermes gateway stop
export HERMES_HOME="$HOME/.hermes-relay-redacted"
hermes config set terminal.backend local
hermes config set terminal.cwd .
python3.14 run.py --provider nvidia --model "$NVIDIA_MODEL" \
--destination phoenix --output /tmp/relay-triage-redacted --redaction-demo --verify

The runner checks that the synthetic contact is absent from emitted payloads. The source file and provider conversation retain it; redaction affects telemetry.

4. Clean up

Retain the run evidence. Stop each example home’s gateway before stopping Phoenix:

HERMES_HOME="$HOME/.hermes-relay-observability" hermes gateway stop
HERMES_HOME="$HOME/.hermes-relay-redacted" hermes gateway stop

Remove only the dedicated Hermes installation, example homes, and output directories, or delete the disposable Brev instance.