> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/dev/user-guides/connecting-to-observability-platforms/llms.txt. # Connecting to Observability Platforms > Export Relay telemetry from your existing deployment and verify delivery to a tracing platform. Enterprise teams need to investigate agent activity in the observability platform they already operate. This guide connects a [NVIDIA NeMo Relay](https://github.com/NVIDIA/NeMo-Relay) capture path to that platform and verifies that agent or turn activity, model calls, and tools arrive together. It applies to managed coding agents, framework integrations, and instrumented applications. The optional [NVIDIA Brev](https://brev.nvidia.com/) example uses [Nous Research Hermes Agent](https://github.com/NousResearch/hermes-agent) with its native Relay integration and [Arize Phoenix](https://arize.com/phoenix/). Hermes reads synthetic incident requests, uses model and file tools to prioritize them, and writes a handoff. These products are example choices; the export process applies to any Relay capture path. > **Try the complete example on Brev** > > [Run the complete example](#run-the-brev-example) to capture a Hermes agent task > and verify its traces in Phoenix. > > **What is Brev?** > > [NVIDIA Brev](https://brev.nvidia.com/) provides cloud development machines you > can access from your local tools, so you can try the example without installing > the agent stack on your workstation. ## Before you start | Requirement | Your deployment | Brev example | | ----------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Relay | A capture path that sees the required model and tool activity | Hermes native integration with Relay 0.11.0, managed by Hermes | | Host | A platform supported by your chosen integration | CPU Linux VM, Python 3.14, Git, uv | | Access | Trace-ingestion permission and any provider credentials | NVIDIA API key and an accessible model; local Phoenix needs no API key | | Changes | Exporter configuration in the process that owns capture | Dedicated Hermes installation and home, exporter policy, and output directories | | Cleanup | Retain required audit evidence before removal | Stop Phoenix and remove only the example directories you created | > **Keep examples isolated** > > Use a fresh host without existing system-wide Relay policy. That policy also > applies to Hermes, SDK, and framework processes, including explicit plugin files. ## 1. Choose the Relay capture path Choose the process that observes your workload. Configure its exporter there. #### [Managed coding agents](/daemon/about) A daemon brokers supported harness connections. Per-user workers capture routed model calls and supported hooks in processes running as those users; install the exporter policy in the workers. #### [Framework integration](/supported-integrations/about) LangChain, LangGraph, and Deep Agents integrations observe supported model and tool boundaries inside the application. Configure that application's Relay host. #### [Instrumented application](/instrument-applications/about) Python, Node.js, or Rust code creates agent scopes and wraps the model and tool calls it owns. #### [Hermes native integration](/getting-started/quick-start/hermes) Hermes initializes Relay in process. Select its plugin configuration through HERMES_NEMO_RELAY_PLUGINS_TOML; a separate Relay CLI gateway is unnecessary. The Brev example uses this path. For an existing deployment, finish the selected integration setup, then return here once you can capture one agent or turn scope, one model call, and one tool call. An exporter sends existing events; it does not add instrumentation. The [support matrix](/reference/support-matrix) describes each integration's coverage. ## 2. Enable the plugin and select a destination Place the following `plugins.toml` in the configuration loaded by that worker or application. This minimal example sends OpenInference traces to local Phoenix and records Agent Trajectory Observability Format (ATOF) events locally for capture verification. It omits full prompt and tool payloads from exported traces: ```toml version = 1 [[components]] kind = "observability" enabled = true [components.config] version = 4 enable_full_payloads = false [components.config.atof] enabled = true [[components.config.atof.sinks]] type = "file" output_directory = "./logs" filename = "events.jsonl" mode = "append" [components.config.opentelemetry] enabled = true [[components.config.opentelemetry.endpoints]] type = "openinference" endpoint = "http://127.0.0.1:6006/v1/traces" transport = "http_binary" service_name = "relay-support-triage" [components.config.opentelemetry.endpoints.resource_attributes] "openinference.project.name" = "relay-support-triage" ``` Choose a writable log directory private to the exporting user or application. Local ATOF records can contain sensitive data even when full payload export is disabled; apply your retention and access policy to `logs/events.jsonl`. | Capture path | Where to configure plugins | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | Managed workers | Administrator-owned `/etc/nemo-relay/plugins.toml` on Linux and macOS, or `%ProgramData%\nemo-relay\plugins.toml` on Windows. | | Framework integration | The plugin file selected when the integration initializes its Relay host. | | Instrumented application | Pass the file path to the SDK's plugin activation API, as below. | | Hermes | Set `HERMES_NEMO_RELAY_PLUGINS_TOML` to the plugin file before starting Hermes. | An explicit SDK plugin file adds a configuration layer; system plugin policy still applies. Before upgrading an existing deployment to Relay 1.0, migrate worker plugins to the current protocol and check native-plugin compatibility. See the [migration guide](/reference/migration-guides#migrate-worker-plugins-to-nemo-relay-100). For a directly instrumented Python application, import the observability component and activate the host around application work: ```python from nemo_relay import plugin import nemo_relay.observability async with plugin.activate(plugin.PluginConfig(), "plugins.toml"): await run_application() ``` Restart or reopen the affected process after changing its configuration. Check activation diagnostics before testing delivery. [Plugin initialization](/configure-plugins/initialize-plugins) explains the lifecycle. Choose the destination recipe for the endpoint, projection, and authentication changes. A projection maps Relay events to the destination's telemetry format. Credentials belong in the exporting process's environment; `header_env` maps header names to environment-variable names. #### [Phoenix](/user-guides/connecting-to-observability-platforms/phoenix) OpenInference projects with agent, model, and tool spans. #### [LangSmith](/user-guides/connecting-to-observability-platforms/langsmith) Send OpenInference traces to a project or agent environment. #### [W\&B Weave](/user-guides/connecting-to-observability-platforms/weave) Send GenAI spans to Agents and verify conversation grouping for multi-turn work. #### [Datadog](/user-guides/connecting-to-observability-platforms/datadog) Export OTLP through your Agent or Collector to the appropriate Datadog site. ## 3. Exercise a representative workload Run work with a clear expected outcome. Keep the service name, run time, and local event-log path so you can find the same activity in the destination. Include an agent or turn scope (the parent work unit), a model call, and a tool call. For a concrete workload, use the [Brev example below](#run-the-brev-example). Approve payload fields and redaction policy before enabling payload capture. ## 4. Verify capture and destination delivery Accept the integration when all these checks pass: | Check | Evidence | | ------------------- | ------------------------------------------------------------------------------------------- | | Correct destination | The intended project and service contain the run at its recorded time. | | Captured activity | Expected agent or turn, LLM, and tool spans exist for the chosen capture path. | | Correlation | Observed calls share the trace and have the expected parents; exported parents are present. | | Useful fields | Model identity and usage appear when the capture path supplies them. | | Privacy | Restricted payloads are absent or redacted according to the approved policy. | Local ATOF records prove capture. Inspect the destination tree to prove delivery; an HTTP success or a closed model scope alone is insufficient. ## Next steps Apply the same exporter settings to your chosen capture path. Use [exporter diagnostics](/configure-plugins/observability/configuration) to distinguish capture gaps, authentication failures, and delivery problems. --- ## Run the Brev example Example Hermes Agent Arize Phoenix Follow this section from start to finish. It installs Hermes’s native integration and supplies the exporter configuration; the earlier manual configuration steps explain how to adapt your own deployment. Supporting links are optional. ### 1. Provision and connect to a VM In the [Brev Console](https://brev.nvidia.com/), create a fresh instance using these recommended starting resources. See [Brev provisioning](https://docs.nvidia.com/brev/cli/instance-creation) for instance selection; available CPU types depend on your organization and provider. | Setting | Recommendation | | ---------------- | -------------------------------------------------- | | Operating system | Ubuntu 22.04 or 24.04 LTS, x86_64 | | CPU and memory | 4 vCPUs and 16 GB RAM | | Storage | At least 40 GB | | GPU | None; inference runs on the model provider | | Access | Administrator account with sudo and outbound HTTPS | These are starting resources for the example, not Relay sizing requirements. Use a separate VM from the other examples, because system-wide plugin policy can affect other Relay processes on the same host. On your **local computer**, install the Brev CLI if needed. On macOS: ```bash brew install brevdev/homebrew-brev/brev ``` For other operating systems, use the [Brev CLI installation instructions](https://docs.nvidia.com/brev/getting-started/quickstart). Connect to the instance you created: ```bash brev login brev refresh brev list brev shell YOUR_INSTANCE ``` Run the remaining setup in **Bash on the VM host**. ### 2. Install the runtime and run the workload ```bash sudo apt-get update sudo apt-get install -y ca-certificates curl git xz-utils curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:$PATH" uv python install 3.14 python3.14 --version uv --version git --version ``` Install the pinned Hermes revision with its own installer, using a dedicated home for the example: ```bash HERMES_COMMIT=95fc3e668fc1e13ad05d22a4f02fccc6a6d6ce02 export HERMES_HOME="$HOME/.hermes-relay-observability" curl -fsSL "https://raw.githubusercontent.com/NousResearch/hermes-agent/$HERMES_COMMIT/scripts/install.sh" -o /tmp/install-hermes-relay.sh bash /tmp/install-hermes-relay.sh --commit "$HERMES_COMMIT" \ --hermes-home "$HERMES_HOME" --skip-setup --skip-browser --skip-computer-use export PATH="$HOME/.local/bin:$PATH" hermes --version hermes config set terminal.backend local hermes config set terminal.cwd . ``` This [Hermes revision](https://github.com/NousResearch/hermes-agent/tree/95fc3e668fc1e13ad05d22a4f02fccc6a6d6ce02) uses Python 3.14 and declares Relay `>=0.11,<0.12`. Keep its supported dependency; do not install a different Relay SDK over it. The browser and computer-use components are optional and are omitted from this local file task. The terminal configuration keeps the task on the VM host. Get the companion files: ```bash git clone --depth 1 https://github.com/NVIDIA/NeMo-Relay.git relay-observability cd relay-observability/docs/user-guides/_examples/observability ``` Inspect the exporter configuration before the first run: ```bash python3.14 run.py --destination phoenix --output /tmp/relay-triage-config --prepare-only HERMES_NEMO_RELAY_PLUGINS_TOML=/tmp/relay-triage-config/plugins.toml hermes doctor ``` In **NeMo Relay Plugins**, confirm that the selected file is listed, plugins are enabled, and no Relay configuration errors appear. Other optional-service warnings do not require enabling those services. The runner selects a new exporter file for each actual run. In a second terminal on the same host, start disposable Phoenix: ```bash PHOENIX_HOST=127.0.0.1 uvx --from arize-phoenix==21.0.0 phoenix serve ``` Back in the example directory, select a tool-capable model from the [NVIDIA API catalog](https://build.nvidia.com/explore/discover) and enter its API key: ```bash export NVIDIA_BASE_URL=https://integrate.api.nvidia.com/v1 export NVIDIA_MODEL=YOUR_ACCESSIBLE_MODEL_ID IFS= read -r -s -p 'NVIDIA API key: ' NVIDIA_API_KEY; printf '\n' export NVIDIA_API_KEY python3.14 run.py --provider nvidia --model "$NVIDIA_MODEL" \ --destination phoenix --output /tmp/relay-triage --verify ``` Hermes uses its native Relay integration to capture its model and tool calls. The runner selects the exporter file, validates the handoff and local parentage, and checks Phoenix delivery. Failed runs are preserved for inspection. To use another working Hermes provider, omit `--provider` and `--model` and keep your existing provider configuration. Configure one with `hermes model` if needed. The [Hermes Quick Start](/getting-started/quick-start/hermes) explains provider setup and the NeMo Relay Plugins diagnostics in `hermes doctor`. For a hosted destination, use that recipe's environment variables and `--destination langsmith`, `weave`, or `datadog`; `--verify` is Phoenix-specific. Use a fresh Hermes home for each destination or changed plugin policy. In another terminal on your **local computer**, forward the Phoenix UI: ```bash brev port-forward YOUR_INSTANCE --port 6006:6006 ``` Open `http://localhost:6006` and find project and service `relay-support-triage`. The runner should report successful workload and trace verification. In the trace tree, confirm the agent, model, and tool spans have their expected parents. Inspect `logs/events.jsonl` and `delivery.json` under the chosen output directory if verification fails. Use a new output directory for every run. #### Expected trace Find the Hermes agent or session scope and follow its model and tool activity. Tool names and call counts depend on the agent's choices: | Activity | Expected evidence | | ----------------- | ---------------------------------------------------------------------- | | Agent | A closed Hermes agent scope anchors the run. | | Read the queue | A tool reads `requests.csv`. | | Model work | One or more model calls interpret the incidents and choose actions. | | Write the handoff | A tool creates `handoff.md` with every incident ID. | | Correlation | Captured model and tool scopes connect to the agent in the same trace. | ### 3. Check synthetic redaction Use only the supplied synthetic fixture. This second run explicitly enables payloads and an email redaction profile. Use a separate Hermes home for this policy so it starts an independent gateway and plugin lifecycle. Keep Phoenix running while gateways finish exporting: ```bash hermes gateway stop export HERMES_HOME="$HOME/.hermes-relay-redacted" hermes config set terminal.backend local hermes config set terminal.cwd . python3.14 run.py --provider nvidia --model "$NVIDIA_MODEL" \ --destination phoenix --output /tmp/relay-triage-redacted --redaction-demo --verify ``` The runner checks that the synthetic contact is absent from emitted payloads. The source file and provider conversation retain it; redaction affects telemetry. ### 4. Clean up Retain the run evidence. Stop each example home’s gateway before stopping Phoenix: ```bash HERMES_HOME="$HOME/.hermes-relay-observability" hermes gateway stop HERMES_HOME="$HOME/.hermes-relay-redacted" hermes gateway stop ``` Remove only the dedicated Hermes installation, example homes, and output directories, or delete the disposable Brev instance. ## Docs - [Connect to Phoenix](https://docs.nvidia.com/nemo/relay/dev/user-guides/connecting-to-observability-platforms/phoenix.md): Export Relay OpenInference traces to Phoenix and confirm agent, model, and tool spans. - [Connect to LangSmith](https://docs.nvidia.com/nemo/relay/dev/user-guides/connecting-to-observability-platforms/langsmith.md): Export Relay traces to a LangSmith project and inspect correlated agent work. - [Connect to W&B Weave](https://docs.nvidia.com/nemo/relay/dev/user-guides/connecting-to-observability-platforms/weave.md): Send Relay GenAI traces to a Weave project and inspect them in the Agents view. - [Connect to Datadog](https://docs.nvidia.com/nemo/relay/dev/user-guides/connecting-to-observability-platforms/datadog.md): Send Relay traces through an enterprise OTLP collector or tracing backend to Datadog.