> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # Basic Usage The `nemo-relay` binary observes coding agents that do not expose every LLM call site directly. It combines agent-specific hook endpoints with a passthrough LLM gateway so NeMo Relay owns both the agent lifecycle and the model request lifecycle. Use the gateway when you need one observability boundary for OpenAI Codex and Claude Code without replacing each agent's canonical hook payload. ## Hook Endpoints Each hook endpoint accepts the agent's native hook JSON directly. Do not wrap the payload in a shared gateway envelope. * `POST /hooks/codex` accepts Codex hook JSON and returns the Codex-compatible hook response object. * `POST /hooks/claude-code` accepts Claude Code hook JSON and returns Claude-compatible fields such as `continue` and permission decisions when the hook event supports them. * `POST /hooks/pi` accepts the hook JSON posted by the NeMo Relay pi extension. It returns an empty object for an allowed call, or `{"tool_call": {"tool_call_id": "…", "input": {…}}}` when a request intercept rewrote the arguments and the extension must execute those instead. It returns HTTP 403 with `error.type = "nemo_relay_guardrail_rejected"` when a guardrail blocks one. When a hook closes a turn, subagent, or session scope, Relay returns the hook response after subscribers finish processing that scope-end event. This makes synchronous outputs such as the final ATIF trajectory visible at the hook boundary without draining subscriber work queued later by another session. Full gateway shutdown uses a process-wide flush after closing all remaining sessions and before clearing plugins. The adapters preserve vendor fields such as session IDs, working directories, transcript paths, model names, tool payloads, shell payloads, MCP payloads, file payloads, user identity, and subagent metadata in NeMo Relay event metadata. ## Gateway Routes Route all coding-agent LLM traffic through the gateway when full LLM lifecycle observability is required. * `POST /v1/responses` * `POST /v1/chat/completions` * `POST /v1/messages` * `POST /v1/messages/count_tokens` * `GET /v1/models` The gateway forwards raw provider JSON without rewriting OpenAI or Anthropic payload schemas. It removes only hop-by-hop transport headers, forwards streaming responses as streams, and emits NeMo Relay LLM start and end events under the active session scope. ## Transparent Run Use the agent shortcuts for local observability. The wrapper starts a gateway on a dynamic `127.0.0.1` port, injects the resolved hook and gateway configuration into the launched coding agent, and stops the gateway when the agent exits. Relay owns the launched wrapper and its descendants as one process tree, so a gateway failure cannot leave an agent running without its private gateway and hook configuration. Interactive Unix launches transfer foreground terminal ownership and preserve Ctrl-C and Ctrl-Z job control. Non-interactive Unix launches forward termination signals to the complete agent process tree before Relay removes the injected configuration. ```bash nemo-relay codex nemo-relay claude nemo-relay pi ``` `nemo-relay codex` and `nemo-relay claude` need no install: the wrapper injects their hooks per run. `nemo-relay pi` is the exception. pi has no native hook-configuration file, so its hooks can only originate inside a NeMo Relay extension that is already installed. On a clean machine the first-time setup wizard offers to install it; once a Relay configuration exists the wizard no longer runs, so install it with `nemo-relay install pi`. See [pi](/nemo-relay-cli/pi). Use `nemo-relay run -- ` when you want to launch an explicit command instead of the built-in shortcut: ```bash nemo-relay run -- codex ``` When `--agent` or an agent shortcut selects the host, pass only the host's arguments after `--`: ```bash nemo-relay run --agent claude -- -p "Review this change" ``` Add `--dry-run` before `--` to validate and inspect the resolved launch plan without running setup, starting the gateway, or launching the agent. During dry-run validation, Relay logs a warning when the selected executable is repeated after `--`: ```bash nemo-relay run --agent claude --dry-run -- claude nemo-relay claude --dry-run -- claude ``` Live launches do not perform this diagnostic or rewrite forwarded arguments. The selected agent receives the arguments exactly as provided. Dry-run output includes forwarded arguments. Do not share it when those arguments contain prompts, credentials, or other sensitive values. For Claude Code and Codex, transparent mode leaves the caller's source settings, selected profile, and installed plugin state unchanged. A process marker makes any installed Relay MCP borrow the wrapper-owned dynamic gateway. Claude's persistent hooks exit without forwarding; Codex disables the known Relay plugin hook identities in its process-local CLI state. Only the injected wrapper hooks deliver events. If a launcher or wrapper hides the real agent name, set that wrapper as the configured command and pass `--agent`. The same pattern applies to Claude Code and Codex: ```toml [agents.codex] command = "my-codex-wrapper" ``` ```bash nemo-relay run --agent codex ``` Use `--dry-run --print` to inspect the generated hook config, gateway environment, gateway URL, and final command without launching the agent. ## Persistent Host Integrations Use persistent integration installation to let a coding agent load Relay without a wrapper command: ```bash nemo-relay install claude-code nemo-relay install codex nemo-relay install pi nemo-relay install all ``` `nemo-relay install pi` is not a marketplace install: pi has no marketplace, so Relay writes the extension into pi's own auto-discovery directory instead. See [pi](/nemo-relay-cli/pi). Claude Code and Codex use marketplace plugins. Their MCP clients can share the native gateway on `127.0.0.1:47632`. Every `nemo-relay mcp` process acquires that gateway when it launches. Claude Code uses `alwaysLoad` to wait for the MCP connection. For Claude Code and Codex, installation writes a local marketplace, installs the generated `nemo-relay-plugin`, and configures host-specific hooks and provider routing. Both paths use the local `nemo-relay` binary on `PATH`. Use plugin doctor and uninstall for the installed host state: ```bash nemo-relay doctor --plugin codex nemo-relay uninstall codex ``` Refer to [Coding Agent Installation](/nemo-relay-cli/plugin-installation) for install directories, host-specific behavior, and the shared-sidecar lifecycle. ## Shared Configuration Shared TOML config is optional. The gateway loads defaults, then the explicit file when supplied or the XDG user file otherwise, and finally the system file. System config has the highest file-level priority. CLI flags and environment variables override file config. Repository-local `.nemo-relay/config.toml` files are ignored unless selected explicitly. ### Interactive Setup Run `nemo-relay config` to set up Relay interactively: 1. Select the coding agents that Relay should observe. 2. Review and save the user `config.toml`. 3. Choose whether to continue to the plugin editor. The base `config.toml` stores the agent settings. Select **Yes** at the plugin prompt to configure optional Relay components and save them separately in `plugins.toml`. Setup writes the XDG user configuration. When the top-level command receives `--config path/to/config.toml`, the plugin editor instead uses the sibling `path/to/plugins.toml`, matching runtime selection. Plugin files use the same file-level precedence order: explicit-or-user, then the system file. An explicit `--plugin-config-path`, or the sibling selected by `--config`, replaces the ambient XDG user plugin file. Likewise, an explicit `--config` replaces the ambient XDG user base file. Neither explicit file suppresses system configuration. Select **No** to finish after saving `config.toml`. Canceling the prompt or leaving the plugin editor does not remove the saved base configuration. You can open the plugin editor again later with the resume command printed by Relay. For an explicit `--config path/to/config.toml` flow, the equivalent command is `nemo-relay --config path/to/config.toml plugins edit`. Otherwise, use `nemo-relay plugins edit` for user configuration. ### Edit Gateway Configuration Use `nemo-relay config edit` to update the user `config.toml` with structured controls for gateway limits, provider upstreams, and operational logging: ```bash nemo-relay config edit ``` Use `--global` to edit system configuration at `/etc/nemo-relay/config.toml` on Unix or `%ProgramData%\nemo-relay\config.toml` on Windows. The editor creates a missing target only after you select **Save**, preserves unrelated TOML sections, and lets you clear a setting to restore normal configuration precedence and defaults. Global saves are system-readable (`0644` on Unix), so they reject authorization headers; store credentials in a user config or environment variables instead. When the top-level CLI receives `--config path/to/config.toml`, the editor uses that exact file as its user target, so the default editor and `config edit --user` both open it. Use `--global` to edit the system layer. This selects only the file opened by the editor; it does not change runtime discovery, layering, or merge precedence. Agent command setup remains under `nemo-relay config`; plugin components remain under `nemo-relay plugins edit`. When an explicit plugin file is selected, it becomes the editor's user target: the default editor and `--user` open that file. Use `--global` to edit the system file. Use `nemo-relay plugins edit --global` for `/etc/nemo-relay/plugins.toml` on Unix or `%ProgramData%\nemo-relay\plugins.toml` on Windows. Global plugin configuration is system-readable (`0644` on Unix), so do not store credentials there. The editor rejects schema-declared secret values in global plugin configuration. The upstream authorization-header controls show only whether a value is configured and never print it in menus or previews. Prefer `NEMO_RELAY_OPENAI_AUTH_HEADER` and `NEMO_RELAY_ANTHROPIC_AUTH_HEADER` instead of storing credentials in `config.toml`. ### Provider Upstreams Set provider base URLs under `[upstream]` when you want Relay to forward compatible requests to a proxy, enterprise endpoint, or another provider host: ```toml [upstream] openai_base_url = "https://api.openai.com/v1" anthropic_base_url = "https://api.anthropic.com" ``` Relay normally chooses the upstream from the request path, regardless of which compatible client sends the request. The gateway routes map to settings as follows: | Gateway Endpoint | Upstream Setting | | ------------------------------------------- | -------------------- | | `/responses`, `/v1/responses` | `openai_base_url` | | `/chat/completions`, `/v1/chat/completions` | `openai_base_url` | | `/models`, `/v1/models` | `openai_base_url` | | `/v1/messages`, `/v1/messages/count_tokens` | `anthropic_base_url` | For ordinary provider requests, Relay resolves credentials in this order: 1. Relay preserves an inbound `Authorization`, `x-api-key`, `api-key`, or `anthropic-api-key` header. 2. If no inbound credential is present, Relay uses the route-specific custom `Authorization` value from `[upstream]` or its environment override. 3. If no custom value is configured, Relay reads `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` and applies the provider's standard authentication scheme. Set `openai_auth_header` or `anthropic_auth_header` under `[upstream]` only when an enterprise gateway, proxy, or compatible endpoint requires a complete `Authorization` value such as `Bearer ...` or `Basic ...`. Prefer the corresponding environment variables so you don't store credentials in `config.toml`: ```bash export NEMO_RELAY_OPENAI_AUTH_HEADER="Bearer " export NEMO_RELAY_ANTHROPIC_AUTH_HEADER="Basic " ``` Configure a custom base URL and its authorization header in the same layer. For environment configuration, set `NEMO_RELAY_OPENAI_BASE_URL` with `NEMO_RELAY_OPENAI_AUTH_HEADER`, or set the corresponding Anthropic variables together. A CLI `--openai-base-url` or `--anthropic-base-url` override can inherit an environment header only when the environment selected the same URL. If the CLI flag changes the URL, Relay clears the environment header; use the paired environment variables instead when the new endpoint requires custom authentication. The exact configured base URL is the upstream identity during file, environment, and command-line layering. A higher layer that keeps the same URL can inherit or replace the lower authorization header. A higher layer that changes the URL clears the lower header unless that same layer supplies a replacement, preventing credentials for one endpoint from being sent to another. An MCP-managed gateway uses custom or environment credentials only after the client proves that it belongs to that managed gateway. Codex ChatGPT credentials are the exception to the normal inbound precedence. On OpenAI routes, Relay recognizes `Bearer eyJ...` JWTs and `Bearer at-...` access tokens. If `openai_auth_header`, `NEMO_RELAY_OPENAI_AUTH_HEADER`, or a nonempty `OPENAI_API_KEY` is available, Relay removes the ChatGPT credential, applies the replacement, and forwards the request to the configured `openai_base_url`. Without a replacement, Relay preserves the ChatGPT credential and routes the request to the Codex ChatGPT backend instead of `openai_base_url`. ### Operational Logging The CLI initializes operational logging before operational commands run. `nemo-relay config` and `nemo-relay plugins edit` skip initialization so invalid logging settings do not prevent configuration repair. File sinks from layered `config.toml` files aggregate by their resolved destination `path`, with system entries first. A higher layer recursively overlays a sink with the same path; path aliases such as `relay.log` and `./relay.log` resolve to one sink, while distinct destinations remain active together. Configure temporary CLI settings with `--log-level` and `--log-stderr-format`, or select an absolute TOML file with `--log-config-path`. For defaults, source precedence, environment variables, TOML file sinks, and Rust APIs, refer to [Operational Logging](/reference/operational-logging). ## Add Model Pricing for Cost Estimates Model pricing is configured with the same `plugins.toml` discovery path as Observability. The configured sources apply to transparent agent runs, standalone gateway runs, and evals or custom agents that initialize the same gateway plugin config. Framework integrations, harnesses, and custom hosts do not need their own model pricing logic when they emit managed LLM calls or manual lifecycle events with normalized response annotations: Relay attaches cost to the annotated response when the provider reports cost or when a configured model pricing source matches the response model and token usage. Create a Relay model pricing catalog JSON file: ```json { "version": 1, "entries": [ { "provider": "openai", "model_id": "gpt-4o-mini", "aliases": ["openai/openai/gpt-4o-mini"], "currency": "USD", "unit": "per_token", "rates": { "input_per_million": 0.15, "output_per_million": 0.6, "cache_read_per_million": 0.075 }, "prompt_cache": { "read_accounting": "included_in_prompt_tokens" }, "pricing_as_of": "2026-06-06", "pricing_source": "internal-pricing-snapshot" } ] } ``` Validate and add the file-backed source: ```bash nemo-relay model-pricing validate nemo-relay model-pricing init --user nemo-relay model-pricing add-source --user ``` Replace `` with the path to your pricing catalog before running either command. Use `--global` for `/etc/nemo-relay/plugins.toml` on Unix or `%ProgramData%\nemo-relay\plugins.toml` on Windows. `model-pricing add-source` prepends the source by default, so the new file becomes the highest-priority source for that scope. Use `--append` to add it as a lower-priority fallback. Resolve a model before running an agent: ```bash nemo-relay model-pricing resolve gpt-4o-mini \ --provider openai \ --prompt-tokens 1000 \ --completion-tokens 500 ``` `model-pricing resolve` prints the source that won, the matched provider/model, and an estimated total when token counts are supplied. Use it to debug overlapping system, explicit, and user model pricing sources. Relay now treats missing rates for nonzero cache usage as unknown totals, not free usage. If a response reports nonzero cache-read or cache-write tokens and the matching catalog omits `cache_read_per_million` or `cache_write_per_million`, Relay omits the aggregate total instead of undercounting it. Set either rate to `0.0` when that category is intentionally free. Catalogs that define only input and output rates therefore emit no total for responses that report cached tokens. Run doctor to validate the active model pricing sources alongside exporter checks: ```bash nemo-relay doctor codex ``` Doctor fails when an enabled model pricing source is unreadable or contains an invalid catalog, and it reports passing sources as `Model pricing source`. Relay does not ship a canonical price catalog. Unknown models and missing token fields leave cost absent instead of defaulting to zero. For the catalog schema, provider-aware lookup behavior, threshold-based model pricing, and custom `PricingSource` integrations, refer to [Provider Response Codecs and Model Pricing](/integrate-into-frameworks/provider-response-codecs#cost-estimation). Transparent runs always bind the managed gateway to `127.0.0.1:0`. The selected port is discovered by the wrapper and exposed to hooks through `NEMO_RELAY_GATEWAY_URL`. Common environment variables for direct gateway server use are: * `NEMO_RELAY_GATEWAY_BIND` * `NEMO_RELAY_OPENAI_AUTH_HEADER` * `NEMO_RELAY_OPENAI_BASE_URL` * `NEMO_RELAY_ANTHROPIC_AUTH_HEADER` * `NEMO_RELAY_ANTHROPIC_BASE_URL` * `NEMO_RELAY_MAX_HOOK_PAYLOAD_BYTES` * `NEMO_RELAY_MAX_PASSTHROUGH_BODY_BYTES` The default hook payload limit is `20MiB`. The default provider passthrough body limit is `100MiB`. Set both values in bytes. Plugin configuration controls process-level Observability exporters. Per-session configuration controls structured metadata on the top-level agent begin event and the plugin configuration metadata associated with the session. For example, attach a user identity to a transparent run: ```bash nemo-relay run --session-metadata '{"user_id":"alice"}' -- codex ``` The CLI emits one `session.start` mark for the lifecycle and copies the merged metadata onto each top-level trace scope. It also adds a non-overridable `session_instance_id` derived from the session's existing root scope UUID. Keep `user_id` a top-level string when OTLP exporters should expose it as `user.id`. `hook-forward` can also pass per-session configuration through headers: * `x-nemo-relay-config-profile` * `x-nemo-relay-session-metadata` * `x-nemo-relay-plugin-config` * `x-nemo-relay-gateway-mode` The accepted gateway mode values are `hook-only`, `passthrough`, and `required`. The gateway records this value as session metadata so downstream exporters and review tooling can distinguish hook-only traces from sessions where provider traffic was expected to pass through the gateway. ## Runtime Mapping The gateway normalizes vendor hook payloads into private internal events before calling NeMo Relay APIs. * Agent start emits a `session.start` mark on a dedicated `ScopeStackHandle`. Harnesses with a reliable session boundary also open a top-level `ScopeType::Agent` scope; Claude Code and Codex use bounded turn scopes. * Subagent start opens a child `ScopeType::Agent` scope. Subagent stop closes that scope when it is still active. * Tool pre-use starts a NeMo Relay tool span. Tool post-use, denial, or failure closes it. * Generated `UserPromptSubmit` and `Stop` hooks are retained as private correlation hints. The adapters do the same when a custom or older integration delivers a response or agent-thought hook. These hints are not emitted as NeMo Relay events. * Compaction, notification, and unknown hook events become mark events under the active session scope. * Harnesses that report the *start* of a turn as well as its end open the turn scope at that signal instead of lazily. pi does; Claude Code and Codex report only `Stop`, so their turns stay lazily opened by the first event of the turn. For a harness with an explicit turn start, a mark that arrives between turns is recorded on the session scope rather than opening a turn to hold it. * Gateway requests emit NeMo Relay LLM start and end events under the active session scope. Before each LLM start, the gateway uses explicit subagent headers, pending hints, shared conversation/generation/request identifiers, and the previous correlated owner to choose the parent scope. * LLM responses that contain future tool-use suggestions are retained as private tool-call hints. The next matching tool hook can then inherit the subagent scope that owned the LLM response, even when the hook payload does not include a subagent id. Gateway requests can provide explicit correlation identifiers with these headers: * `x-nemo-relay-session-id` * `x-nemo-relay-subagent-id` * `x-nemo-relay-conversation-id` * `x-nemo-relay-generation-id` * `x-nemo-relay-request-id` When those headers are absent, the gateway also looks for `conversation_id`/`conversationId`/`conversation.id`, `generation_id`/`generationId`/`generation.id`, and `request_id`/`requestId`/`request.id` fields in the provider request body. Correlation hints expire after five minutes. If the gateway cannot select one unambiguous hint, it falls back to the previous LLM owner, then to the only active subagent, then to the top-level agent scope. Every gateway LLM event includes `llm_correlation_status` metadata. Managed requests can use `explicit`, `single_hint`, `matched_hint`, `request_affinity`, `sticky_last_owner`, `recent_tool_owner`, `subagent_start`, `active_subagent`, `agent_fallback`, or `ambiguous_fallback`. Claude Code startup probes use `pre_turn_probe`. Matched hints can also add `llm_correlation_source`, `llm_correlation_subagent_id`, `llm_correlation_conversation_id`, `llm_correlation_generation_id`, `llm_correlation_request_id`, and `llm_correlation_agent_type`. Generated hook bundles subscribe to the events needed for that mapping. pi has no generated bundle — its extension posts these names directly: | Agent | LLM Lifecycle and Correlation Hooks | Scope, Tool, and Mark Hooks | | ----------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Claude Code | `UserPromptSubmit`, `Stop` | `SessionStart`, `SessionEnd`, `UserPromptExpansion`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `PermissionRequest`, `Notification`, `PreCompact`, `PostCompact` | | Codex | `UserPromptSubmit`, `Stop` | `SessionStart`, `SubagentStart`, `SubagentStop`, `PreToolUse`, `PostToolUse`, `PermissionRequest`, `PreCompact`, `PostCompact` | | pi | none — pi's LLM spans come from real gateway traffic, not from hooks | `session_start`, `session_shutdown`, `agent_start`, `agent_end`, `agent_settled`, `turn_start`, `turn_end`, `session_before_compact`, `session_compact`, `model_redirect`, `tool_call`, `tool_arguments_transformed`, `tool_execution_end`, `user_bash`, `user_bash_end` | Three of pi's names are not pi hooks. The extension posts `model_redirect` after deciding whether to point the active model's provider at the gateway, from `session_start` and from every `model_select`; `tool_arguments_transformed` after applying a request intercept's rewrite; and `user_bash_end` to close the inline-shell span, because pi reports no completion for it. pi's own `tool_execution_start` is deliberately never forwarded — it fires before argument validation and for calls that never execute, so the extension uses it only to remember a tool name for the matching end. ## Hook Forwarding Transparent Claude Code and Codex hooks invoke `nemo-relay hook-forward ` with the canonical payload on standard input. The wrapper-owned hook command embeds its ephemeral gateway URL and is marked as transparent so it cannot recover the fixed gateway. Persistent Claude Code and Codex hooks also use `nemo-relay hook-forward `. Each generated command identifies the fixed gateway and its private install-generation fence, waits for the MCP-owned Relay gateway, verifies it, and forwards the unchanged payload once. `hook-forward` reads the canonical hook payload from standard input, sends it to the matching endpoint, and prints the endpoint response. Generated `PreToolUse` and `PermissionRequest` hooks use `--fail-closed`; generated lifecycle and after-the-fact hooks use `--fail-open`. This blocks permission-bearing operations when Relay cannot evaluate them without making observability-only hooks a runtime dependency. Rerun `nemo-relay install --force` after upgrading to replace legacy generated hooks. Manually authored commands fail open when neither policy flag is present. `NEMO_RELAY_FAIL_CLOSED=1` changes that default for compatibility, while an explicit flag takes precedence over the environment. These flags control delivery and metadata: * `--gateway-url ` selects the Relay gateway that receives the payload. * `--forward-only` allows source plugins and custom automation to use an existing compatible gateway without an installer-owned generation fence. It verifies the gateway but never launches or recovers Relay. Generated installed hooks use a private generation fence instead. * `--session-metadata` sets `x-nemo-relay-session-metadata`. * `--profile` sets `x-nemo-relay-config-profile`. * `--gateway-mode` sets `x-nemo-relay-gateway-mode`. * `--fail-open` allows the agent to continue after a delivery failure, even when `NEMO_RELAY_FAIL_CLOSED=1` is set. Structured guardrail rejections still return a failure. * `--fail-closed` returns a failure when delivery fails or Relay rejects the hook. ## Agent Guides Use the per-agent guide for end-to-end setup, smoke tests, and GUI or application-mode caveats. * [Claude Code](/nemo-relay-cli/claude-code) * [Codex](/nemo-relay-cli/codex) * [pi](/nemo-relay-cli/pi) * [Coding Agent Installation](/nemo-relay-cli/plugin-installation) Each guide covers transparent run setup, gateway routing, hook smoke tests, Agent Trajectory Interchange Format (ATIF) export verification at the host's supported snapshot boundary, and troubleshooting missing LLM lifecycle data.