> This page is for NeMo Relay, version dev. > For other versions, use one of these documentation indexes: > - Latest (v0.10.3) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt > - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt > - v0.10.3: https://docs.nvidia.com/nemo/relay/v0.10.3/llms.txt > - v0.10.2: https://docs.nvidia.com/nemo/relay/v0.10.2/llms.txt > - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt > - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt > - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # Deep Agents Quick Start > Run one Deep Agents workflow with NeMo Relay observability. ## Goal Run one Deep Agents workflow and write its agent, model, and tool lifecycle to a local Agent Trajectory Observability Format (ATOF) file. Use the event relationships to follow model and tool work through a run and check where the expected activity occurred. ## Prerequisites Before you start, make sure you have: * Python 3.11 or newer * `uv` * Inference access: an [NVIDIA API key](https://build.nvidia.com/settings/api-keys) for this example, or credentials for your own provider ## Install Create a project and install the NeMo Relay version for these docs with the Deep Agents and NVIDIA model provider extras. For an existing `uv` project, run only the `uv add` command from its directory. ```bash mkdir relay-deep-agents-quickstart cd relay-deep-agents-quickstart uv init uv add "nemo-relay[deepagents,langchain-nvidia]==0.11.0" ``` Set your NVIDIA API key for the current shell: #### Unix shell ```bash export NVIDIA_API_KEY="your-api-key" ``` #### Windows PowerShell ```powershell $env:NVIDIA_API_KEY = "your-api-key" ``` > **Note** > > Relay does not impose a model choice. This copyable example selects the public > [`nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`](https://build.nvidia.com/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning/playground) > model through the NVIDIA provider. To use another model or provider, replace > the example's `model` argument and configure that provider's credentials. > The NVIDIA provider reads `NVIDIA_API_KEY`. In the LangChain string, the first `nvidia:` > selects the provider; the remaining `nvidia/...` value is the model ID. > **Note** > > The provider can warn that this model is "not known to support tools" when its > capability catalog lags the endpoint. This warning is advisory; the run must > still produce the actual tool result and all three success lines below. ## Configure `plugins.toml` enables Relay's observability component and writes its events to `logs/events.jsonl`. In your own project, use the same plugin configuration with the integration code below; the marker task and success checks are only the quick start's validation example. Save the following configuration as `plugins.toml` in your project directory: ```toml title={"plugins.toml"} # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 version = 1 [policy] unknown_component = "error" unknown_field = "error" unsupported_value = "error" [[components]] kind = "observability" enabled = true [components.config] version = 4 enable_full_payloads = false [components.config.atof] enabled = true [[components.config.atof.sinks]] type = "file" output_directory = "logs" filename = "events.jsonl" mode = "overwrite" ``` > **Warning** > > ATOF output can contain sensitive prompts, responses, tool arguments, and tool > results. Protect `logs/events.jsonl` even when full payload export is disabled. ## Run The example creates a Deep Agents agent with one marker tool. `add_nemo_relay_integration` adds Relay-aware middleware, including supported subagent instrumentation. `NemoRelayDeepAgentsCallbackHandler` records the framework lifecycle. Plugin activation loads `plugins.toml`, the single outer `deepagents-request` scope groups the request, and flushing finishes writing queued events before verification. You do not need to add a Relay mark around every operation. In your own agent, keep that integration and activation lifecycle, replace the marker tool and prompt, and use an outer scope for the request you own. The marker and event-file assertions are quick-start validation code. Save this file as `quickstart.py` beside `plugins.toml`: ```python title={"quickstart.py"} # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 import asyncio import json from pathlib import Path from deepagents import create_deep_agent from langchain_core.tools import tool import nemo_relay from nemo_relay.integrations.deepagents import ( NemoRelayDeepAgentsCallbackHandler, add_nemo_relay_integration, ) @tool def emit_marker() -> str: """Return the quick-start validation marker.""" return "RELAY_QUICKSTART_TOOL_OK" agent = create_deep_agent( **add_nemo_relay_integration( model="nvidia:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning", tools=[emit_marker], name="main-agent", system_prompt="Call emit_marker exactly once. Keep the final answer brief.", ) ) async def main() -> None: """Run the instrumented agent and verify its tool, response, and event markers.""" async with nemo_relay.plugin.activate({}, "plugins.toml"): try: with nemo_relay.scope.scope("deepagents-request", nemo_relay.ScopeType.Agent): result = await agent.ainvoke( { "messages": [ { "role": "user", "content": ( "Use one available tool to produce " "RELAY_QUICKSTART_TOOL_OK. Then include " "RELAY_QUICKSTART_RESPONSE_OK in your final response." ), } ] }, config={"callbacks": [NemoRelayDeepAgentsCallbackHandler()]}, ) tool_outputs = [ str(message.content) for message in result["messages"] if getattr(message, "type", None) == "tool" ] if "RELAY_QUICKSTART_TOOL_OK" not in tool_outputs: raise RuntimeError("Expected tool result is missing") final_message = result["messages"][-1] if getattr(final_message, "type", None) != "ai": raise RuntimeError("Expected a final assistant message") final_response = str(final_message.content) if "RELAY_QUICKSTART_RESPONSE_OK" not in final_response: raise RuntimeError("Expected final response is missing") print("RELAY_QUICKSTART_TOOL_OK") print("RELAY_QUICKSTART_RESPONSE_OK") finally: await nemo_relay.subscribers.flush_async() events = [json.loads(line) for line in Path("logs/events.jsonl").read_text().splitlines()] if not events or any( not isinstance(event, dict) or event.get("atof_version") != "0.1" or event.get("kind") not in {"scope", "mark"} for event in events ): raise RuntimeError("Expected a nonempty ATOF 0.1 event stream") starts = { event["uuid"]: event for event in events if event["kind"] == "scope" and event.get("scope_category") == "start" } closed = [ event for event in events if event["kind"] == "scope" and event.get("scope_category") == "end" and event.get("uuid") in starts and all(event.get(key) == starts[event["uuid"]].get(key) for key in ("name", "category", "parent_uuid")) ] # Match the required identities, not just categories that unrelated work could satisfy. for category, name in (("agent", "deepagents-request"), ("tool", "emit_marker"), ("llm", None)): if not any( event.get("category") == category and (name is None or event.get("name") == name) for event in closed ): raise RuntimeError(f"Expected paired {category} scope: {name or 'model call'}") print("RELAY_QUICKSTART_EVENTS_OK") asyncio.run(main()) ``` Run it from the project directory: ```bash uv run python quickstart.py ``` ## Verify The program checks the tool result, final assistant response, and paired ATOF agent, tool, and LLM records. A successful run prints these markers in order: ```text RELAY_QUICKSTART_TOOL_OK RELAY_QUICKSTART_RESPONSE_OK RELAY_QUICKSTART_EVENTS_OK ``` > **Note** > > The model can use different wording around the response marker. These checks > validate a live provider call; they do not make the model deterministic. ### Inspect the Recorded Activity Print one completed model call from the event file: ```bash uv run python -c " import json from pathlib import Path events = [json.loads(line) for line in Path('logs/events.jsonl').read_text().splitlines()] call = next(event for event in events if event.get('scope_category') == 'end' and event.get('category') == 'llm') fields = ('name', 'category', 'category_profile', 'uuid', 'parent_uuid', 'timestamp', 'metadata') print(json.dumps({key: call.get(key) for key in fields}, indent=2)) " ``` `category` and `name` identify the recorded work. The same `uuid` pairs its start and end records; `parent_uuid` connects it to its enclosing scope. `timestamp` records when this event occurred. For model calls, `category_profile` and `metadata` can include the model identity and completion status, depending on the integration. These fields let you follow a call through the run without requiring full prompt or response capture. ## For More Details For more details, refer to: * [Deep Agents integration guide](/supported-integrations/deepagents) * [ATOF exporter](/configure-plugins/observability/atof)