# Comprehensive Reference for NVIDIA AIQ and NeMo Agent Toolkits

This documentation collection provides detailed API references, usage guides, tutorials, and troubleshooting resources for NVIDIA’s Agent Intelligence (AIQ) Toolkit and NeMo Agent Toolkit. It covers end‑to‑end workflows—including dataset handling, model evaluation, profiling, observability, front‑end development, registry management, and custom plugin creation—along with concrete examples for building, extending, and debugging AI agents, LLM integrations, and retrieval‑augmented pipelines. Users can leverage this material to configure, optimize, and troubleshoot complex AIQ/NeMo deployments across a wide range of use cases.

## Getting Started
- [A developer should read this page when they’re preparing to set up and run an NVIDIA Agent Intelligence Toolkit project—e.g., when installing the toolkit on a workstation or a Brev GPU instance, configuring GPU drivers, and launching the first training or inference job.](docs.nvidia.com/aiqtoolkit/latest/quick-start/index.html.md)
- [When you’re preparing to run an AIQ workflow—e.g., pulling NIM containers or serving a model with vLLM, and you require detailed instructions for container setup, configuration files, and command‑line invocation.](docs.nvidia.com/aiqtoolkit/latest/workflows/using-local-llms.html.md)
- [When you need to set up a training or inference script that should alter its logging, data sampling, or breakpoint strategy only when a debugger is present, call `is_debugger_attached` to detect a debugger and adjust the behavior accordingly.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/utils/debugging_utils/index.html.md)
- [Read this page whenever you’re building or debugging an AIQ workflow that uses the `code_execution` function—such as setting up a local Docker sandbox or connecting to a Piston server, configuring the URI, timeout, and output limits, and verifying that the function’s stdout is captured correctly.](docs.nvidia.com/aiqtoolkit/latest/workflows/functions/code-execution.html.md)
- [Use this page when you need to run an NVIDIA AIQ Toolkit workflow locally on a GPU, whether by pulling NIM containers or by serving a model with vLLM, and you require detailed instructions for container setup, configuration files, and command‑line invocation.](docs.nvidia.com/aiqtoolkit/latest/workflows/using-local-llms.html.md)
- [Read this page when you need to programmatically launch, run, or profile an evaluation workflow in the NVIDIA Agent Intelligence Toolkit—e.g., when an engineer is automating a CI pipeline that executes a local or remote evaluation run, cleans output directories, and collects evaluator results for a new dataset or model checkpoint.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/eval/evaluate/index.html.md)
- [Read the page whenever you’re building or debugging an AIQ workflow that uses the `code_execution` function—such as setting up a local Docker sandbox or connecting to a Piston server, configuring the URI, timeout, and output limits, and verifying that the function’s stdout is captured correctly.](docs.nvidia.com/aiqtoolkit/latest/workflows/functions/code-execution.html.md)
- [When you’re building or debugging a FastAPI service that powers the NVIDIA Agent Intelligence Toolkit—such as adding new API endpoints, customizing the logger, or diagnosing startup errors—you should consult this page.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/fastapi/main/index.html.md)

## Installation & Environment
- [Read this page when you need to install, uninstall, or query AIQ Toolkit components on the local machine—for example, to validate a configuration before deploying a new component, to clean up unused registries during a test, or to script automated registry maintenance in a CI/CD pipeline.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/registry_handlers/local/register_local/index.html.md)
- [Read this page when you’re adding or debugging a custom REST handler in the NVIDIA Agent Intelligence Toolkit—e.g., when you need to register a new endpoint for a model inference service or troubleshoot why an existing handler isn’t being discovered by the AIQ registry.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/registry_handlers/rest/index.html.md)
- [Read this page when you need to publish, pull, search, or remove AIQ Toolkit artifacts with a custom configuration via the `aiq.cli.commands.configure.configure_command` function, such as during CI/CD pipelines or when configuring automated deployments of AIQ packages.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/configure/configure/index.html.md)

## Core Concepts – Dataset & Evaluation
- [Use this page when you’re building or debugging a dataset handler for AIQ evaluations – e.g., configuring `DatasetHandler` to read, filter, deduplicate, and replicate rows before converting to `EvalInput`.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/eval/dataset_handler/index.html.md)
- [Use this page when you need to apply column‑based allowlist or denylist filtering to a pandas DataFrame before running evaluation – for example, when preparing a subset of a dataset for model benchmarking or when cleaning a data frame by removing rows that match unwanted values.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/eval/dataset_handler/dataset_filter/index.html.md)
- [When you need to convert raw token‑usage metrics (prompt, completion, or total tokens) into a validated data object for a callback, for example when implementing custom logging, billing calculations, or monitoring in an NVIDIA AIQ‑powered LLM workflow.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/profiler/callbacks/token_usage_base_model/index.html.md)
- [When you need to configure a tool‑calling agent in NVIDIA AIQ Toolkit—such as specifying which LLM to use, which tools to expose, whether to log verbose output, how to handle tool errors, or limiting the number of iterations—before passing that configuration to a builder to create the agent workflow.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/tool_calling_agent/register/index.html.md)
- [Use this page whenever you need to add or customize a forecasting model pipeline in the NVIDIA Agent Intelligence Toolkit – such as selecting the model type, debugging training failures, or extending the factory method with new models.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/profiler/forecasting/index.html.md)
- [When you need to add or customize how a NAT pipeline records token usage and latency statistics – e.g., enabling prompt‑caching, bottleneck analysis, or token‑uniqueness metrics – read the profiler configuration pages.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/profiler/inference_optimization/index.html.md)

## Agents
- [Use this page when you’re building or debugging a ReAct‑style LangChain agent that must parse an LLM’s “Thought → Action → Action Input → Observation” or “Thought → Final Answer” output.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/react_agent/output_parser/index.html.md)
- [Use this page when you’re building or debugging a ReAct‑style LangChain agent – e.g., to inspect the agent’s state machine, configure the prompt templates, or parse the LLM’s output.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/react_agent/index.html.md)
- [Use this page when you’re building or debugging a ReAct‑style agent in NVIDIA AIQ Toolkit – for example, to inspect the `ReActAgentGraph` implementation, understand its retry logic, or verify the system‑prompt validation used in prompt generation.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/react_agent/agent/index.html.md)
- [When you need to register or retrieve an NVIDIA AIQ agent—such as creating a new agent, updating its metadata, or querying its status during development or production deployment.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/register/index.html.md)
- [Use this page when you need to build or debug a dual‑node agent that separates reasoning (the agent node) from tool execution (the tool node) and routes between them via conditional edges.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/dual_node/index.html.md)
- [When building or debugging an NVIDIA AIQ dual‑node agent – i.e., when configuring the `ToolCallAgentGraph`, toggling `detailed_logs`, or enabling `handle_tool_errors` to diagnose why a tool call failed or how the agent selects the next tool.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/tool_calling_agent/agent/index.html.md)
- [When building or debugging a reasoning agent workflow in NVIDIA AIQ – e.g., selecting an LLM that supports `<think>` tags, configuring the reasoning function, or troubleshooting plan generation and execution logic.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/reasoning_agent/reasoning_agent/index.html.md)
- [When building or debugging a ReWOO agent in NVIDIA AIQ – e.g., configuring the planner/solver prompts, customizing the tool list, or evaluating token‑efficient planning strategies.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/agent/rewoo_agent/index.html.md)

## Front‑Ends
- [Use this page when you’re building or extending a FastAPI front end for an AIQ Toolkit workflow – e.g., instantiating `FastApiFrontEndPlugin` with an `AIQConfig` and invoking its `run()` method.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/fastapi/fastapi_front_end_plugin/index.html.md)
- [When building or extending a FastAPI front‑end worker for an NVIDIA AIQ Toolkit – such as adding custom evaluation routes, configuring CORS, or integrating a new step adaptor.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/fastapi/fastapi_front_end_plugin_worker/index.html.md)
- [When integrating NVIDIA’s AIQ Toolkit into a FastAPI application that uses WebSocket endpoints – e.g., building a chatbot or interactive agent – you’ll consult this page to understand how incoming user messages are validated, queued, and routed.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/fastapi/message_handler/index.html.md)
- [When building or debugging a FastAPI console front‑end for an AIQ Toolkit workflow – e.g., configuring the `ConsoleFrontEndPlugin`, setting up CLI prompt callbacks, or troubleshooting the `run_workflow` execution logic.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/console/console_front_end_plugin/index.html.md)
- [When building or debugging a console‑based front end for the NVIDIA AIQ Toolkit – e.g., configuring the `ConsoleFrontEndConfig` and validating command‑line arguments.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/console/index.html.md)
- [When building or debugging an MCP front‑end in NVIDIA AIQ – e.g., registering the front end with the agent’s configuration, validating the `MCPFrontEndConfig` parameters, or debugging endpoint registration.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/mcp/mcp_front_end_plugin/index.html.md)
- [When configuring a cron‑based front‑end for AIQ – e.g., scheduling recurring model training, inference, or data‑preprocessing jobs.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/front_ends/cron/index.html.md)

## LLM Configuration & Utilities
- [Use this page when you need to configure an NVIDIA NIM LLM – e.g., setting `api_key`, `base_url`, `model_name`, and validation before creating the `nim_model`.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/llm/nim_llm/index.html.md)
- [Use this page when you need to configure an OpenAI LLM – e.g., setting the `api_key`, `model_name`, `temperature`, and retry options.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/llm/openai_llm/index.html.md)
- [Use this page when you need to add a new LLM provider to the AIQ toolkit – e.g., subclassing `LLMBaseConfig`, registering the provider and its clients.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/builder/llm/index.html.md)
- [When you need to set up or inspect global runtime settings – e.g., default timeouts, logging levels, or resource limits – before launching an AIQ pipeline.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/settings/index.html.md)
- [When you need to add or configure a large‑language‑model to the NeMo Agent Toolkit – e.g., before launching a deployment or fine‑tuning job.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/llm/register/index.html.md)

## Embedders
- [Use this page when you need to add a NVIDIA NIM embedder to your pipeline – e.g., configuring API keys, model name, and truncation options.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/embedder/nim_embedder/index.html.md)
- [When adding an OpenAI embedder to an AIQ workflow – e.g., configuring the API key, base URL, and model name.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/embedder/openai_embedder/index.html.md)
- [When you need to add or configure an embedder in a NeMo Agent workflow – e.g., registering a new embedder type or validating its configuration.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/embedder/register/index.html.md)

## Retrievers
- [Use this page when you need to configure a Milvus retriever – e.g., setting the URI, collection name, and search parameters.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/retriever/milvus/register/index.html.md)
- [When configuring or debugging the NeMo Retriever service – e.g., registering a custom retriever, selecting a retrieval strategy, or troubleshooting retrieval outputs.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/retriever/nemo_retriever/register/index.html.md)

## Memory
- [Use this page whenever you’re wiring or extending memory in a NeMo Agent – e.g., creating a `MemoryItem` payload, validating `SearchMemoryInput`, or troubleshooting memory‑related validation errors.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/memory/models/index.html.md)
- [When adding a custom memory backend to a NeMo Agent Toolkit workflow – e.g., registering a new `MemoryBaseConfig` and `MemoryEditor`.](docs.nvidia.com/aiqtoolkit/latest/extend/memory.html.md)

## Tools
- [Use this page when you need to add a custom text‑file ingestion tool to a NeMo Agent workflow – e.g., leveraging LangChain’s `DirectoryLoader`.](docs.nvidia.com/nemo/agent-toolkit/latest/tutorials/create-a-new-workflow.html.md)
- [When adding a tool that fetches a file from GitHub – e.g., retrieving a script or configuration file for a workflow.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/github_tools/get_github_file/index.html.md)
- [When adding a tool to create or update a GitHub issue – e.g., reporting CI failures.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/github_tools/create_github_issue/index.html.md)
- [When adding a tool to create a GitHub pull request – e.g., automating code review.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/github_tools/create_github_pr/index.html.md)
- [When adding a tool to list or get GitHub issues – e.g., triaging bugs.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/github_tools/get_github_issue/index.html.md)
- [When adding a tool to list or fetch GitHub pull requests – e.g., automation of PR handling.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/github_tools/get_github_pr/index.html.md)
- [When adding a tool to update a GitHub issue – e.g., marking it resolved after a run.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/github_tools/update_github_issue/index.html.md)
- [When adding a tool to execute code in a sandbox – e.g., running user‑provided Python snippets safely.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/code_execution/index.html.md)
- [When adding a tool to obtain the current date and time – e.g., timestamping logs.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/datetime_tools/index.html.md)
- [When adding a tool to retrieve documents from Milvus – e.g., for RAG.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/document_search/index.html.md)
- [When adding a tool to interact with the Model Context Protocol (MCP) – e.g., wrapping a remote tool as an AIQ function.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/tool/mcp/mcp_tool/index.html.md)

## Observability & Telemetry
- [Use this page when you need to add OpenTelemetry tracing to an AIQ Toolkit workflow – e.g., automatically creating spans for FUNCTION_START and FUNCTION_END events.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/observability/register/index.html.md)
- [Use this page when you need to add a custom profiling callback – e.g., instrument training loops with new metrics.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/profiler/callbacks/base_callback_class/index.html.md)
- [When building or extending a telemetry exporter for NeMo Agent Toolkit – e.g., adding a Prometheus or OpenTelemetry exporter.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/observability/exporter/index.html.md)
- [When configuring or debugging telemetry exporters – e.g., setting up collector endpoints or file exporters.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/observability/mixin/collector_config_mixin/index.html.md)
- [When you need to add a custom telemetry exporter that processes raw `IntermediateStep` objects before exporting.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/observability/exporter/raw_exporter/index.html.md)
- [When you need to compute profiling metrics such as NOVA columns after a run.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/profiler/inference_optimization/llm_metrics/index.html.md)
- [When you need to compute bottleneck analysis for a workflow – e.g., pinpointing slow steps.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/profiler/inference_optimization/bottleneck_analysis/simple_stack_analysis/index.html.md)

## CLI & Commands
- [Use this page when you need to run the **evaluate** CLI command – e.g., to evaluate a dataset with a specified evaluation config.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/evaluate/index.html.md)
- [Use this page when you need to publish an AIQ Toolkit artifact – e.g., uploading a plugin wheel.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/registry/publish/index.html.md)
- [Use this page when you need to pull artifacts from a registry – e.g., downloading a model checkpoint.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/registry/pull/index.html.md)
- [Use this page when you need to remove artifacts from a registry – e.g., cleaning up old versions.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/registry/remove/index.html.md)
- [Use this page when you need to list components in the registry.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/info/list_components/index.html.md)
- [Use this page when you need to configure global settings via the CLI.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/configure/index.html.md)
- [Use this page when you need to run a workflow from the command line – e.g., `aiq run`.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/start/index.html.md)
- [Use this page when you need to size a deployment – e.g., estimating GPU count.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/sizing/calc/index.html.md)
- [Use this page when you need to override configuration values from the CLI.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/cli_utils/config_override/index.html.md)
- [Use this page when you need to validate an AIQ configuration file before running a job.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/validate/index.html.md)

## Registry & Plugins
- [Use this page when you need to publish or pull AIQ Toolkit artifacts with a custom configuration – e.g., during CI/CD.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/cli/commands/publish/index.html.md)
- [When adding a new component to the AIQ registry or debugging component discovery, consult this page.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/registry_handlers/metadata_factory/index.html.md)
- [Use this page when you need to add a custom local registry handler – e.g., for local development.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/registry_handlers/local/index.html.md)
- [Use this page when you need to add a custom PyPI registry handler – e.g., publishing wheels to a private PyPI index.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/registry_handlers/pypi/index.html.md)
- [Use this page when you need to add a custom REST registry handler – e.g., exposing custom endpoints for component discovery.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/registry_handlers/rest/index.html.md)
- [Use this page when you need to share a custom component – e.g., packaging and publishing a plugin wheel.](docs.nvidia.com/aiqtoolkit/latest/extend/sharing-components.html.md)

## Evaluation – Metrics & Pipelines
- [Use this page when you need to run a RAG evaluator – e.g., computing answer relevance.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/eval/rag_evaluator/index.html.md)
- [Use this page when you need to run a trajectory evaluator – e.g., evaluating agent paths.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/eval/trajectory_evaluator/index.html.md)
- [Use this page when you need to run a SWEBench evaluator – e.g., benchmarking code‑generation models.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/eval/swe_bench_evaluator/index.html.md)
- [When you need to add a custom evaluator – e.g., implementing a new metric.](docs.nvidia.com/aiqtoolkit/latest/extend/custom-evaluator.html.md)
- [When you need to configure evaluation datasets – e.g., specifying formats, S3 credentials, or allowlists.](docs.nvidia.com/aiqtoolkit/latest/api/aiq/data_models/dataset_handler/index.html.md)

## Test‑Time Compute & Experimental Features
- [Use this page when you need to configure a test‑time compute strategy – e.g., selecting a plan‑selection method.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/experimental/test_time_compute/models/strategy_base/index.html.md)
- [When adding a new experimental decorator – e.g., flagging a function as experimental.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/experimental/decorators/experimental_warning_decorator/index.html.md)
- [When building a custom test‑time compute editor – e.g., iterative plan refinement.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/experimental/test_time_compute/editing/iterative_plan_refinement_editor/index.html.md)

## Utilities & Helpers
- [Use this page for type conversion utilities – e.g., registering custom converters.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/utils/type_converter/index.html.md)
- [Use this page for reactive programming utilities – e.g., creating Observables and Subjects.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/utils/reactive/index.html.md)
- [Use this page for debugging utilities – e.g., detecting if a debugger is attached.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/utils/debugging_utils/index.html.md)
- [Use this page for string utilities – e.g., pretty‑printing complex objects.](docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/utils/string_utils/index.html.md)

## Release Notes, Contributing, Licensing & Support
- [Read the release notes for the latest AIQ Toolkit features and known issues.](docs.nvidia.com/aiqtoolkit/latest/release-notes.html.md)
- [Read the contributing guide before submitting patches or opening pull requests.](docs.nvidia.com/aiqtoolkit/latest/resources/contributing.html.md)
- [Read the code‑of‑conduct before interacting with the community.](docs.nvidia.com/aiqtoolkit/latest/resources/code-of-conduct.html.md)
- [Read the licensing page when adding new source files or third‑party code.](docs.nvidia.com/nemo/agent-toolkit/latest/resources/licensing.html.md)
- [Read the troubleshooting page when encountering runtime errors or unexpected behavior.](docs.nvidia.com/aiqtoolkit/latest/troubleshooting.html.md)
- [Read the support page for known issues and how to file a new issue.](docs.nvidia.com/aiqtoolkit/latest/support.html.md)