Agent Customization#

The VSS Agent is built on NVIDIA NeMo Agent Toolkit (NAT), which allows you to extend the agent with custom functions, tools, and MCP servers.

Custom Functions#

Custom functions are registered in the functions: section of your config file. Each function references a registered Python function by its _type name.

functions:
  my_custom_tool:
    _type: my_custom_tool
    param1: value1
    param2: value2

workflow:
  _type: top_agent
  tool_names:
    - my_custom_tool

For detailed documentation on writing and registering custom functions, see: NAT Writing Custom Functions

Adding Custom MCP Servers#

You can connect to your own MCP (Model Context Protocol) servers to extend the agent’s capabilities. MCP clients are configured in the function_groups: section.

Supported Transports:

Transport	Use Case
`streamable-http`	HTTP-based MCP servers (recommended)
`sse`	Server-Sent Events transport
`stdio`	Local process communication

Configuration Example:

function_groups:
  my_custom_mcp:
    _type: mcp_client
    server:
      transport: streamable-http
      url: http://localhost:9000/mcp
    include:
      - my_tool_1
      - my_tool_2

workflow:
  _type: top_agent
  tool_names:
    - my_custom_mcp.my_tool_1
    - my_custom_mcp.my_tool_2

CLI Tool Inspection:

You can discover available tools from an MCP server before configuring:

# List all tools from an MCP server
nat mcp client tool list --url http://localhost:9000/mcp

# Get details about a specific tool
nat mcp client tool list --url http://localhost:9000/mcp --tool my_tool_1

For complete MCP client documentation including authentication, timeout configuration, reconnection settings, and troubleshooting, see: NAT MCP Client Documentation

Building Your Own MCP Server#

To create your own MCP server that can be consumed by the VSS Agent:

Customizing Report Generation#

To learn more about customizing agent report generation, see: Report Generation

Customize Models

Adding a Knowledge Layer (RAG)#

Opt-in. A pluggable retrieval surface registered as _type: knowledge_retrieval, exposed to the LLM as a named tool. Five backends ship today — two in the default agent image, three behind opt-in Python extras (marked experimental below):

Backend	Use case	`backend_config` fields
`frag_api`	Document grounding via a deployed NVIDIA RAG Blueprint rag-server.	`rag_url` `collection_name` `api_key` (optional)
`es_caption`	Q&A over stored Video Summarization stream captions in Elasticsearch.	`elasticsearch_url` `index` (default: `default_`) `api_key` (optional)*
`frag_lib` (experimental)	In-process document grounding via the nvidia-rag Python library (no rag-server hop).	`milvus_uri` `collection_name` `embedder_base_url`, `embedder_model_name` `llm_base_url`, `llm_model_name`
`langchain` (experimental)	Embedded Chroma vector store via LangChain. Requires a pre-built persist directory mounted into the agent.	`persist_dir` `collection_name` `embed_model`, `embed_base_url` `embed_api_key`
`llama_index` (experimental)	Embedded Chroma vector store via LlamaIndex. Requires a pre-built persist directory mounted into the agent.	`persist_dir` `collection_name` `embed_model`, `embed_base_url` `embed_api_key`

frag_api and es_caption also accept verify_ssl and timeout. es_caption is already wired into the Video Summarization profile as lvs_caption_retrieval (see Agent Profiles).

Note

Filtering by stream name is not supported today; by default it searches all matching indices (per the index pattern) and uses the stream ID as a filter.

To enable an experimental backend, add --extra <backend> to the uv sync line in services/agent/docker/Dockerfile and rebuild the agent image.

To enable a retriever, update the profile’s config file (or add a new one) with three patches:

1. Register the function block.

functions:
  my_retriever:                 # tool name surfaced to the LLM
    _type: knowledge_retrieval
    backend: frag_api           # or es_caption
    top_k: 5
    backend_config:
      # backend-specific fields per the table above

2. Expose it to the workflow.

workflow:
  tool_names:
    - my_retriever

3. Update the routing rules in ``workflow.prompt`` to tell the LLM when to use this tool.

Reference config_rag.yml files ship under deploy/docker/developer-profiles/<profile>/vss-agent/configs/ for both dev-profile-base and dev-profile-lvs. Use one of these — or your own custom config — by pointing VSS_AGENT_CONFIG_FILE at it in your deployment.