Agent Configuration#

Configuration files define LLM connections, tools, agents, prompts, and telemetry settings. The agent is built on NVIDIA NeMo Agent Toolkit (NAT), which allows you to register custom functions and extend the agent’s capabilities.

File Structure#

general:        # Frontend, telemetry
object_stores:  # Object storage
function_groups: # MCP clients
functions:      # Tools
llms:           # LLM/VLM models
workflow:       # Top agent

LLM Configuration#

llms:
  nim_llm:
    _type: nim
    model_name: nvidia/nvidia-nemotron-nano-9b-v2
    base_url: ${LLM_BASE_URL}/v1
    max_tokens: 4096
    temperature: 0.0

  cosmos_reason2:
    _type: nim
    model_name: nvidia/cosmos-reason2-8b
    base_url: ${COSMOS_BASE_URL}/v1
    max_tokens: 1024

Parameters

Parameter	Type	Description
_type	string	LLM provider type (e.g. `nim` for NVIDIA NIM)
model_name	string	Model identifier (e.g. `nvidia/nvidia-nemotron-nano-9b-v2`)
base_url	string	API endpoint URL with `/v1` suffix
max_tokens	int	Maximum tokens to generate
temperature	float	Sampling temperature (0.0 = deterministic)

MCP Clients#

function_groups:
  video_analytics_mcp:
    _type: mcp_client
    server:
      transport: streamable-http
      url: ${VIDEO_ANALYSIS_MCP_URL}/mcp
    include:
      - video_analytics.get_incidents
      - video_analytics.get_incident
      - video_analytics.get_fov_histogram
      - video_analytics.get_sensor_ids

  vst_mcp:
    _type: mcp_client
    server:
      transport: streamable-http
      url: ${VST_MCP_URL}/mcp
    include:
      - sensor_list
      - get_video_storage_url
      - get_replay_picture_url
      - get_live_picture_url

Parameters

Parameter	Type	Description
_type	string	Must be `mcp_client`
server.transport	string	Transport type: `streamable-http`/ `sse`/ `stdio`
server.url	string	MCP server endpoint URL
include	list	List of tool names to expose from this MCP server

Tools Reference#

video_understanding#

Analyzes video content using a Vision Language Model (VLM).

video_understanding:
  _type: video_understanding
  vlm_name: cosmos_reason2
  max_frames: 30
  num_frames: 16
  min_pixels: 1568
  max_pixels: 208544
  reasoning: false
  video_url_tool: vst_video_url

Parameters

Parameter	Default	Description
vlm_name	(required)	Reference to the VLM from llms section
max_frames	60	Maximum number of frames to sample from the video
num_frames	16	Number of frames to extract for VLM processing
min_pixels	1568	Minimum pixel count per frame for VLM processing
max_pixels	208544	Maximum pixel count per frame for VLM processing
reasoning	false	Enable VLM reasoning mode (Cosmos Reason models only)
filter_thinking	false	Filter out thinking traces from VLM response
video_url_tool	vst_video_url	Tool to get video URL from sensor ID
use_vst	true	Use VST service for video URLs (false uses MinIO)

vst_video_url#

Gets video storage URLs from VST service for a specific stream and time range.

vst_video_url:
  _type: vst_video_url
  get_video_storage_url_tool: vst_mcp.get_video_storage_url
  get_sensor_list_tool: vst_mcp.sensor_list
  vst_base_url: ${VST_BASE_URL}
  overlay_config: true

Parameters

Parameter	Default	Description
get_video_storage_url_tool	vst_get_video_storage_url	MCP tool for getting video URLs
get_sensor_list_tool	vst_sensor_list	MCP tool for getting sensor list
vst_base_url	None	Override base URL from MCP response
expiry_minutes	None	URL expiration time (default: 10080 = 7 days)
container	mp4	Video container format
overlay_config	false	Enable object detection overlays

vst_picture_url#

Gets snapshot picture URLs from VST service (replay or live).

vst_picture_url:
  _type: vst_picture_url
  get_sensor_list_tool: vst_mcp.sensor_list
  get_replay_picture_url_tool: vst_mcp.get_replay_picture_url
  get_live_picture_url_tool: vst_mcp.get_live_picture_url
  vst_base_url: ${VST_BASE_URL}
  max_retries: 5
  retry_delay_seconds: 2

Parameters

Parameter	Default	Description
get_replay_picture_url_tool	vst_get_replay_picture_url	MCP tool for replay pictures
get_live_picture_url_tool	None	MCP tool for live pictures (optional)
get_sensor_list_tool	vst_sensor_list	MCP tool for getting sensor list
vst_base_url	None	Override base URL from MCP response
expiry_minutes	None	URL expiration time (default: 7 days)
max_retries	3	Maximum retries for getting picture URL
retry_delay_seconds	1	Delay between retries

chart_generator#

Generates bar and pie charts, stores them in object store.

chart_generator:
  _type: chart_generator
  object_store_name: ${VSS_AGENT_OBJECT_STORE_TYPE}
  object_store_base_url: ${VSS_AGENT_REPORTS_BASE_URL}

Parameters

Parameter	Default	Description
object_store_name	None	Object store reference for storing images
object_store_base_url	http://localhost:8000/static/	Base URL for serving chart images

multi_incident_formatter#

Fetches and formats multiple incidents with video/snapshot URLs and optional charts.

multi_incident_formatter:
  _type: multi_incident_formatter
  video_url_tool: vst_video_url
  picture_url_tool: vst_picture_url
  incidents_tool: video_analytics_mcp.video_analytics.get_incidents
  chart_generator_tool: chart_generator
  generate_chart: true
  chart_base_url: ${VSS_AGENT_REPORTS_BASE_URL}
  display_limit: 20

Parameters

Parameter	Default	Description
video_url_tool	(required)	Tool for getting video URLs
picture_url_tool	(required)	Tool for getting picture URLs
incidents_tool	(required)	Tool for fetching incidents
chart_generator_tool	None	Tool for generating charts (optional)
generate_chart	false	Auto-generate incident visualization charts
chart_base_url	http://localhost:38000/reports/	Base URL for chart images
display_limit	20	Max incidents to display with full details

template_report_gen#

Generates incident reports using VLM analysis and customizable templates.

template_report_gen:
  _type: template_report_gen
  object_store: ${VSS_AGENT_OBJECT_STORE_TYPE}
  base_url: ${VSS_AGENT_REPORTS_BASE_URL}
  template_path: ${VSS_AGENT_TEMPLATE_PATH}
  template_name: ${VSS_AGENT_TEMPLATE_NAME}
  llm_name: nim_llm
  video_understanding_tool: video_understanding
  picture_url_tool: vst_picture_url
  video_url_tool: vst_video_url
  vlm_prompts:
    - "Describe the incident in details..."
    - "Describe the location of the incident..."

Parameters

Parameter	Default	Description
object_store	(required)	Object store reference for storing reports
base_url	http://localhost:8000/static	Base URL for serving report files
template_path	“”	Path to template directory
template_name	None	Template file name (e.g., incident_report.md)
llm_name	“”	LLM for report generation
video_understanding_tool	“”	VLM tool for video analysis
picture_url_tool	vst_picture_url	Tool for snapshot URLs
video_url_tool	None	Tool for video URLs
vlm_prompts	[]	List of prompts for VLM analysis
report_prompt	“”	System prompt for report generation
agent_version	v1.0.0	Agent version to include in report
save_local_copy	false	Save local copy of reports
output_dir	./agent_reports	Directory for local copies

get_fov_counts_with_chart#

Gets field-of-view object counts and generates visualization charts.

get_fov_counts_with_chart:
  _type: get_fov_counts_with_chart
  get_fov_histogram_tool: video_analytics_mcp.video_analytics.get_fov_histogram
  chart_generator_tool: chart_generator
  chart_base_url: ${VSS_AGENT_REPORTS_BASE_URL}

Parameters

Parameter	Default	Description
get_fov_histogram_tool	(required)	Tool for FOV histogram data
chart_generator_tool	(required)	Tool for chart generation
chart_base_url	http://localhost:38000/reports/	Base URL for chart images

geolocation#

Fetches geolocation information from coordinates using OpenStreetMap.

geolocation:
  _type: geolocation
  timeout: 10

Parameters

Parameter	Default	Description
timeout	10	Request timeout in seconds for OpenStreetMap API

Agents#

report_agent#

Sub-agent for generating detailed single incident reports.

report_agent:
  _type: report_agent
  get_incidents_tool: video_analytics_mcp.video_analytics.get_incidents
  get_incident_tool: video_analytics_mcp.video_analytics.get_incident
  template_report_tool: template_report_gen

Parameters

Parameter	Type	Description
_type	string	Must be `report_agent`
get_incidents_tool	string	Tool to fetch incident list
get_incident_tool	string	Tool to fetch single incident details
template_report_tool	string	Tool for generating templated reports

multi_report_agent#

Sub-agent for listing and summarizing multiple incidents.

multi_report_agent:
  _type: multi_report_agent
  multi_incident_tool: multi_incident_formatter
  max_incidents: 9999

Parameters

Parameter	Type	Description
_type	string	Must be `multi_report_agent`
multi_incident_tool	string	Tool for formatting multiple incidents
max_incidents	int	Maximum number of incidents to process

Top Agent (Workflow)#

The main routing agent that orchestrates tools and sub-agents.

workflow:
  _type: top_agent
  llm_name: nim_llm
  log_level: INFO
  max_retries: 3
  max_iterations: 15
  llm_reasoning: false
  tool_names:
    - get_fov_counts_with_chart
    - video_analytics_mcp.video_analytics.get_sensor_ids
    - vst_picture_url
  subagent_names:
    - report_agent
    - multi_report_agent
  prompt: |
    You are a routing agent...

Parameters

Parameter	Description
llm_name	LLM from llms section
log_level	DEBUG, INFO, WARNING, ERROR
max_iterations	Max reasoning iterations
tool_names	Direct tools for agent
subagent_names	Sub-agents for complex tasks
prompt	System prompt

Telemetry#

general:
  telemetry:
    tracing:
      phoenix:
        _type: phoenix
        endpoint: ${PHOENIX_ENDPOINT}/v1/traces
        project: WH-vss-agent-${VSS_AGENT_VERSION}

For details on enabling telemetry and navigating traces in the Phoenix UI, see Observability.

Parameters

Parameter	Type	Description
_type	string	Telemetry provider (e.g. `phoenix`)
endpoint	string	Tracing endpoint URL (e.g. `${PHOENIX_ENDPOINT}/v1/traces`)
project	string	Project name for trace grouping

Evaluation Configuration#

The eval: section configures the evaluation framework for assessing agent performance. For complete evaluation configuration options and evaluator types, see Agent Evaluation.

Environment Variables#

Environment Variables Reference

Variable	Description
HOST_IP	Blueprint endpoint IP address
VSS_AGENT_HOST	Agent server host (default: 0.0.0.0)
VSS_AGENT_PORT	Agent server port (default: 8000)
VSS_AGENT_VERSION	Agent version for telemetry project naming
VSS_AGENT_CONFIG_FILE	Path to agent config YAML file
VSS_AGENT_OBJECT_STORE_TYPE	Object store type (local_object_store or remote_object_store)
VSS_AGENT_REPORTS_BASE_URL	Base URL for serving generated reports
VSS_AGENT_TEMPLATE_PATH	Path to report template directory
VSS_AGENT_TEMPLATE_NAME	Report template filename
VST_BASE_URL	Video Storage Tool base URL
VST_PORT	VST server port (default: 30888)
VST_MCP_URL	VST MCP server URL
VST_MCP_PORT	VST MCP server port (default: 8001)
VIDEO_ANALYSIS_MCP_URL	VA-MCP server URL
VSS_VA_MCP_PORT	VA-MCP server port (default: 9901)
VSS_VA_MCP_CONFIG_FILE	Path to VA-MCP config file
VSS_ES_PORT	Elasticsearch port (default: 9200)
MDX_PORT	MDX API port (default: 8081)
LLM_NAME	LLM model name
LLM_BASE_URL	LLM NIM endpoint URL
NEMOTRON_PORT	Nemotron NIM port (default: 30081)
COSMOS_BASE_URL	Cosmos Reason VLM endpoint URL
COSMOS_PORT	Cosmos NIM port (default: 30082)
PHOENIX_ENDPOINT	Phoenix telemetry endpoint

Example Environment File#

Example .env file

# Host Configuration
HOST_IP='10.20.30.40'

# VSS Agent Configuration
VSS_AGENT_VERSION=v1.0.0
VSS_AGENT_CONFIG_FILE=./deployments/warehouse/vss-agent/configs/config.yml
VSS_AGENT_HOST=0.0.0.0
VSS_AGENT_PORT=8000
VSS_AGENT_OBJECT_STORE_TYPE=local_object_store
VSS_AGENT_REPORTS_BASE_URL=http://${HOST_IP}:${VSS_AGENT_PORT}/static/
VSS_AGENT_TEMPLATE_PATH=./deployments/warehouse/vss-agent/templates
VSS_AGENT_TEMPLATE_NAME=incident_report_template.md

# VA-MCP Configuration
VSS_VA_MCP_CONFIG_FILE=./deployments/warehouse/vss-agent/configs/va_mcp_server_config.yml
VSS_VA_MCP_PORT=9901
VSS_ES_PORT=9200

# VST Configuration
VST_PORT=30888
VST_BASE_URL=http://${HOST_IP}:${VST_PORT}
VST_MCP_PORT=8001
VST_MCP_URL=http://${HOST_IP}:${VST_MCP_PORT}

# LLM Configuration
LLM_NAME=nvidia/nvidia-nemotron-nano-9b-v2
NEMOTRON_PORT=30081
LLM_BASE_URL=http://${HOST_IP}:${NEMOTRON_PORT}

# Cosmos VLM Configuration
COSMOS_PORT=30082
COSMOS_BASE_URL=http://${HOST_IP}:${COSMOS_PORT}

# Telemetry
PHOENIX_ENDPOINT=http://${HOST_IP}:6006