Agent Configuration

Configuration files define model connections, MCP clients, tools, agents, prompts, object stores, and telemetry. The VSS Agent is built on NVIDIA NeMo Agent Toolkit.

The examples on this page follow the shipped Warehouse configuration:

deploy/docker/industry-profiles/warehouse-operations/vss-agent/configs/config.yml

Developer profiles use the same top-level structure but select different tools. Use the configuration file for the profile you deploy as the source of truth:

• deploy/docker/developer-profiles/dev-profile-base/vss-agent/configs/config.yml
• deploy/docker/developer-profiles/dev-profile-lvs/vss-agent/configs/config.yml
• deploy/docker/developer-profiles/dev-profile-search/vss-agent/configs/config.yml
• deploy/docker/developer-profiles/dev-profile-alerts/vss-agent/configs/config.yml

File Structure

general:         # Frontend and telemetry
object_stores:   # Report and chart storage
function_groups: # MCP clients
functions:       # Tools and sub-agents
llms:            # LLM and VLM connections
workflow:        # Top-level routing agent

Model Configuration

The llms section registers model endpoints with NAT. Names in this section are references used by tools and agents.

llms:
  llm:
    _type: ${LLM_MODEL_TYPE:-nim}
    model_name: ${LLM_NAME}
    base_url: ${LLM_BASE_URL}/v1
    max_tokens: 4096
    temperature: 0.0
  rtvi_vlm:
    _type: openai
    model_name: ${VLM_NAME}
    base_url: ${RTVI_VLM_BASE_URL}/v1
    temperature: 0.0

See Configure the LLM and Configure the VLM for supported model-placement examples.

MCP Clients

The Warehouse configuration connects to Video Analytics MCP with the streamable HTTP transport.

function_groups:
  video_analytics_mcp:
    _type: mcp_client
    server:
      transport: streamable-http
      url: ${VIDEO_ANALYSIS_MCP_URL}/mcp
    include:
      - video_analytics__get_incidents
      - video_analytics__get_incident
      - video_analytics__get_fov_histogram
      - video_analytics__get_sensor_ids

Tool references use the client name, a period, and the exposed tool name. For example: video_analytics_mcp.video_analytics__get_incidents.

Tools Reference

video_understanding

Analyzes video with the configured VLM.

video_understanding:
  _type: video_understanding
  vlm_name: ${VLM_MODEL_TYPE:-nim}_vlm
  max_frames: 30
  min_pixels: 3136
  max_pixels: 12845056
  reasoning: false
  video_url_tool: vst_video_url
  vlm_mode: ${VLM_MODE}
  vst_internal_url: ${VST_INTERNAL_URL}
  enable_audio: ${ENABLE_AUDIO:-false}

The time_format value must match the related VST clip and snapshot tools.

VST clip and snapshot tools

Tool instance names are profile-defined. The Warehouse profile names the instances vst_video_url and vst_picture_url, while their registered NAT types are vst.video_clip and vst.snapshot.

vst_video_url:
  _type: vst.video_clip
  vst_internal_url: ${VST_INTERNAL_URL}
  vst_external_url: ${VST_EXTERNAL_URL}
  overlay_config: true
  time_format: iso
  enable_audio: ${ENABLE_AUDIO:-false}
vst_picture_url:
  _type: vst.snapshot
  vst_internal_url: ${VST_INTERNAL_URL}
  vst_external_url: ${VST_EXTERNAL_URL}
  overlay_config: true
  time_format: iso

VST sensor-list tool

The Warehouse top agent uses vst_sensor_list to retrieve the sensor names currently available from VST.

vst_sensor_list:
  _type: vst.sensor_list
  vst_internal_url: ${VST_INTERNAL_URL}

chart_generator

Generates bar and pie charts and stores them in the configured object store.

multi_incident_formatter

Formats incident data with video and snapshot URLs and optional charts.

template_report_gen

Generates incident reports from VLM analysis and an optional Markdown template.

get_fov_counts_with_chart

Fetches field-of-view object counts and generates a chart.

geolocation

The Smart Cities profile can use this tool to resolve coordinates through OpenStreetMap.

Agents

report_agent

Generates a report for an incident or directly uploaded video.

multi_report_agent

Lists and formats multiple incidents.

Top Agent Workflow

The top agent routes requests to direct tools and sub-agents.

workflow:
  _type: top_agent
  llm_name: llm
  log_level: INFO
  max_iterations: 15
  max_history: 20
  llm_reasoning: true
  planning_enabled: true
  tool_names:
    - get_fov_counts_with_chart
    - vst_sensor_list
    - vst_picture_url
  subagent_names:
    - report_agent
    - multi_report_agent
  prompt: |
    You are a routing agent...

Telemetry

general:
  telemetry:
    tracing:
      phoenix:
        _type: phoenix
        endpoint: ${PHOENIX_ENDPOINT}/v1/traces
        project: WH-vss-agent-${VSS_AGENT_VERSION}

For setup and trace inspection, see Agent Observability.

Evaluation Configuration

The eval: section configures the evaluation framework. See Agent Evaluation for evaluator types and examples.

Environment Variables

Profile .env files and the final local Compose override file supply values to the Agent container. Create and select that file as described in Configuration.

GET /api/v1/version reports the installed nvidia-vss-core version, stamped into the image at build time as <release line>+tree.<source tree sha> — the release line being the nearest v* git tag (3.3.0-rc0+tree.<sha> on the 3.3.0 line before its release). No environment variable sets or overrides it; nothing needs to be configured for a stock image to report its version.

The checked-in profile files already define the service URLs and stable defaults. For a Docker Compose deployment, a user-overrides.env file should contain only values you need to change. See Configuration for file precedence. For example, an override can contain:

VLM_MODE=local_shared
LLM_NAME=nvidia/nemotron-3.5-lightning-30b-a3b
LLM_BASE_URL=http://vss-llm-nim:8000
VLM_NAME=nim_nvidia_cosmos3-nano-reasoner_bf16-final
VLM_BASE_URL=http://rtvi-vlm:8000
VLM_MODEL_TYPE=rtvi