Getting Started#

This guide will help you access and start using the VSS Auto Calibration Microservice and User Interface.

Prerequisites#

Before using the microservice and UI, ensure you have:

System Requirements

  • x86_64 system

  • OS Ubuntu 24.04

  • NVIDIA GPU with hardware encoder (NVENC)

  • NVIDIA driver 590

  • Docker (setup to run without sudo privilege)

  • NVIDIA container toolkit (Refer to the Prerequisites section)

Required

  • One or more camera video files (MP4) or time-synchronized RTSP streams; one for single-camera calibration, or at least two for multi-camera

  • Layout/map image (PNG format)

Optional

  • Ground truth data (ZIP file) for calibration evaluation

  • Pre-existing alignment data (JSON file)

  • Focal length values for cameras

  • Config parameters if any for your dataset

Deployment Steps (Docker Compose)

Deploy the UI and backend microservice using Docker Compose. Refer to the Warehouse Blueprint Introduction for more details.

To use RTSP capture via VIOS, ensure the VIOS server is set up and running. Set VIOS_BASE_URL in the .env file as described in the deployment environment variable setup.

  1. Setup NGC access and authenticate Docker to pull images from nvcr.io:

    # Setup NGC access
export NGC_CLI_API_KEY=<NGC_CLI_API_KEY>
export NGC_CLI_ORG='nvidia'

# Authenticate Docker to nvcr.io (required before docker compose pull/up)
docker login nvcr.io -u '$oauthtoken' -p "${NGC_CLI_API_KEY}"

    Generate an NGC API key from NGC Setup if you do not already have one. Docker must be logged in to nvcr.io before the containers in the steps below can be pulled.

  2. Download deployment resources by cloning the Git repository:

    git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git

  3. Navigate to the VSS Auto Calibration directory:

    cd video-search-and-summarization/deploy/docker/services/auto-calibration

    Your directory structure should be:

    ├── compose.yml
├── ms
│   └── compose.yml
└── ui
    └── compose.yml

  4. Create a .env file in your current directory with the following environment variables:

    VSS_AUTO_CALIBRATION_PORT=8000
VSS_AUTO_CALIBRATION_UI_PORT=5000
VSS_APPS_DIR=/path/to/video-search-and-summarization/deploy/docker
VSS_DATA_DIR=/path/to/your/data_dir
HOST_IP=<HOST_IP_ADDRESS>
# VIOS_BASE_URL=http://<VIOS_HOST_IP>:30888  # Uncomment and update this to support RTSP Stream input

    Replace the paths, ports and IP address with your actual values.

    Note

    This guide sets VSS_AUTO_CALIBRATION_PORT=8000 in .env, which maps the backend to port 8000 on the host. The Auto Calibration service compose.yml in the repository may use a different default (for example 8010) when VSS_AUTO_CALIBRATION_PORT is not set—always use the port you configured in .env when opening the API or troubleshooting connectivity.

  5. Download and set up the VGGT model and create projects directory:

    1. Download the VGGT commercial model from HuggingFace.

      Note

      You need to sign up for a HuggingFace account and accept the model license to download.

    2. Move the downloaded model file (vggt_1B_commercial.pt) to the VGGT model directory:

      mkdir -p ${VSS_DATA_DIR}/auto-calib/vggt
mv vggt_1B_commercial.pt ${VSS_DATA_DIR}/auto-calib/vggt/

    3. Create projects directory:

      mkdir -p ${VSS_APPS_DIR}/services/auto-calibration/projects

    Note

    Projects will be saved in: ${VSS_APPS_DIR}/services/auto-calibration/projects

  6. Change the ownership of the directory to UID 1000 and GID 1000:

    sudo chown -R 1000:1000 ${VSS_DATA_DIR}/auto-calib
sudo chown -R 1000:1000 ${VSS_APPS_DIR}/services/auto-calibration/projects

  7. Ensure you have access to NGC to pull the containers.

  8. Start both the microservice and UI servers:

    docker compose --profile "auto_calib" up -d

  9. Open your browser and navigate to:

    http://<HOST_IP>:<VSS_AUTO_CALIBRATION_UI_PORT>

    For example, with the default settings: http://<HOST_IP>:5000

  10. To stop the containers

    docker compose --profile "auto_calib" down

First Time Setup#

When you first access the UI, you’ll see the main interface with a stepper showing 6 workflow steps.

First Launch Screen

Interface Overview

The interface consists of:

  1. Header Bar

    • Application name and version

    • Theme toggle button (light/dark mode)

  2. Stepper Navigation

    • Visual progress indicator

    • Click on steps to navigate (after selecting a project)

    • Current step is highlighted

  3. Main Content Area

    • Step-specific content and controls

    • Forms, file uploads, and interactive tools

  4. Navigation Buttons

    • “Previous” button to go back

    • “Next” button to proceed

    • Disabled when requirements aren’t met

  5. Footer

    • Copyright information

    • Application version

  6. Notifications

    • Success/error messages appear in bottom-right corner

    • Auto-dismiss after 6 seconds

Quick Start Guide#

Follow these steps to perform your first calibration:

Step 0: Deploy the UI (If Not Already Running)

If you’re deploying via Docker Compose, follow the steps in the Deployment Steps (Docker Compose) section above. Once deployed, access the UI at http://<HOST_IP>:<VSS_AUTO_CALIBRATION_UI_PORT> (default: port 5000).

Step 1: Create a Project

  1. On the Project Setup page, enter a project name (e.g., warehouse_calibration)

  2. Click “Create” button

  3. Your new project appears in the list below

  4. Click “Select” on your project card

Project Setup

Step 2: Upload Files

  1. Click “Next” to go to Video Configuration

  2. Provide one or more time-synchronized camera inputs using either file upload or RTSP capture (one camera for single-camera calibration, two or more for multi-camera; the UI blocks mixing an active file upload queue with RTSP, and file-uploaded clips must be removed before switching to RTSP).

    Option A: Upload video files

    • Click “Select Videos” button

    • Select video files named cam_00.mp4, cam_01.mp4, etc.

    • Reorder videos by dragging.

    • Click “Upload Videos” to upload the videos.

    Option B: RTSP capture (VIOS) (shown only when the Auto Calibration service exposes RTSP capture; VIOS is configured on the server side)

    • Finish or clear any pending Video Files selection or upload before starting RTSP; if the project already has clips from file upload, remove them under Video Files before using RTSP.

    • In the RTSP capture (VIOS) card, set Duration (seconds). The server requires at least 60 seconds.

    • Under Streams, enter all camera RTSP URLs for this project in one session (use Add stream for additional cameras; one stream for single-camera, two or more for multi-camera). Optionally set Camera name for each stream.

    • Click Start capture once with every stream listed—capture runs for all streams together. Do not add streams at different times or run separate captures for different subsets; staggered captures break time synchronization.

    • Wait for recording to progress (status chip and progress bar). You may click Stop early only after at least 60 seconds of active recording have elapsed.

    • When the session is finished (COMPLETED or CANCELLED), click Ingest to project to add the captured clips to the project’s video list. While capture or ingest is running, Video Files upload stays disabled until the RTSP pipeline completes.

    Note

    RTSP inputs must be time-synchronized: configure every stream, then run one Start capture for the full set. Re-capturing or ingesting additional streams later (instead of a single combined capture) will desync clips and is not supported for calibration. List each RTSP URL in order of overlapping field of view (FOV)—the same ordering used when uploading video files (first stream = first camera in the overlap chain).

    Note

    For VIOS pre-registered RTSP streams, use the source URL (for example, the NVStreamer URL if the stream originates from NVStreamer) rather than the VIOS-proxied URL.

  3. Upload layout image:

    • Click “Upload Layout” button

    • Select your PNG format layout/map image

    • Confirm upload success

Video Configuration

Step 3: Configure Parameters

  1. Click “Next” to go to Parameters

  2. Under Annotation target, choose where to draw ROIs and tripwires (all optional):

    Camera — per-camera annotations on a video frame

    • Select a camera from Select Camera

    • Use Drawing Tools on the video frame:

      • Draw ROI: click to add points (minimum 3), press F or right-click to finish; saved automatically

      • Draw Tripwire: click twice for start and end (red line); saved automatically

      • Tripwire Direction: click twice for a directional line (yellow, with arrow); saved automatically

    Global ROIs / tripwires — layout-map annotations (requires a layout image from Step 2)

    • Select Global ROIs / tripwires (disabled until layout.png is uploaded in Video Configuration)

    • Draw on the floor-plan / layout image using the same Drawing Tools (Draw ROI, Draw Tripwire, Tripwire Direction)

    • Global shapes are stored separately from per-camera annotations and appear under Current Annotations as Global (layout map)

    For both targets you can use Show/Hide to toggle overlays and Reset to clear annotations for the active target.

  3. Export image-mode JSON (optional, right panel):

    • Click Export image-mode JSON to download <project_name>_image_mode_exported.json

    • Exports ROIs and tripwires in pixel coordinates (calibrationType: image); bundle adjustment is not required for this export

  4. Add focal lengths (optional):

    • Enter comma-separated values (one per camera)

    • Click “Save Focal Length”

  5. Configure parameters (optional):

    • On the Parameters step, click the Click the icon to customize calibration parameters icon in the top-right (visible only on this step)

    • Upload a pre-configured settings file, or manually adjust calibration parameters in the dialog

    • Use Download to export the current configuration, Reset to Defaults to restore defaults, then Save Settings

    • Make all changes before starting calibration in Step 5: Execute; do not change settings while AMC is running

Parameters Configuration

Step 4: Create Alignment Data

  1. Click “Next” to go to Manual Alignment

  2. Choose one of two options:

    Option A: Upload Existing Alignment

    • Click “Upload alignment_data.json”

    • Select your JSON file

    • Wait for upload confirmation

    Option B: Create Alignment Interactively

    • Click “Open Alignment Tool”

    • Click the same physical point on Camera 0, Camera 1, and Layout (in order)

    • Repeat for at least 4 different points

    • Click “Save Alignment” when complete

Manual Alignment

Step 5: Run Calibration

  1. Click “Next” to go to Execute

  2. Review the requirements checklist

  3. Click “Verify Project” button

  4. Once verified, click “Start Calibration”

  5. Monitor the progress (status updates every 3 seconds)

  6. Wait for “Calibration completed successfully” message (AMC), or for AMC to finish far enough that optional VGGT can run (see below)

Execute Calibration

Step 5b: Run VGGT Calibration (Optional, Multi-Camera Only)

VGGT (Vision-Geometry Graph Transformer) is an optional model-based calibration path. It is available only when the backend has the VGGT model installed (see Deployment Steps above) and your project has two or more cameras—single-camera projects do not support VGGT.

  1. After Start Calibration (AMC) has run and produced on-disk outputs (typically after rectification), scroll to Calibration Control (VGGT) on the Execute step

  2. Confirm VGGT state is READY (AMC does not need to show COMPLETED; VGGT can remain runnable if AMC failed after rectification)

  3. Click Run VGGT Calibration and monitor progress

  4. When VGGT completes successfully, the project state becomes COMPLETED even if AMC failed—you can proceed to Results using the VGGT tab

Step 6: View Results

  1. Click “Next” to go to Results

  2. View the overlay image showing calibration results. Click “Download” to save overlay image

  3. Review camera parameters for each camera

  4. Export calibration data:

    • Click “Full Export AMC” for complete calibration data. It will load the json which contains the calibration data for all the cameras in the project, which you can edit according to your usecase.

    • Click “MV3DT ZIP AMC” for MV3DT-compatible format

Calibration Results

Settings#

Click the settings icon in the top-right corner to access calibration parameters. Note: The settings icon is only visible when you are on the Parameters step.

Settings Dialog

Additional Available Settings

  • Theme: Switch between light and dark themes

  • Version Information: View current application version

Note: Most settings are configured during deployment and cannot be changed from the UI.

Tips for Success#

Input Video Convention

  • Ensure videos are synchronized in time

  • While uploading videos maintain the order based on their FOV overlapping.

Input Video

  • Use high-resolution videos for better calibration accuracy

  • AMC is heavily dependent on the moving people instances in the videos. The videos should have enough number of clearly visible moving people.

Alignment Points

  • Choose points on the ground plane visible in all cameras

  • Select points at different depths and locations

  • Avoid points on moving objects

  • Use distinct features (corners, markings, etc.)

ROI and Tripwire Drawing

  • Draw ROIs to cover areas of interest

  • Place tripwires perpendicular to expected motion

  • Use tripwire directions to indicate motion direction

  • Test with different zoom levels for precision

Keyboard Shortcuts#

Parameters Step (Drawing)

  • F key: Finish current ROI

  • Esc key: Cancel current drawing

  • Scroll wheel: Zoom in/out on canvas

  • Click + Drag: Pan around zoomed canvas

Manual Alignment Step

  • Scroll wheel: Zoom in/out on alignment canvas

  • Click + Drag: Pan around zoomed canvas (when zoomed)

Next Steps#

Now that you’re familiar with the basics, explore: