Getting Started#

This guide will help you access and start using the AutoMagicCalib Microservice and User Interface.

Tracklet-Based Calibration: Input Video Requirements#

AutoMagicCalib (AMC) estimates camera parameters by detecting and tracking people in your footage, then matching those trajectories (tracklets) across camera views. Calibration quality depends heavily on input video content. Review these requirements before you deploy the service or record footage.

Important

These requirements apply to the tracklet-based AMC pipeline (the default calibration path). This is not applicable to the optional VGGT model-based calibration path.

Video content (required for AMC)

  • Moving people: People must be clearly visible and moving throughout the clip. AMC relies on PeopleNet detection and 3D tracking of people in the scene.

  • Scene coverage: Trajectories should span as much of each camera’s field of view as possible. More unique walkers and broader coverage produce more usable tracklets.

  • Recommended headcount: Plan for many moving people in the scene. As a practical guideline, aim for at least 10 unique individuals walking through the monitored area during the recording window (more is better for multi-camera overlap).

Video duration

  • Recommended: five minutes or longer per camera. There is no strict minimum, but given the size of the space to calibrate and normal walking speed, longer clips let the tracker build stable trajectories across the field of view; short clips often fail during multi-view tracklet matching.

Resolution and format

  • Resolution: 1920 × 1080

  • Format (file upload): MP4

Multi-camera specifics

  • Camera count: Two or more time-synchronized videos or RTSP streams (one stream for single-camera calibration).

  • Time synchronization: All cameras must cover the same time window. Use clips recorded in sync or post-processed to be in sync.

  • FOV order: List cameras in order of overlapping field of view (cam_00 overlaps cam_01, cam_01 overlaps cam_02, and so on).

  • FOV overlap: For the best calibration quality, more overlap between consecutive camera pairs is better. Aim for at least 30% FOV overlap; insufficient overlap reduces matched tracklets and often causes multi-view calibration to fail.

Tracklet thresholds (why the above matters)

AMC filters and matches tracklets before multi-view calibration. Default pipeline settings include:

  • Minimum tracklet length: 90 frames (approximately three seconds at 30 fps)—short or jittery tracks are discarded.

  • Minimum matched tracklets (multi-camera): six cross-camera tracklet correspondences per camera pair (three in robust two-view fallback).

If your videos lack enough moving people or are too short, calibration may fail at detection, tracking, or tracklet matching. You may need to adjust the configuration parameters through UI or capture better videos. See Custom Dataset for capture best practices and Troubleshooting if tracklet matching fails.

Prerequisites#

Before using the microservice and UI, ensure you have:

System Requirements

  • x86_64 system

  • OS Ubuntu 24.04

  • NVIDIA GPU with hardware encoder (NVENC)

  • NVIDIA driver 590

  • Docker (setup to run without sudo privilege)

  • NVIDIA container toolkit (Refer to the Prerequisites section)

Required

  • Input videos or RTSP streams that meet tracklet-based video requirements above

  • Layout/map image (PNG format)

  • Layout pixel per meter: Number of pixels in the layout image per meter in the real world; This needs to be updated in the configuration parameters

Optional

  • Ground truth data (ZIP file) for calibration evaluation

  • Pre-existing alignment data (JSON file)

  • Focal length values for cameras

  • Config parameters if any for your dataset

Deployment Steps (Docker Compose)

Note

To automate Auto Calibration deployment and calibration with the bundled VSS Agent Skills instead of following the manual steps below, see the Agent Skills walkthrough later in this guide.

Deploy the UI and backend microservice using Docker Compose. Refer to the Warehouse Blueprint Introduction for more details.

To use RTSP capture via VIOS, ensure the VIOS server is set up and running. Set VIOS_BASE_URL in the .env file as described in the deployment environment variable setup.

  1. Setup NGC access and authenticate Docker to pull images from nvcr.io:

    # Setup NGC access
export NGC_CLI_API_KEY=<NGC_CLI_API_KEY>
export NGC_CLI_ORG='nvidia'

# Authenticate Docker to nvcr.io (required before docker compose pull/up)
docker login nvcr.io -u '$oauthtoken' -p "${NGC_CLI_API_KEY}"

    Generate an NGC API key from NGC Setup if you do not already have one. Docker must be logged in to nvcr.io before the containers in the steps below can be pulled.

  2. Download deployment resources by cloning the Git repository:

    git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout tags/v3.2.1

  3. Navigate to the AutoMagicCalib directory:

    cd deploy/docker/services/auto-calibration

    Your directory structure should be:

    ├── compose.yml
├── ms
│   └── compose.yml
└── ui
    └── compose.yml

  4. Create a .env file in your current directory with the following environment variables:

    VSS_AUTO_CALIBRATION_PORT=8000
VSS_AUTO_CALIBRATION_UI_PORT=5000
VSS_APPS_DIR=/path/to/video-search-and-summarization/deploy/docker
VSS_DATA_DIR=/path/to/your/data_dir
HOST_IP=<HOST_IP_ADDRESS>
# VIOS_BASE_URL=http://<VIOS_HOST_IP>:30888  # Uncomment and update this to support RTSP Stream input

    Replace the paths, ports and IP address with your actual values.

    Note

    This guide sets VSS_AUTO_CALIBRATION_PORT=8000 in .env, which maps the backend to port 8000 on the host. The Auto Calibration service compose.yml in the repository may use a different default (for example 8010) when VSS_AUTO_CALIBRATION_PORT is not set—always use the port you configured in .env when opening the API or troubleshooting connectivity.

  5. Download and set up the VGGT model (optional):

    1. Download the VGGT commercial model from HuggingFace.

      Note

      You need to sign up for a HuggingFace account and accept the model license to download.

    2. Move the downloaded model file (vggt_1B_commercial.pt) to the VGGT model directory:

      mkdir -p ${VSS_DATA_DIR}/auto-calib/vggt
mv vggt_1B_commercial.pt ${VSS_DATA_DIR}/auto-calib/vggt/

  6. Create project directory and change ownership to UID 1000 and GID 1000:

    1. Create projects directory:

      mkdir -p ${VSS_APPS_DIR}/services/auto-calibration/projects

      Note

      Projects will be saved in: ${VSS_APPS_DIR}/services/auto-calibration/projects

    2. Change ownership:

      sudo chown -R 1000:1000 ${VSS_DATA_DIR}/auto-calib
sudo chown -R 1000:1000 ${VSS_APPS_DIR}/services/auto-calibration/projects

  7. Ensure you have access to NGC to pull the containers.

  8. Start both the microservice and UI servers:

    docker compose --profile "auto_calib" up -d

  9. Open your browser and navigate to:

    http://<HOST_IP>:<VSS_AUTO_CALIBRATION_UI_PORT>

    For example, with the default settings: http://<HOST_IP>:5000

  10. To stop the containers

    docker compose --profile "auto_calib" down

First Time Setup#

When you first access the UI, you’ll see the main interface with a stepper showing 6 workflow steps.

First Launch Screen

Interface Overview

The interface consists of:

  1. Header Bar

    • Application name and version

    • Theme toggle button (light/dark mode)

  2. Stepper Navigation

    • Visual progress indicator

    • Click on steps to navigate (after selecting a project)

    • Current step is highlighted

  3. Main Content Area

    • Step-specific content and controls

    • Forms, file uploads, and interactive tools

  4. Navigation Buttons

    • “Previous” button to go back

    • “Next” button to proceed

    • Disabled when requirements aren’t met

  5. Footer

    • Copyright information

  6. Notifications

    • Success/error messages appear in bottom-right corner

    • Auto-dismiss after 6 seconds

Quick Start Guide#

Follow these steps to perform your first calibration:

Step 0: Deploy the UI (If Not Already Running)

If you’re deploying via Docker Compose, follow the steps in the Deployment Steps (Docker Compose) section above. Once deployed, access the UI at http://<HOST_IP>:<VSS_AUTO_CALIBRATION_UI_PORT> (default: port 5000).

Step 1: Create a Project

  1. On the Project Setup page, enter a project name (e.g., warehouse_calibration)

  2. Click “Create” button

  3. Your new project appears in the list below

  4. Click “Select” on your project card

Project Setup

Step 2: Upload Files

Note

Uploaded or captured videos must satisfy tracklet-based video requirements (moving people, duration, 1920×1080, multi-camera sync). Calibration often fails late in the pipeline if these are not met.

  1. Click “Next” to go to Video Configuration

  2. Provide one or more time-synchronized camera inputs (which contain moving people/objects) using either file upload or RTSP capture (one camera for single-camera calibration, two or more for multi-camera; the UI blocks mixing an active file upload queue with RTSP, and file-uploaded clips must be removed before switching to RTSP).

    Option A: Upload video files

    • Click “Select Videos” button

    • Select video files named cam_00.mp4, cam_01.mp4, etc.

    • Reorder videos by dragging.

    • Click “Upload Videos” to upload the videos.

    Option B: RTSP capture (VIOS) (shown only when the Auto Calibration service exposes RTSP capture; VIOS is configured on the server side)

    • Finish or clear any pending Video Files selection or upload before starting RTSP; if the project already has clips from file upload, remove them under Video Files before using RTSP.

    • In the RTSP capture (VIOS) card, set Duration (seconds). The server requires at least 60 seconds.

    • Under Streams, enter all camera RTSP URLs for this project in one session (use Add stream for additional cameras; one stream for single-camera, two or more for multi-camera). Optionally set Camera name for each stream.

    • Click Start capture once with every stream listed—capture runs for all streams together. Do not add streams at different times or run separate captures for different subsets; staggered captures break time synchronization.

    • Wait for recording to progress (status chip and progress bar). You may click Stop early only after at least 60 seconds of active recording have elapsed.

    • When the session is finished (COMPLETED or CANCELLED), click Ingest to project to add the captured clips to the project’s video list. While capture or ingest is running, Video Files upload stays disabled until the RTSP pipeline completes.

    Note

    RTSP inputs must be time-synchronized: configure every stream, then run one Start capture for the full set. Re-capturing or ingesting additional streams later (instead of a single combined capture) will desync clips and is not supported for calibration. List each RTSP URL in order of overlapping field of view (FOV)—the same ordering used when uploading video files (first stream = first camera in the overlap chain).

    Note

    For VIOS pre-registered RTSP streams, use the source URL (for example, the NVStreamer URL if the stream originates from NVStreamer) rather than the VIOS-proxied URL.

  3. Upload layout image:

    • Click “Upload Layout” button

    • Select your PNG format layout/map image

    • Confirm upload success

Video Configuration

Step 3: Configure Parameters

  1. Click “Next” to go to Parameters

  2. Under Annotation target, choose where to draw ROIs and tripwires (all optional):

    Camera — per-camera annotations on a video frame

    • Select a camera from Select Camera

    • Use Drawing Tools on the video frame:

      • Draw ROI: click to add points (minimum 3), press F or right-click to finish; saved automatically

      • Draw Tripwire: click twice for start and end (red line); saved automatically

      • Tripwire Direction: click twice for a directional line (yellow, with arrow); saved automatically

    Global ROIs / tripwires — layout-map annotations (requires a layout image from Step 2)

    • Select Global ROIs / tripwires (disabled until layout.png is uploaded in Video Configuration)

    • Draw on the floor-plan / layout image using the same Drawing Tools (Draw ROI, Draw Tripwire, Tripwire Direction)

    • Global shapes are stored separately from per-camera annotations and appear under Current Annotations as Global (layout map)

    For both targets you can use Show/Hide to toggle overlays and Reset to clear annotations for the active target.

  3. Export image-mode JSON (optional, right panel):

    • Click Export image-mode JSON to download <project_name>_image_mode_exported.json

    • Exports ROIs and tripwires in pixel coordinates (calibrationType: image); bundle adjustment is not required for this export

  4. Add focal lengths (optional):

    • Enter comma-separated values (one per camera)

    • Click “Save Focal Length”

  5. Configure parameters (optional):

    • On the Parameters step, click the Click the icon to customize calibration parameters icon in the top-right (visible only on this step)

    • Upload a pre-configured settings file, or manually adjust calibration parameters in the dialog

    • Use Download to export the current configuration, Reset to Defaults to restore defaults, then Save Settings

    • Make all changes before starting calibration in Step 5: Execute; do not change settings while AMC is running

Parameters Configuration

Step 4: Create Alignment Data

  1. Click “Next” to go to Manual Alignment

  2. Choose one of two options:

    Option A: Upload Existing Alignment

    • Click “Upload alignment_data.json”

    • Select your JSON file

    • Wait for upload confirmation

    Option B: Create Alignment Interactively

    • Click “Open Alignment Tool”

    • Click the same physical point on Camera 0, Camera 1, and Layout (in order)

    • Repeat for at least 4 different points

    • Click “Save Alignment” when complete

Manual Alignment

Step 5: Run Calibration

  1. Click “Next” to go to Execute

  2. Review the requirements checklist

  3. Click “Verify Project” button

  4. Once verified, click “Start Calibration”

  5. Monitor the progress (status updates every 3 seconds)

  6. Wait for “Calibration completed successfully” message (AMC), or for AMC to finish far enough that optional VGGT can run (see below)

Execute Calibration

Step 5b: Run VGGT Calibration (Optional, Multi-Camera Only)

VGGT (Vision-Geometry Graph Transformer) is an optional model-based calibration path. It is available only when the backend has the VGGT model installed (see Deployment Steps above) and your project has two or more cameras—single-camera projects do not support VGGT.

  1. After Start Calibration (AMC) has run and produced on-disk outputs (typically after rectification), scroll to Calibration Control (VGGT) on the Execute step

  2. Confirm VGGT state is READY (AMC does not need to show COMPLETED; VGGT can remain runnable if AMC failed after rectification)

  3. Click Run VGGT Calibration and monitor progress

  4. When VGGT completes successfully, the project state becomes COMPLETED even if AMC failed—you can proceed to Results using the VGGT tab

Step 6: View Results

  1. Click “Next” to go to Results

  2. View the overlay image showing calibration results. Click “Download” to save overlay image

  3. Review camera parameters for each camera

  4. Export calibration data:

    • Click “Full Export AMC” for complete calibration data. It will load the json which contains the calibration data for all the cameras in the project, which you can edit according to your usecase.

    • Click “MV3DT ZIP AMC” for MV3DT-compatible format

Calibration Results

Lens Distortion Output (distortion.yaml)#

AMC can accept videos that contain lens distortion. When rectification is enabled (distortion model set to something other than pinhole in the rectification settings), the pipeline:

  1. Estimates per-camera distortion parameters (primarily k1)

  2. Rectifies video frames internally so that subsequent calibration (tracking, tracklet matching, bundle adjustment) runs on undistorted imagery

  3. Writes estimated parameters to disk for downstream use

Important for downstream applications

Rectified videos (rectified.mp4) are produced under each camera’s single-view output folder on the server, but they are not automatically substituted for your original camera feeds in downstream applications. Exported calibration JSON and MV3DT ZIP files describe cameras in the rectified (undistorted) image space.

To stay consistent with AMC calibration results, downstream applications must either:

  • Apply the saved distortion.yaml parameters to undistort live or recorded frames before using AMC projection matrices, or

  • Manually replace input videos with the per-camera rectified.mp4 files from the calibration output (not wired automatically by the microservice)

Output location

Per camera (example paths on the calibration server):

<project_output>/single_view_results/cam_00/distortion.yaml
<project_output>/single_view_results/cam_00/geocalib_distortion.yaml   # when GeoCalib estimates distortion
<project_output>/single_view_results/cam_00/rectified.mp4

``distortion.yaml`` format

This is the parameter file AMC rectification writes for application use. It contains the distortion coefficient in AMC/OpenCV pixel-centered convention (focal length treated as 1, origin at image center):

k1: -1.2345678e-06

``geocalib_distortion.yaml`` format (when distortion estimation runs)

GeoCalib writes a richer record that includes the distortion model name and both coordinate conventions:

model: simple_radial        # or radial, simple_divisional (must match rectification config)
k1: -1.2345678e-06          # AMC/OpenCV convention: radial factor = 1 + k1 * r_pixel^2
k1_geocalib: -4.5678901e-03 # GeoCalib internal normalized coordinates
focal_length: 1269.0        # focal length used for k1 conversion
source: geocalib
k2: ...                     # present only when model is radial

Distortion model

AMC rectification uses a Brown–Conrady radial model (inverse mapping for undistortion). For the default simple_radial model, only k1 is non-zero:

  • Undistorted normalized radius: r_u = r_d * (1 + k1 * r_d^2 + k2 * r_d^4 + k3 * r_d^6) (iterative inverse solve)

  • Pixel coordinates use the image center as the principal point; k1 in distortion.yaml is already scaled to full-resolution pixel units

Supported model names (set in rectification config, must match between GeoCalib and rectification):

  • pinhole — no distortion; rectification skipped

  • simple_radial — single k1 term (most common)

  • radial — k1 and k2 terms

  • simple_divisional — division model variant

Applying distortion correction downstream

Use k1 from distortion.yaml with OpenCV’s undistortion APIs. Example (Python, per frame):

import cv2
import numpy as np
import yaml

with open("distortion.yaml") as f:
    k1 = float(yaml.safe_load(f)["k1"])

h, w = frame.shape[:2]
camera_matrix = np.array([[1, 0, w / 2], [0, 1, h / 2], [0, 0, 1]], dtype=np.float64)
dist_coeffs = np.array([k1, 0, 0, 0], dtype=np.float64)
rectified = cv2.undistort(frame, camera_matrix, dist_coeffs)

Ensure the same image resolution (1920 × 1080) and that you undistort before projecting detections with AMC camera matrices. If rectification was disabled (pinhole / general.skip: true), no distortion.yaml is needed and raw video matches calibration space.

See also Custom Dataset (lens distortion guidance) and the AutoMagicCalib rectification_config.yaml in the source repository for tuning search ranges and model selection.

Settings#

Click the settings icon in the top-right corner to access calibration parameters. Note: The settings icon is only visible when you are on the Parameters step.

Settings Dialog

Additional Available Settings

  • Theme: Switch between light and dark themes

  • Version Information: View current application version

Note: Most settings are configured during deployment and cannot be changed from the UI.

Agent Skills walkthrough#

This section walks through the Auto Calibration user journeys using a coding agent (Claude Code, Codex, or NemoClaw) and the vss-generate-video-calibration Agent Skill. For each journey it documents the example prompt you give the agent and what the agent does in response.

Note

Install the skills into your coding agent first (see Installing Skills for the install flow). For how each harness loads skills, see Agent Harnesses.

The agent may pause for user input during the journey. Common prompts include NGC_CLI_API_KEY for NGC image pulls, HF_TOKEN when staging the VGGT model, VSS_APPS_DIR, VSS_DATA_DIR, HOST_IP, the Auto Calibration service ports, a VIOS URL for RTSP capture, local video paths, RTSP URLs, a layout image, alignment_data.json, a settings file, detector choice (resnet or transformer), and confirmation before starting long-running calibration or changing host permissions. If required files are missing, the agent directs you to the AMC UI step that supplies them and waits for confirmation before continuing.

Deploy Auto Calibration#

Example prompts:

  1. Start the Auto Calibration service.

  2. Deploy AMC so I can calibrate videos.

What the agent does:

  1. Loads the vss-generate-video-calibration skill and selects the deploy path.

  2. Detects the VSS repository path, VSS_APPS_DIR, VSS_DATA_DIR, HOST_IP, service ports, Docker/NVIDIA runtime, and NGC_CLI_API_KEY. It asks for any missing values and does not echo secret key values in its final response.

  3. Creates or updates the Auto Calibration .env and prepares the projects directory that the container writes to.

  4. Optionally stages VGGT refinement by asking for a Hugging Face token after you have accepted the model license.

  5. Starts the Auto Calibration backend and UI, verifies the /v1/ready endpoint, and returns the UI URL. It does not create a calibration project, upload videos, or run the bundled sample dataset unless you explicitly ask for a sample or calibration run.

Run the bundled sample dataset#

Example prompts:

  1. Calibrate the sample dataset.

  2. Verify the AMC install with the bundled sample dataset.

What the agent does:

  1. Loads the sample-dataset path and deploys AMC first if it is not already running.

  2. Obtains the bundled sample data, then uploads the sample videos, layout, alignment data, and ground truth files through the REST API.

  3. Verifies the project, starts calibration with the fixed sample detector configuration, polls until completion, and reports progress while the run is active.

  4. Returns evaluation metrics when ground truth is present, plus the AMC UI URL and output paths for the overlay image and project files.

Calibrate local video files#

Example prompts:

  1. Calibrate local video files at <path/to/files>.

  2. Calibrate videos at <path/to/files> with the default detector.

  3. Calibrate videos at <path/to/files> and run VGGT too.

What the agent does:

  1. Loads the videos path and deploys AMC first if it is not already running.

  2. Collects or auto-detects the video directory, layout image, alignment_data.json, optional settings file, optional ground-truth ZIP, optional focal lengths, and detector choice.

  3. If layout, alignment, or settings must be created interactively, directs you to the matching AMC UI step and waits until you confirm the changes are saved.

  4. Uploads the inputs through the REST API, verifies the project, summarizes the calibration plan, and asks for confirmation before starting calibration.

  5. Polls until completion, reports progress, surfaces errors with the relevant log tail, and returns the results URL and output paths.

Calibrate live RTSP streams#

Example prompts:

  1. Calibrate RTSP streams <rtsp-url-1>, <rtsp-url-2>, ...

  2. Calibrate live RTSP streams <rtsp-url-1>, <rtsp-url-2>, <rtsp-url-3>, <rtsp-url-4> using the transformer detector.

What the agent does:

  1. Loads the rtsp path and deploys AMC first if it is not already running.

  2. Collects the RTSP URLs, VIOS base URL, capture duration, layout image, optional settings file, optional alignment data, and detector choice.

  3. Verifies VIOS reachability, starts one synchronized capture session for all streams, ingests the captured clips into the AMC project, and prevents mixing file uploads with active RTSP capture state.

  4. If layout, alignment, or settings are missing, directs you to the AMC UI to provide them and waits for confirmation before calibration continues.

  5. Verifies the project, summarizes the plan, asks for confirmation, starts calibration, and reports progress until results are ready.

Review results and exports#

Example prompts:

  1. Show me the calibration results for project warehouse_calibration.

  2. Export calibration.json and the MV3DT ZIP for this project.

What the agent does:

  1. Queries project state, evaluation statistics, overlay images, and calibration logs for the requested project.

  2. Returns the AMC UI URL, overlay image path, project directory, and available exports.

  3. For MV3DT use cases, fetches the MV3DT ZIP and the exported calibration.json when requested.

Stop the service#

Example prompts:

  1. Stop the Auto Calibration service.

  2. Tear down AMC but keep my calibration projects.

What the agent does:

  1. Stops the Auto Calibration backend and UI containers.

  2. Preserves project state under ${VSS_APPS_DIR}/services/auto-calibration/projects/ unless you explicitly ask to remove it.

  3. Asks for confirmation before deleting project data, VGGT model files, or other host-mounted state.

Tips for Success#

Input Video

  • Follow tracklet-based video requirements at the top of this guide (resolution, duration, moving people, multi-camera sync).

  • Ensure videos are synchronized in time and uploaded in order of overlapping FOV.

  • If cameras have lens distortion, read Lens Distortion Output and plan how your application will undistort frames or use rectified.mp4.

Alignment Points

  • Choose points on the ground plane visible in all cameras

  • Select points at different depths and locations

  • Avoid points on moving objects

  • Use distinct features (corners, markings, etc.)

ROI and Tripwire Drawing

  • Draw ROIs to cover areas of interest

  • Place tripwires perpendicular to expected motion

  • Use tripwire directions to indicate motion direction

  • Test with different zoom levels for precision

Keyboard Shortcuts#

Parameters Step (Drawing)

  • F key: Finish current ROI

  • Esc key: Cancel current drawing

  • Scroll wheel: Zoom in/out on canvas

  • Click + Drag: Pan around zoomed canvas

Manual Alignment Step

  • Scroll wheel: Zoom in/out on alignment canvas

  • Click + Drag: Pan around zoomed canvas (when zoomed)

Next Steps#

Now that you’re familiar with the basics, explore: