AutoMagicCalib#

Overview#

Auto Magic Calib Microservice enables automatic calibration of multi-camera and single-camera systems through a simple, web-based, step-by-step workflow.

It is powered by AutoMagicCalib (AMC), which estimates camera intrinsics and extrinsics to generate accurate projection matrices and lens distortion parameters for 3D and multi-view applications.

Unlike checkerboard-based methods, AMC uses naturally moving objects in the scene. With NVIDIA DeepStream detection and tracking, it follows objects (e.g., people) and derives camera parameters from normal operational footage—no setup, no downtime, and supports calibration from archived videos.

The service supports both a geometry-based approach (AMC) using object trajectories and geometric relationships, and a model-based approach (VGGT) that leverages learned models for higher accuracy and robustness. VGGT is optional, available for multi-camera projects only (two or more cameras), and requires the VGGT model on the backend.

Auto Magic Calib microservice architecture#

Note

This document describes the Auto Magic Calib Microservice and UI. For standalone usage, detailed options, and advanced usage, see the AutoMagicCalib GitHub Repository.

Key Features#

Guided Workflow: 6-step process with progress tracking and real-time validation at each step
Project Management: Create and manage multiple projects; track states (INIT, READY, RUNNING, COMPLETED, ERROR); refresh, delete individual projects, or Delete all; data persistence
Flexible Video Input: Upload MP4 files or time-synchronized RTSP streams (via VIOS when configured on the server); one camera for single-camera calibration, two or more for multi-camera
Interactive Tools: Per-camera and global (layout-map) ROI drawing, tripwire configuration, manual alignment (camera-to-world), real-time preview; Export image-mode JSON for pixel-coordinate ROIs/tripwires without running calibration
Dual Calibration Methods: AMC (primary) and optional VGGT (Vision-Geometry Graph Transformer); VGGT can complete successfully even if AMC failed after rectification (e.g., during multi-view tracklet matching)
Export Options: Full export dialog (AMC or VGGT when both available), MV3DT-compatible ZIP (AMC and VGGT), per-camera parameter export, ROI/tripwire world-coordinate verification on BEV

The calibration process follows: Project Setup → Video Configuration → Parameters → Manual Alignment → Execute → Results. Each step validates inputs before allowing progression to the next.

System Requirements#

x86_64 system
OS Ubuntu 24.04
NVIDIA GPU with hardware encoder (NVENC)
NVIDIA driver 590
NVIDIA container toolkit
Docker (configured to run without sudo privilege)

Required for calibration

One or more camera video files (MP4 format) or time-synchronized RTSP streams: one for single-camera calibration, at least two for multi-camera; moving objects (e.g., people) for tracking; maintain order by overlapping field of view (FOV)
Layout/map image (PNG format)

Optional

Ground truth data (ZIP file) for calibration evaluation
Pre-existing alignment data (JSON file)
Focal length values for cameras
Config parameters for your dataset
VIOS server and VIOS_BASE_URL configuration for RTSP URLs input

Quick Start#

Deploy the UI and backend microservice using Docker Compose.

NGC Setup#

This step is needed to pull AutoMagicCalib docker image.

Visit NGC sign in page, enter your email address and click Next, or Create an Account
Choose your Organization/Team
Generate an API key following the instructions
Log in to the NGC docker registry:

docker login nvcr.io
Username: "$oauthtoken"
Password: "YOUR_NGC_API_KEY"

Project Setup#

Clone the repo to your local directory.

git clone https://github.com/NVIDIA-AI-IOT/auto-magic-calib.git
cd auto-magic-calib

Optional VIOS Setup for RTSP Capture#

RTSP support requires VIOS_BASE_URL. If a VIOS server is already reachable from the AMC host, verify it before setting VIOS_BASE_URL in compose/.env. Without VIOS, the rest of the calibration workflow can run, but RTSP support remains disabled.

If VIOS is not deployed yet, use the VIOS deployment assets from the VSS repository. Follow 1click_README.md in that directory. A typical development deployment command is:

sudo python3 oneclick_dc_deployment_for_dev.py --auto

Verify VIOS before enabling it:

curl http://<vios-host>:30888/vst/api/v1/sensor/list

Configure Environment Variables#

Edit the compose/.env file to set the required environment variables.

Variable	Required	Default	Description
`HOST_IP`	Yes	—	IP address of the host machine
`AUTO_MAGIC_CALIB_MS_PORT`	No	`8000`	Port for the microservice API
`AUTO_MAGIC_CALIB_UI_PORT`	No	`5000`	Port for the web UI
`PROJECT_DIR`	No	`../../projects`	Path to the projects directory (can be modified to absolute path to your projects directory)
`MODEL_DIR`	No	`../../models`	Path to the models directory (can be modified to absolute path to your models directory)
`VIOS_BASE_URL`	No	—	VIOS server URL for RTSP capture APIs

AUTO_MAGIC_CALIB_MS_PORT=8000
AUTO_MAGIC_CALIB_UI_PORT=5000
PROJECT_DIR=../../projects
MODEL_DIR=../../models
HOST_IP="<your_host_ip>"
# VIOS_BASE_URL=http://<VIOS_HOST_IP>:30888  # Uncomment and update this to support RTSP streams

HOST_IP must not be empty. If it is empty, the UI API URL renders as http://:<port>/v1.

Leave VIOS_BASE_URL unset to keep RTSP capture disabled.

Set Directory Permissions#

The projects and models directories must be owned by UID/GID 1000 for the containers to read/write properly.

chown 1000:1000 -R projects
chown 1000:1000 -R models

Launch Services#

Start all services using Docker Compose. Docker images will be pulled automatically on the first run.

cd compose
docker compose up -d

The microservice will be available at http://<HOST_IP>:<AUTO_MAGIC_CALIB_MS_PORT> (default port 8000) and the UI at http://<HOST_IP>:<AUTO_MAGIC_CALIB_UI_PORT> (default port 5000).

Note

To stop: docker compose down.

For VGGT, download the model from HuggingFace (e.g. VGGT-1B-Commercial) and place vggt_1B_commercial.pt in the models directory.

Workflow Steps#

Step 1: Project Setup — Create a project (name 3–50 characters), click “Create”, then “Select”. Project cards show state (INIT, READY, RUNNING, COMPLETED, ERROR), video count, and file status (GT, Layout, Alignment). Refresh, delete individual projects, or use Delete all (projects with calibration running or RTSP capture in progress are skipped).

Step 2: Video Configuration — Provide camera inputs using either file upload or RTSP URLs (VIOS; shown only when configured on the server). Minimum one video (MP4, 1920×1080 recommended); two or more for multi-camera. Order videos by overlapping FOV (cam_00, cam_01, …). The UI does not allow mixing an active file-upload queue with RTSP—remove file-uploaded clips before switching to RTSP, and vice versa.

Option A: Upload video files — Select, reorder by drag-and-drop, then upload.

Option B: RTSP URLs (VIOS is Configured) — Set duration (minimum 60 seconds). Enter all RTSP URLs under Streams before a single Start capture (one session for every camera). Do not add streams later or run staggered captures—time synchronization requires one combined capture. Optionally Stop early after 60 seconds of recording; when COMPLETED or CANCELLED, click Ingest to project. For VIOS pre-registered streams, use the source URL (e.g., NVStreamer URL), not the VIOS-proxied URL.

Also upload a required layout image (PNG) and optional ground truth (ZIP) or alignment (JSON).

Step 3: Parameters — Under Annotation target, choose Camera (per-stream ROIs/tripwires on video frames) or Global ROIs / tripwires (draw on the layout map; requires layout from Step 2). Use Draw ROI (polygon, min 3 points; press F or right-click to finish), Draw Tripwire, and Tripwire Direction; Show/Hide and Reset per target. Annotations auto-save.

Optional: Export image-mode JSON (pixel coordinates, calibrationType: image, no calibration required); focal lengths (comma-separated, one per camera); settings icon on this step only—upload config, adjust manually, download, Reset to Defaults, Save Settings (do not change settings while AMC is running).

Step 4: Manual Alignment — Upload alignment_data.json (Option A) or open the alignment tool (Option B).

Multi-camera (2+ videos): Click the same physical point on Camera 0, Camera 1, and the layout map for each of at least 4 point sets (3 clicks per set); then “Save Alignment”.

Single-camera (1 video): Click the same physical point on the camera and layout map for each of at least 4 point sets (2 clicks per set); then “Save Alignment”.

If you want to create the alignment data interactively, you can use the alignment tool.

Step 5: Execute — Ensure requirements are met (videos, layout, alignment with 4+ point sets), click “Verify Project”, then “Start Calibration”. Monitor AMC progress (status updates every ~3 seconds; live logs). On failure, use “Relaunch Calibration” or “Reset Project”.

VGGT (optional, multi-camera only): After AMC has produced on-disk outputs (typically after rectification), Run VGGT Calibration when VGGT state is READY—AMC does not need to show COMPLETED. A successful VGGT run sets project state to COMPLETED even if AMC failed (e.g., at tracklet matching). VGGT is not offered for single-camera projects.

Step 6: Results — Available when AMC or VGGT completed successfully. View overlay image (AMC/VGGT tabs), evaluation metrics (L2 distance if ground truth uploaded), and per-camera parameters (YAML; AMC/VGGT tabs).

Export via Full Export (dialog to choose AMC or VGGT when both exist; optional metadata and JSON editor), MV3DT ZIP AMC, or MV3DT ZIP VGGT (when available). Use Delete Results to re-run calibration.

Click Show ROI & Tripwire Verification
Choose Camera or Global ROIs / tripwires in the annotation target toggle
Camera target: pick a stream from Select Camera; review the left rectified frame and the right world map; use AMC / VGGT tabs on the world map as needed
Global target: left panel shows global shapes on layout.png (pixel coordinates); right panel shows their world-map projection in purple
Under the camera view, use checkboxes labeled Global ROI: / Global tripwire: to include or exclude that camera in each ROIs/Tripwires sensors list (only global items projected to that camera are listed)
Use world-map zoom controls for detail; click Close when finished

Custom Dataset#

Prepare:

Input videos — One or more camera video files (or time-synchronized RTSP URLs); upload/capture order defines camera pairing and FOV overlap chain; consecutive cameras should have overlapping FOV for multi-camera setups.
Floor map — Layout/map image of the surveillance area (PNG).
Ground truth (optional) — For evaluation, a ZIP containing calibration.json and ground_truth.json.

calibration.json — Sensors array; each has id, intrinsicMatrix (3x3), extrinsicMatrix (3x4), cameraMatrix (3x4), attributes (e.g. frameWidth, frameHeight). Matrices follow OpenCV-style definitions.

ground_truth.json — Frame index as key; each value is an array of objects with object id, object type, object name, 3d location [x,y,z], and 2d bounding box visible (camera id → [x_min, y_min, x_max, y_max]).

Guidelines for Input Videos to Achieve Optimal Calibration Results#

To ensure the most accurate camera calibration, careful consideration should be given to how the input videos are captured. The following points detail how to maximize the quality of the calibration outcome.

1. Minimizing Lens Distortion

The current calibration methodology performs best when input videos are “linear,” meaning they exhibit no lens distortion. While the tool can handle minor distortion, optimal results are achieved when lens distortion is zero.

No lens distortion - linear video — Example of linear video with minimal lens distortion#

2. Maximizing Camera Overlap

Accurate calibration requires a significant degree of overlap between the fields of view of the different cameras. It is essential to maximize the overlap between cameras as much as possible.

3. Leveraging Unique Scene Features

The presence of diverse and unique objects in the input videos contributes significantly to calibration accuracy. The automatic calibration tool specifically utilizes people moving within the field of view, so videos with many moving people are ideal. The trajectories of these moving subjects should cover the Field of View (FOV) as broadly as possible.

Additionally, large, unique objects can enhance accuracy. For instance, in a setting like a warehouse with multiple cameras, views can become challenging due to repetitive elements (e.g., similar racks). In such environments, large, distinct objects, such as forklifts, are beneficial for better calibration accuracy.

Forklift scene for calibration — Large distinct objects (e.g., forklifts) in warehouse views aid calibration#

Calibration Output#

After calibration, the UI provides overlay images (AMC and/or VGGT tabs), evaluation metrics (e.g. L2 distance average, std dev, min/max when ground truth is provided), and per-camera parameters (projection matrix, intrinsics/extrinsics in YAML).

With ground truth: Compare L2 distance statistics and overlay trajectories (colored reconstructed paths vs. white ground truth lines) between AMC and VGGT to select the better result.

Without ground truth: Compare overlay images qualitatively—camera positions, object trajectories, and FOV boundaries against the floor map.

Export options include:

Full Export — Complete calibration JSON with ROI/tripwire world coordinates; choose AMC or VGGT in the dialog when both completed ({project_name}_exported.json or {project_name}_exported_vggt.json)
Export image-mode JSON (Parameters step) — ROIs/tripwires in pixel coordinates without running bundle adjustment
MV3DT ZIP AMC / MV3DT ZIP VGGT — MV3DT-compatible archives for verification

Integration with MV3DT#

Export calibration in MV3DT-compatible form from the Results step: use MV3DT ZIP AMC or MV3DT ZIP VGGT. The generated ZIP can be used with Multi-View 3D Tracking (MV3DT) verification tools.

MV3DT visualization using AMC calibration — MV3DT real-time multi-view tracking with AMC-generated calibration parameters#

Troubleshooting#

Cannot access UI — Confirm backend and UI are running (e.g. docker ps). Check URL/port, firewall, and connectivity (e.g. ping, telnet <ip> <port>).

Port already in use — Change AUTO_MAGIC_CALIB_MS_PORT and AUTO_MAGIC_CALIB_UI_PORT in compose/.env and restart; or stop the process using the port (e.g. sudo lsof -i :5000).

API_URL_NOT_PROVIDED — Set HOST_IP in compose/.env and restart: docker compose down then docker compose up.

Verification fails — Ensure videos (min 1), layout, and alignment (4+ point sets). Check alignment JSON and video integrity; re-upload if needed.

Calibration fails or takes too long — Check input files and alignment (ground plane, visible in all views/cameras). Typical AMC run 5–15 minutes; VGGT typically 2–3 minutes. If AMC fails after rectification, try Run VGGT Calibration when VGGT state is READY. Reset project and retry if needed.

RTSP URLs as Input issues — Ensure VIOS is configured; use source URLs (not VIOS-proxied) for pre-registered streams; configure all streams before one Start capture; minimum 60 s duration; remove file-uploaded clips before RTSP and vice versa.

VGGT not available — Install VGGT model on backend; VGGT requires multi-camera (2+ videos). Single-camera projects do not support VGGT.

Export or ROI verification issues — Ensure at least one calibration path (AMC or VGGT) completed. Use Show ROI & Tripwire Verification after calibration; for full world-coordinate export, use Full Export and choose AMC or VGGT as needed.

Additional Resources#

GitHub Repository: AutoMagicCalib for standalone AMC and sample data.
VGGT model: Optional; download from HuggingFace (e.g. VGGT-1B-Commercial) and place in the models directory for VGGT refinement.