Auto-Label#

Annotating an image or video dataset by hand is tedious and time-consuming, and this is especially true for segmentation and reasoning tasks. Drawing an accurate polygon around an object can take 10 times longer than drawing a bounding box, and authoring question-answer pairs for a video can take longer still. The Auto-Label service of TAO Data Services reduces this effort by generating annotations automatically from existing images, videos, captions, and bounding boxes.

The Auto-Label service groups several pipelines under a single auto_label generate command. You select a pipeline through the autolabel_type field in the specification file, and each pipeline reads its own configuration block. The service supports the following pipelines:

  • Mask Auto Label (MAL) generates instance segmentation masks from groundtruth bounding boxes.

  • Grounding DINO generates bounding box annotations from category names or noun chunks through iterative open-vocabulary detection.

  • 2D Grounding uses a vision-language model to extract referring expressions from images and ground them to bounding boxes.

  • Video Reasoning Annotation uses vision-language and language models to produce reasoning-style question-answer annotations from videos.

Tip

Most pipelines also ship as a TAO skill that an agent runs for you, which is the quickest way to get started. Each pipeline page opens with a “Quickstart with a TAO Skill” section; the configuration tables and the auto_label generate command-line path follow for when you need finer control.

Data Input for Auto-Label#

Each pipeline accepts a different input modality. The image-mask and open-vocabulary detection pipelines expect the groundtruth annotations for a directory of images in a COCO-format JSON file. The 2D Grounding pipelines accept image directories with captions or KITTI-format labels, and the Video Reasoning Annotation pipeline accepts a directory of videos or a list of JSONL files. Refer to each pipeline page for the exact input layout that it requires.

Running the Auto-Label Tool#

The Auto-Label service exposes a single task:

  • generate: Runs the pipeline selected by autolabel_type and writes the generated annotations to results_dir.

To run a pipeline, pass an experiment specification file to the generate task:

auto_label generate -e /path/to/spec.yaml results_dir=/path/to/results

Required arguments:

  • -e: Path to the experiment specification file.

  • results_dir: Directory in which to write the generated annotations.

Refer to each pipeline page for the full configuration reference and a sample specification file.