Mask Auto Label

Mask Auto Label (MAL) generates instance segmentation masks for a dataset that already has groundtruth bounding boxes. Because drawing an accurate polygon around an object can take 10 times longer than drawing a bounding box, MAL lets you label only the bounding boxes and then derive the masks automatically. MAL is a transformer-based, class-agnostic mask auto-labeler, so it generates masks for object categories that it has not seen during training.

Quickstart with a TAO Skill

The quickest way to run mask auto-labeling is through the models/tao-train-mask-auto-label skill (Minimal Annotation Learning, weakly supervised segmentation from point or box annotations), which an agent runs for you. For example:

“Run the models/tao-train-mask-auto-label skill on s3://my-bucket/raw-images/ using bounding boxes from boxes.json . Write masks to s3://my-bucket/masks/ .”

Refer to models/tao-train-mask-auto-label/SKILL.md for the supported actions and specification keys. The rest of this page describes the configuration and the command-line path for when you need finer control.

Data Input for Mask Auto Label

MAL expects the groundtruth annotations for a directory of images in a COCO-format JSON file. Each annotation must contain a bounding box, and MAL generates a segmentation mask for each one.

Configuring the Specification File for Mask Auto Label

You select MAL by setting autolabel_type to mal and populating the mal configuration block. The following table describes the top-level fields of the mal block.

Parameter	Datatype	Description
dataset	collection	Dataset configuration, including the image directory and the COCO-format annotation file
model	collection	Model architecture configuration for the mask auto-labeler
train	collection	Training configuration used when fine-tuning the mask auto-labeler
inference	collection	Inference configuration used when generating masks
evaluate	collection	Evaluation configuration used to measure mask quality
checkpoint	string	Path to the mask auto-labeler model checkpoint
results_dir	string	Directory in which to write the generated masks

For the full set of dataset, model, training, inference, and evaluation parameters, refer to the Mask Auto Labeler configuration reference.

Running Mask Auto Label

MAL runs through the generate task. The following example generates masks for a COCO-format dataset:

auto_label generate \
    -e /path/to/box2seg.yaml \
    results_dir=/path/to/results

The following example shows a complete specification file:

results_dir: /path/to/results
gpu_ids: [0]
batch_size: 4
num_workers: 8
autolabel_type: "mal"

mal:
  checkpoint: /path/to/mask_auto_label.pth
  inference:
    ann_path: /path/to/annotations.json   # COCO-format groundtruth boxes
    img_dir: /path/to/images
    label_dump_path: /path/to/results/masks.json

MAL reads the bounding boxes from mal.inference.ann_path, generates one mask per box, and writes the mask annotations to mal.inference.label_dump_path.