Gap Analysis#

The gap_analysis command is a console command that ships in the TAO Data Services (tao_ds) container. It performs Root Cause and Corrective Action gap analysis: it compares the predictions of a key performance indicator (KPI) run against ground truth and emits the specific samples that the model gets wrong. With that list in hand, you can target data curation at the actual failures instead of retraining blindly.

Gap analysis provides three subtasks, one for each data domain:

object_detection: Compares object-detection predictions against ground-truth boxes by intersection over union (IoU) overlap. This subtask is new in TAO 7.2.0.
vcn_aoi: Analyzes TAO Visual ChangeNet Classify KPI runs for Automated Optical Inspection (AOI), such as printed circuit board inspection.
vlm_bcq: Analyzes vision-language model (VLM) binary classification questions, which are yes-or-no questions asked over video. This subtask writes no output files when it finds no gaps.

All three subtasks feed TAO Data Enhanced Fine-tuning (DEFT) loops. Refer to Used by TAO DEFT Workflows for the details of that integration.

Ask your agent to run the subtask you need. The following are example prompts, one per subtask:

object_detection: “Run object detection gap analysis on my KITTI KPI set. Ground truth labels are in /workspace/kpi/labels, inference labels are in /workspace/runs/iter1/inference/labels, and images are in /workspace/kpi/images. Tag the run iter1, use an IoU threshold of 0.5, and write the results to /workspace/runs/iter1/gaps.”
vcn_aoi: “Run Visual ChangeNet AOI gap analysis on the inference results in /workspace/vcn/inference, using the train configuration at /workspace/vcn/train.yaml. Write the results to /workspace/vcn/gaps.”
vlm_bcq: “Run vision-language model binary classification gap analysis on the predictions at /workspace/vlm/predictions.json, with videos under /workspace/vlm/videos. Write the results to /workspace/vlm/gaps.”

If you do not use an agent, run the command directly inside the tao_ds container. Every subtask takes the same form:

gap_analysis <subtask> -e /path/to/spec.yaml [key=value ...]

The -e flag, also spelled --experiment_spec_file, is the only subtask argument, and it is mandatory for all three subtasks. If you omit it, TAO raises a ValueError whose message is The subtask `object_detection` requires the following argument: -e/--experiment_spec_file. If you point it at a path that does not exist, TAO raises a FileNotFoundError.

There is no -r or --results_dir flag. To change the output directory from the command line, either edit the experiment specification file or pass the Hydra override results_dir=/path. Hydra overrides use bare key=value syntax appended after the -e argument. A command-line override takes precedence over the experiment specification file, which in turn takes precedence over the built-in default.

Note

The gap_analysis command is absent from the default_specs supported module list. That list contains exactly analytics, annotations, augmentation, auto_label, and image. The argument parser accepts gap_analysis default_specs, but the command always fails with a Module 'gap_analysis' is not supported error. Start instead from the specification files that ship in the container at nvidia_tao_ds/rcca/gap_analysis/experiment_specs/. This page reproduces them. In each shipped specification file, a value of ??? marks a required field. Fill in a value for every such field before you run the subtask.

Each subtask is explained in detail in the following sections.

Object Detection Gap Analysis#

The object_detection subtask compares ground truth against predicted detections image by image and class by class. It emits every false positive and every false negative box, the per-image per-class precision, recall, and average precision at an IoU of 0.5 (AP50), and the set of weak images that fail the per-class metric gates you configure.

Data Input for Object Detection Gap Analysis#

The object_detection subtask reads three inputs: a directory of images, a ground-truth annotation source, and an inference annotation source. Getting these three inputs to agree is the part of the configuration that most often goes wrong, so this section spells out the rules in detail.

The images_dir value defines the complete image universe. TAO walks that directory recursively and collects every file whose filename extension is .jpg, .jpeg, .png, .bmp, .tif, or .tiff. TAO matches the extension without regard to letter case, and it sorts the results so that a run is deterministic. Images that have neither ground-truth nor prediction annotations still belong to the image universe. They simply produce no output rows. If images_dir is not a directory, TAO raises a FileNotFoundError. If the directory contains no supported images, TAO raises a ValueError.

TAO never infers input_format. You must declare either kitti or coco explicitly, and TAO applies that one format to both the ground-truth source and the prediction source. You therefore cannot pair KITTI ground truth with COCO predictions.

Note

The gap_analysis object_detection subtask requires the lowercase spellings kitti and coco, while the analytics kpi_analyze task in the same container requires the uppercase spellings KITTI and COCO. Both spellings are correct for their own task, so match the spelling to the command you are running.

TAO parses the two annotation sources with different loaders by design. It reads ground_truth_ann_path with the unscored loader and inference_ann_path with the scored loader. TAO therefore ignores a 16th confidence column in a ground-truth KITTI file and never reads it as a score.

For the KITTI format, both ground_truth_ann_path and inference_ann_path are directories of per-image .txt label files. The file stem is the join key, so 000042.txt matches 000042.png. A directory that contains no .txt files raises a FileNotFoundError. Each line holds the 15 standard KITTI fields followed by an optional 16th confidence field:

class truncated occluded alpha x1 y1 x2 y2 h w l tx ty tz ry [score]

Fields 5 through 8 are the corner-format box x1 y1 x2 y2. TAO skips lines that have fewer than nine fields, skips rows whose class is DontCare (matched without regard to letter case), and skips lines whose coordinates do not parse as floating-point numbers. Only the scored loader reads the 16th field, so the following behavior applies to inference_ann_path alone: if the 16th field is present but does not parse as a floating-point number, TAO logs a warning and treats the box as unscored.

For the COCO format, ground_truth_ann_path and inference_ann_path are each a single JSON file that holds a COCO object. TAO raises a ValueError if the file is not valid JSON, if the top level is not an object, or if either images or annotations is missing. Because the top level must be an object, TAO rejects a bare COCO results array. TAO reads categories leniently, but in practice you must supply it: TAO silently drops any annotation whose category_id resolves to no name. TAO takes the image path key from the first non-empty value of file_name, filepath, or file_path, in that order. The bbox field is [x, y, w, h] and must have exactly four entries. A prediction score is an optional per-annotation score field. A non-numeric value logs a warning and the box becomes unscored.

TAO resolves each image from the image universe against the annotation maps in a fixed order: the exact full path first, then the file basename, then the file stem, and finally the image path taken relative to images_dir. To avoid merging unrelated images, TAO removes from the basename fallback any basename that more than one distinct COCO image path shares. Those images still resolve by full path or by relative path. In practice, make your COCO file_name values either absolute or relative to images_dir.

Unscored predictions are legal. A prediction that carries no score bypasses conf_threshold entirely, so TAO always keeps it. In the AP50 computation, TAO counts such a prediction as a confidence of 1.0. One consequence deserves planning: during matching TAO treats an absent score as 0.0 for ordering purposes, so TAO matches unscored predictions last, and they get last pick of the ground-truth boxes.

TAO applies class_mapping to ground-truth labels and prediction labels alike, immediately after loading. Labels that are absent from the mapping pass through unchanged. The typical use is folding model label synonyms onto your KPI class names before matching.

If your inference step already applied its own confidence threshold when it wrote the labels, the conf_threshold value here composes with that gate rather than replacing it.

Creating an Experiment Specification File for the Object Detection Subtask#

The container ships a template specification file for the object_detection subtask at nvidia_tao_ds/rcca/gap_analysis/experiment_specs/object_detection.yaml. Copy it and fill it in.

ground_truth_ann_path: ???
inference_ann_path: ???
images_dir: ???
results_dir: ???
kpi: ???
input_format: ???
iou_threshold: 0.5
conf_threshold: 0.0
min_area: 0
class_mapping: {}
# Per-class thresholds: {class_name: {recall: float, precision: float, ap50: float}}
# Example: {car: {recall: 0.9, precision: 0.8, ap50: 0.5}, pedestrian: {recall: 0.85}}
weak_thresholds: {}
default_recall_threshold: 0.5
default_precision_threshold: 0.0
default_ap50_threshold: 0.5

The following filled-in example comes from a DEFT loop. It analyzes a KITTI KPI set, tags the run iter1, and gates three classes on AP50 alone.

ground_truth_ann_path: /workspace/kpi/labels
inference_ann_path: /workspace/runs/iter1/inference/labels
images_dir: /workspace/kpi/images
results_dir: /workspace/runs/iter1/gaps
kpi: iter1
input_format: kitti
iou_threshold: 0.5
conf_threshold: 0.0
min_area: 0
weak_thresholds:
  car: {ap50: 0.99}
  bicycle: {ap50: 0.7}
  person: {ap50: 0.7}
default_ap50_threshold: 0.0
default_recall_threshold: 0.0
default_precision_threshold: 0.0
class_mapping: {}

This example sets all three default_*_threshold values to 0.0 on purpose. The defaults gate every class that is not listed in weak_thresholds, which marks images weak on classes the run never targeted. That inflates both the gap set and any mining budget you derive from it. Setting a default to 0.0 turns that gate off, so only the three listed classes can make an image weak.

The following table describes the configurable parameters.

Parameter	Datatype	Default	Description	Supported Values
`ground_truth_ann_path`	string		The path to the ground-truth annotation source: a KITTI label directory or a COCO JSON file. This field is required.	Any valid path
`inference_ann_path`	string		The path to the inference annotation source: a KITTI label directory or a COCO JSON file. Boxes can carry an optional per-box confidence score. This field is required.	Any valid path
`images_dir`	string		The root directory of images. This directory establishes the full image universe, including images that have no ground-truth or prediction annotations. This field is required.	Any valid path
`results_dir`	string		The output directory for `box_gaps.parquet`, `image_metrics.parquet`, `weak_images.parquet`, and `gap_report.json`. This field is required.	Any valid path
`kpi`	string		The KPI identifier tag that TAO writes to every output row. This field is required.	Any string
`input_format`	string		The annotation format. You must declare this value explicitly; TAO never infers it from the path.	kitti, coco
`iou_threshold`	float	0.5	The IoU threshold at or above which TAO accepts a prediction as a true positive.	>=0.0, <=1.0
`conf_threshold`	float	0.0	The minimum prediction confidence. TAO drops scored predictions below this value before matching. This value has no effect on predictions that carry no score.	>=0.0, <=1.0
`min_area`	int	0	The minimum box area in pixels, computed as width multiplied by height. TAO discards boxes strictly below this area.	>=0
`class_mapping`	dict	{}	A mapping from raw annotation label strings to canonical class names. TAO keeps labels that are absent from the mapping unchanged.
`weak_thresholds`	dict	{}	Per-class metric thresholds in the form `{class_name: {recall: float, precision: float, ap50: float}}`. Any omitted metric key falls back to the corresponding `default_*_threshold` value.
`default_recall_threshold`	float	0.5	The fallback recall threshold for classes that are not listed in `weak_thresholds`.	>=0.0, <=1.0
`default_precision_threshold`	float	0.0	The fallback precision threshold. Set this value to 0.0 to disable precision-based weak-image selection.	>=0.0, <=1.0
`default_ap50_threshold`	float	0.5	The fallback AP50 threshold. Set this value to 0.0 to disable AP50-based weak-image selection.	>=0.0, <=1.0

Running the Object Detection Subtask#

Ask your agent to run the object_detection subtask. Refer to the example prompt at the top of this page. To run the subtask yourself, use the tao_ds container:

gap_analysis object_detection -e /path/to/object_detection.yaml

You can override individual fields on the command line using Hydra syntax, for example:

gap_analysis object_detection -e /path/to/object_detection.yaml \
    results_dir=/workspace/runs/iter2/gaps conf_threshold=0.3

Object Detection Outputs#

TAO creates results_dir when it does not already exist, and writes all four artifacts into it on every successful run. TAO writes the files even when it finds no gaps and marks no images weak, so downstream consumers can rely on the files existing with the correct schemas.

Before reading the schemas, it helps to know how TAO derives the numbers in them. Matching is greedy and runs per class. TAO sorts the predictions by descending confidence, and each prediction in turn claims the highest-IoU unmatched ground-truth box of the same class. TAO accepts a match only when the IoU is greater than or equal to iou_threshold, and it matches each ground-truth box at most once. Ties break deterministically on the original annotation order.

The per-image per-class metrics follow from that matching. The precision value is NaN when there are no true positives and no false positives, and the recall value is NaN when there are no true positives and no false negatives. TAO computes ap50 per image and per class with torchmetrics at an IoU of 0.5 over a 101-point recall grid. TAO deliberately does not apply conf_threshold to AP50, because the metric sweeps confidence internally to build the full precision-recall curve. TAO does apply min_area to AP50. The ap50 value is NaN when the class has no ground truth, and 0.0 when the class has ground truth but no predictions. A class that appears only in the predictions still produces a metrics row and false-positive gap rows.

TAO marks an image weak if any class trips any of the recall, precision, or AP50 gates. Each gate uses the per-class value from weak_thresholds when that value is present, and the matching default_*_threshold value otherwise. The merge happens key by key, so you can override only recall for one class and still inherit the other two defaults. Setting a threshold to 0.0 disables that gate, and a NaN metric always passes.

Warning

TAO drops boxes at two different points, and only one of them is visible in the logs. At load time, TAO drops boxes that are non-finite or that have a non-positive width or height, and logs a warning that names the box and the image. At match time, TAO drops boxes whose area is strictly below min_area, and that drop is silent. The min_area value applies to ground truth and predictions alike.

The box_gaps.parquet file holds one row per unmatched box.

Column	Datatype	Description
`kpi`	string	The `kpi` value from the experiment specification file.
`image_id`	string	The image filename stem.
`filepath`	string	The image path from the `images_dir` walk.
`class`	string	The class name after TAO applies `class_mapping`.
`gap_type`	string	`FP` for a false positive or `FN` for a false negative.
`bbox`	list of float	The box in `[xmin, ymin, xmax, ymax]` corner format.
`confidence`	float	The prediction score for a false positive. This value is `NaN` for a false negative, and also `NaN` when the false positive comes from an unscored source.
`best_iou`	float	For a false positive, the highest IoU against any same-class ground-truth box that survives the area filter, or 0.0 when there is none. For a false negative, the highest IoU against any same-class prediction that survives the area and confidence filters, or 0.0 when there is none.
`gt_label_path`	string	The `ground_truth_ann_path` value from the experiment specification file.
`pred_label_path`	string	The `inference_ann_path` value from the experiment specification file.

The image_metrics.parquet file holds one row per combination of image and class that is present in the ground truth or in the predictions.

Column	Datatype	Description
`kpi`	string	The `kpi` value from the experiment specification file.
`image_id`	string	The image filename stem.
`filepath`	string	The image path from the `images_dir` walk.
`class`	string	The class name after TAO applies `class_mapping`.
`tp`	int	The number of true positives for this image and class.
`fp`	int	The number of false positives for this image and class.
`fn`	int	The number of false negatives for this image and class.
`precision`	float	The precision for this image and class. This value is `NaN` when there are no true positives and no false positives.
`recall`	float	The recall for this image and class. This value is `NaN` when there are no true positives and no false negatives.
`ap50`	float	The average precision at IoU 0.5 for this image and class. This value is `NaN` when the class has no ground truth, and 0.0 when it has ground truth but no predictions.

The weak_images.parquet file holds one row per weak image. The weak_recall, weak_precision, and weak_ap50 columns are three parallel boolean lists, each aligned index by index with weak_classes.

Column	Datatype	Description
`kpi`	string	The `kpi` value from the experiment specification file.
`image_id`	string	The image filename stem.
`filepath`	string	The image path from the `images_dir` walk.
`weak_classes`	list of string	The classes that trip at least one metric gate on this image.
`weak_recall`	list of bool	Whether the recall gate fails, aligned index by index with `weak_classes`.
`weak_precision`	list of bool	Whether the precision gate fails, aligned index by index with `weak_classes`.
`weak_ap50`	list of bool	Whether the AP50 gate fails, aligned index by index with `weak_classes`.

The gap_report.json file is a JSON summary of the run. It holds the kpi tag, a counts_by_type object with the false-positive and false-negative totals, a counts_by_class object that breaks the same counts down by class, and a settings echo of iou_threshold, conf_threshold, and min_area. The file has the following shape:

{
  "kpi": "iter1",
  "counts_by_type": {
    "FP": 128,
    "FN": 342
  },
  "counts_by_class": {
    "car": {
      "FP": 96,
      "FN": 210
    },
    "person": {
      "FN": 132
    },
    "bicycle": {
      "FP": 32
    }
  },
  "settings": {
    "iou_threshold": 0.5,
    "conf_threshold": 0.0,
    "min_area": 0
  }
}

Caution

The count keys are data dependent. The counts_by_type object is empty when the run finds no gaps, and an FP or FN key appears only when that count is non-zero. Read the counts defensively rather than assuming that both keys exist. The report also does not carry a weak-image count. To get that number, read the row count of weak_images.parquet.

Visual ChangeNet AOI Gap Analysis#

The vcn_aoi subtask ranks the samples of a TAO Visual ChangeNet Classify AOI KPI run by how badly the model handles them, and it emits the weakest samples for each ground-truth label. Visual ChangeNet Classify is the TAO printed circuit board inspection model, and the weakest samples it produces are the images that a DEFT loop targets first.

Data Input for Visual ChangeNet AOI Gap Analysis#

The vcn_aoi subtask reads two inputs: the inference results of a Visual ChangeNet Classify KPI run, and the train configuration file of the model that produced them.

The inference_results_dir directory must contain a file named exactly inference.csv. TAO reads <inference_results_dir>/inference.csv and does not search subdirectories. When that file is absent, TAO raises a FileNotFoundError that names the expected path.

The CSV file must provide the label, siamese_score, input_path, and object_name columns. TAO strips surrounding whitespace from label and compares it without regard to letter case. Any value other than PASS counts as NO_PASS. A CSV file that contains only a header row raises a ValueError when TAO computes the threshold automatically.

The train_config field points to the Visual ChangeNet train configuration YAML file. TAO reads exactly two values from that file: dataset.classify.input_map, which lists the lighting conditions, and dataset.classify.image_ext, which gives the filename extension. TAO reads both keys directly, so a missing file raises a FileNotFoundError and a missing key raises a KeyError. The image_ext value must include the leading dot, for example .png.

The kpi_media_path field is the root directory that TAO prepends to the relative image paths in the CSV file. When you leave it empty, TAO uses the input_path values exactly as they appear in the CSV file.

Creating an Experiment Specification File for the Visual ChangeNet AOI Subtask#

The container ships a template specification file for the vcn_aoi subtask at nvidia_tao_ds/rcca/gap_analysis/experiment_specs/vcn_aoi.yaml. Copy it and fill it in.

inference_results_dir: ???
train_config: ???
kpi_media_path: ""
threshold: -1.0
min_recall: 1.0
top_k_per_label: 50
results_dir: ???

The following table describes the configurable parameters.

Parameter	Datatype	Default	Description	Supported Values
`inference_results_dir`	string		The directory that contains the TAO Visual ChangeNet `inference.csv` file. TAO reads `<inference_results_dir>/inference.csv` only; it does not search subdirectories. This field is required.	Any valid path
`train_config`	string		The path to the Visual ChangeNet train configuration YAML file. TAO reads `dataset.classify.input_map` and `dataset.classify.image_ext` from this file. This field is required.	Any valid path
`results_dir`	string		The output directory for the gap-analysis results. This field is required.	Any valid path
`kpi_media_path`	string	“”	The root directory that TAO prepends to the image paths in the CSV file. When this value is empty, TAO uses the `input_path` values exactly as they appear.	Any valid path
`threshold`	float	-1.0	The classification threshold. TAO predicts `NO_PASS` when `siamese_score` is strictly greater than this value. A negative value triggers threshold auto-computation.
`min_recall`	float	1.0	The minimum `NO_PASS` recall that a candidate threshold must achieve during auto-computation.	>=0.0, <=1.0
`top_k_per_label`	int	50	The maximum number of weakest samples to keep for each ground-truth label. This value acts as a per-label augmentation budget.	>=1

TAO predicts NO_PASS when siamese_score is strictly greater than threshold. A score that equals the threshold exactly predicts PASS.

When threshold is negative, TAO computes a threshold from the inference results. It evaluates every unique siamese_score as a candidate, and it adds one further candidate just below the minimum score so that it also considers predicting NO_PASS for every sample. TAO keeps only the candidates whose NO_PASS recall reaches min_recall, then selects the candidate with the best F1 score, breaking ties first on precision and then on the higher threshold. TAO writes the winning value to threshold.txt. When threshold is zero or positive, TAO uses the value as given and does not write threshold.txt.

Warning

When no candidate threshold reaches min_recall, the command fails and writes no artifacts. TAO formats the min_recall value as a percentage with no decimals, so at the default min_recall: 1.0 the message reads ValueError: No threshold achieves 100% recall on the NO_PASS class. This is a hard failure rather than a fallback, and it is the most common first-run problem. A dataset that contains no NO_PASS rows can never satisfy the constraint, so lower min_recall or supply an explicit threshold value in that case.

TAO measures the weakness of each sample as a signed, continuous distance from the threshold in the wrong direction. For a PASS sample the weakness is siamese_score - threshold, and for a NO_PASS sample it is threshold - siamese_score. A positive value means the model classified the sample incorrectly. A negative value means the model classified it correctly, and the magnitude is the margin.

TAO then sorts every sample by descending weakness and keeps the first top_k_per_label samples for each ground-truth label. TAO groups on the uppercased, whitespace-stripped label, so labels that differ only in letter case or padding share one budget. The top_k_per_label value is a per-label budget rather than a misclassification filter, so TAO keeps correctly classified samples with a negative weakness whenever the budget allows. This behavior matters most at min_recall: 1.0, where a well-performing model produces few misclassified samples.

TAO expands each kept CSV row into one output row for every lighting condition in input_map, at the path <kpi_media_path>/<input_path>/<object_name>_<lighting><image_ext>. TAO does not check that those image files exist.

Running the Visual ChangeNet AOI Subtask#

Ask your agent to run the vcn_aoi subtask. Refer to the example prompt at the top of this page. To run the subtask yourself, use the tao_ds container:

gap_analysis vcn_aoi -e /path/to/vcn_aoi.yaml

You can override individual fields on the command line using Hydra syntax, for example:

gap_analysis vcn_aoi -e /path/to/vcn_aoi.yaml \
    top_k_per_label=25 \
    results_dir=/results/aoi_gaps

You can supply the entire specification as Hydra overrides, which is how the TAO DEFT AOI workflow invokes the subtask.

Visual ChangeNet AOI Outputs#

TAO writes the results into results_dir and creates that directory when it does not already exist.

kpi_gaps.parquet

TAO writes this file on every successful run, with a fixed schema even when it holds no rows. One source CSV row expands into one output row per lighting condition, and all of those rows share the same label, siamese_score, and weakness values.

Column	Datatype	Description
`filepath`	string	The full path to the image for one lighting condition, built as `<kpi_media_path>/<input_path>/<object_name>_<lighting><image_ext>`.
`label`	string	The ground-truth label of the source sample, with surrounding whitespace stripped and the original letter case preserved.
`siamese_score`	float	The Siamese similarity score that the Visual ChangeNet inference run assigned to the source sample.
`weakness`	float	The signed distance from the threshold in the wrong direction. A positive value marks a misclassified sample.

weak_samples_breakdown.txt

TAO writes this plain-text summary on every successful run. It reports the threshold in use, the top_k_per_label value, the total KPI sample count, and the total number of weak samples kept. It then adds one line per label giving the count for that label, the percentage it represents, and a split into misclassified samples, whose weakness is greater than zero, and marginal samples, which make up the remainder.

threshold.txt

TAO writes this file only when threshold is negative and TAO computes the threshold automatically. The file holds a single floating-point value.

VLM Binary Classification Question Gap Analysis#

The vlm_bcq subtask extracts the false positives and false negatives from the answers that a VLM gives to a binary classification question, that is, a question whose ground-truth answer is either yes or no. The resulting list of disagreements tells you which videos the model handles incorrectly, so you can mine for more data like them.

Data Input for VLM Binary Classification Question Gap Analysis#

The predictions_json file must be a JSON array of objects. Each object must provide a video_id string that gives the path to the video file, a response string that holds the free-form answer of the model, and a gt string that holds the ground-truth label. An object can also provide an optional question string, which TAO carries through to the output.

The following example shows the shape of the predictions array:

[
  {
    "video_id": "clips/intersection_0042.mp4",
    "question": "Is there a pedestrian in the crosswalk?",
    "response": "Yes, a pedestrian is crossing from the left.",
    "gt": "no"
  },
  {
    "video_id": "clips/intersection_0043.mp4",
    "response": "No pedestrians are visible in this clip.",
    "gt": "yes"
  }
]

TAO validates the whole file before it does any work, so a malformed file fails immediately. A top level that is not an array raises ValueError: predictions_json must be a JSON array of prediction objects. An element that is not an object raises ValueError: predictions_json item <i> is not an object. An element that omits gt, response, or video_id raises ValueError: predictions_json item <i>: missing '<key>'. A missing file raises a FileNotFoundError.

The videos_dir field is the base directory that TAO uses to resolve relative video_id values. When it is empty or contains only whitespace, TAO treats the video_id values as absolute paths. TAO resolves the output paths to absolute paths in both cases.

Creating an Experiment Specification File for the VLM Binary Classification Question Subtask#

The container ships a template specification file for the vlm_bcq subtask at nvidia_tao_ds/rcca/gap_analysis/experiment_specs/vlm_bcq.yaml. Copy it and fill it in.

predictions_json: ???
videos_dir: ""
results_dir: ???

The following table describes the configurable parameters.

Parameter	Datatype	Default	Description	Supported Values
`predictions_json`	string		The path to the predictions JSON file. Each item must have a `video_id`, a `response`, and a `gt` field. This field is required.	Any valid path
`results_dir`	string		The output directory for the gap-analysis results. This field is required.	Any valid path
`videos_dir`	string	“”	The base directory that TAO uses to resolve relative `video_id` paths. When this value is empty, TAO treats `video_id` values as absolute paths.	Any valid path

TAO extracts a yes or no answer from both response and gt by lowercasing the text and searching it for the whole words yes and no. The search matches on word boundaries, so a word such as nothing does not match no.

Note

When both yes and no appear in the same text, or when neither appears, the extraction is ambiguous. TAO logs a warning that names the sample and skips it. Skipped samples appear in no artifact and in no count.

TAO marks a sample as a false positive (FP) when the response is yes and the ground truth is no, and as a false negative (FN) when the response is no and the ground truth is yes. TAO drops the pairs that agree.

Running the VLM Binary Classification Question Subtask#

Ask your agent to run the vlm_bcq subtask. Refer to the example prompt at the top of this page. To run the subtask yourself, use the tao_ds container:

gap_analysis vlm_bcq -e /path/to/vlm_bcq.yaml

You can override individual fields on the command line using Hydra syntax, for example:

gap_analysis vlm_bcq -e /path/to/vlm_bcq.yaml \
    videos_dir=/data/its_clips \
    results_dir=/results/bcq_gaps

VLM Binary Classification Question Outputs#

Both outputs of the vlm_bcq subtask are conditional on the presence of gaps.

Note

When TAO finds no false positives and no false negatives, it logs No KPI Gaps found!, exits successfully, and writes no output files at all. TAO does not create results_dir in that case, so a consumer that lists the directory receives a no such file or directory error rather than an empty directory. The object_detection subtask behaves differently and always writes its four artifacts.

When gaps exist, TAO writes the following two files into results_dir.

kpi_gaps.jsonl

This file holds one JSON object per line, encoded as UTF-8, and preserves the order of the input array.

Column	Datatype	Description
`video_id`	string	The resolved absolute path to the video file.
`error_type`	string	The error class of the sample: `FP` for a false positive or `FN` for a false negative.
`question`	string	The question passed through from the input item, or an empty string when the input item carried no question.
`ground_truth`	string	The raw `gt` string from the input item, not the extracted yes or no answer.
`response`	string	The raw model response string from the input item.

kpi_gaps_report.txt

This plain-text file holds a table that counts the false positives, the false negatives, and the total, in the following shape:

KPI Gaps Report (binary ground truth: yes/no)
+------------+-------+---------------------------------+
| Error Type | Count | Description                     |
+------------+-------+---------------------------------+
| FP         | 12    | model said yes, ground truth no |
| FN         | 7     | model said no, ground truth yes |
| Total      | 19    |                                 |
+------------+-------+---------------------------------+

KPI Metric Corrections in TAO 7.2.0#

This section describes two corrections to the analytics kpi_analyze task, which belongs to Data Analytics and not to gap_analysis. The corrections are documented here because a DEFT object-detection loop runs analytics kpi_analyze and gap_analysis object_detection in the same container on the same data at adjacent stages, so it is easy to attribute a metric change to the wrong task. Neither correction ever affected the AP50 values that gap_analysis object_detection produces, because that subtask computes AP50 through an independent torchmetrics code path.

The first correction changes the false-negative sentinel. The analytics kpi_analyze task encodes each undetected ground-truth box as a synthetic scored entry that carries a sentinel confidence, and that sentinel changed from 0.0 to -1.0. The scoring loop counts an entry as a true positive when its confidence is greater than or equal to conf_threshold. With the old 0.0 sentinel and kpi.conf_threshold set to 0.0, every false negative satisfied that test and became a true positive, which pinned recall at 1.0 and inflated precision and accuracy. The new sentinel sits below every valid confidence, so TAO now counts false negatives correctly at any threshold. This correction affects only the runs that set kpi.conf_threshold explicitly to 0.0. The shipped specification file uses 0.3 and the built-in default is 0.5, and at any positive threshold the earlier behavior was already correct.

The second correction fixes the argument order of the voc_ap helper. That helper takes recall first and precision second, but the caller passed the two values in the opposite order, so TAO computed average precision by sweeping precision and maximizing recall. This correction is unconditional and applies at every conf_threshold value.

Note

The average precision and mean average precision values that analytics kpi_analyze reports in TAO 7.2.0 are not comparable to the values from earlier releases. The precision, recall, true positive, false positive, false negative, and accuracy values remain comparable, except for runs at conf_threshold: 0.0, which the sentinel correction also affects.

Used by TAO DEFT Workflows#

The DEFT loops of TAO Skill Bank each evaluate a baseline, analyze the gaps, enhance the data, retrain the model, and gate the result on a metric contract. Every gap_analysis subtask serves as the analysis stage of one such loop, and the following table maps each subtask to the loop that consumes it.

Subtask	DEFT Workflow	How the Output Is Used
`object_detection`	DEFT Object Detection loop, which fine-tunes Grounding DINO on Object Detection Visual Grounding annotations	The row count of `weak_images.parquet` gates the loop: zero rows means that every class met its configured AP50 threshold on every image, and the loop stops early. Otherwise the loop feeds `weak_images.parquet` directly into image embedding generation, because its `filepath` column needs no conversion, and the weak file paths drive nearest-neighbor matching against a source pool. Refer to Data Mining. The loop records `gap_report.json` on the stage commit.
`vcn_aoi`	DEFT AOI loop, which improves Visual ChangeNet printed circuit board inspection	The loop routes `kpi_gaps.parquet` into two independently computed subsets, one for data mining and one for synthetic anomaly generation, using the `filepath` and `label` columns.
`vlm_bcq`	TAO DEFT Cosmos Reason loop, which mines intelligent transportation systems video	A workflow script builds the predictions JSON file by rewriting Cosmos Reason prediction identifiers into resolved video paths, and `kpi_gaps.jsonl` then drives nearest-neighbor video mining. Because the subtask writes nothing when there are no gaps, the loop treats a missing or empty `kpi_gaps.jsonl` after a successful run as zero weak samples and stops.

Each DEFT loop has its own page that documents the prerequisites and the stages of that loop in full.