DEFT for Automated Optical Inspection#

The DEFT automated optical inspection (AOI) workflow improves an NVIDIA TAO Visual ChangeNet classify model for printed circuit board inspection. It runs as a closed loop: evaluate a baseline model, perform root cause analysis (RCA) on the samples the model gets wrong, generate synthetic defects, mine matching real images from a production pool, retrain on the enlarged dataset, and evaluate again. The loop scores every iteration against a customer-defined metric contract, and the loop continues until the metric meets its target or the iteration budget is exhausted. The skill treats the run as a disk-backed state machine rather than a prose recipe. This page summarizes the workflow so that you can plan a run; the skill SKILL.md file and the files under its references/ directory remain the executable contract and take precedence wherever they differ from this page. Refer to Data Enhanced Fine-tuning (DEFT) for the shared concepts and to DEFT for Object Detection for the detection variant.

Use this workflow when the request identifies an AOI, printed circuit board defect, Visual ChangeNet, or ChangeNet task. Do not use it for a standalone TAO training run, a one-off inference job, generic anomaly generation, an analysis that only needs RCA, or CLIP and SigLIP image retrieval.

Supported Models#

The loop trains TAO Visual ChangeNet classify with either an end-to-end C-RADIOv2-B backbone or a frozen DINOv3 backbone. The file specs/baseline_spec.yaml defines the architecture, and the bundled template ships the type c_radio_v2_vit_base_patch16_224 with freeze_backbone set to false. Refer to Visual ChangeNet and Visual ChangeNet-Classification for the model documentation. For a frozen DINOv3 backbone, set the backbone type in the baseline specification to one of the DINOv3 profile identifiers listed below and set model.backbone.freeze_backbone to true. The supported profile identifiers are the following:

vit_small_dinov3
vit_small_plus_dinov3
vit_base_dinov3
vit_large_dinov3
vit_huge_plus_dinov3
vit_7b_dinov3

Staging pulls the matching timm/vit_*_patch16_dinov3.lvd1689m weights, which requires that you accept the DINOv3 license and set HF_TOKEN in network-enabled mode. In air-gapped mode the exact profile file must already be staged, because the staging helper rejects a missing or mismatched checkpoint.

The backbone path rule is strict. The value of pretrained_backbone_path must be a local file bind-mounted into the container. The loader accepts a local .pth or .ckpt file read with torch.load and a local .safetensors file read with safetensors.torch.load_file. It rejects an https://huggingface.co/ URL and a bare repository identifier such as nvidia/C-RADIOv2-B: either form raises FileNotFoundError and reports Execution status: FAIL within about three seconds. A null or empty value does not fail loudly; it silently degrades held-out evaluation quality. Because the Hugging Face repository nvidia/C-RADIOv2-B ships only model.safetensors and no .pth file, scripts/stage_backbone.py stages it locally with the default filename c_radio_v2_b.safetensors.

Prerequisites and Pre-Flight#

The workflow requires Docker, NVIDIA Container Toolkit, and a GPU on the host. Pre-flight records the exact GPU model and memory with nvidia-smi --query-gpu=name,memory.total --format=csv,noheader. The AnomalyGen container is published for amd64 only and fails silently on arm64 hosts, so pre-flight hard-stops when the host architecture is not in the image platform list; there is no emulation path for GPU workloads. Visual ChangeNet classify with C-RADIOv2-B at specification defaults runs out of memory on a single GPU with 48 GB of memory, so pre-flight recommends dataset.classify.batch_size of at most 16 on 48 GB GPUs and at most 8 on 24 GB GPUs.

Pre-flight selects the host Python interpreter with bash scripts/deft_python.sh and probes no other bare interpreter first. That interpreter must import six modules: pandas, numpy, and matplotlib for key performance indicator (KPI) analysis, pyarrow for parquet input and output in routing and mining, huggingface_hub for backbone staging, and boto3 for S3 operations. Pre-flight runs "$PYTHON" -c "import pandas, numpy, matplotlib, pyarrow, huggingface_hub, boto3". In air-gapped mode a missing import is a hard stop, and the run cannot invoke a package manager. Pre-flight also performs a presence check on two credentials and never prints or greps their values.

Variable	Purpose
`NGC_KEY`	Required for all `nvcr.io` image pulls, covering the TAO toolkit train, inference, deploy, and data services images and the `paidf-anomalygen` synthetic data generation container. Recorded as `N/A (offline)` in air-gapped mode.
`HF_TOKEN`	Required for pre-flight Hugging Face downloads of the Visual ChangeNet backbone, Cosmos diffusion, T5, C-RADIO-V3, DINOv2, SAM2, Qwen-VL, and SigLIP models, cached under `augmentation/anomalygen/base_checkpoints/`. It also gates the printed circuit board fine-tuned checkpoint and reference dataset auto-fetch. Recorded as `N/A (offline)` in air-gapped mode.

Both variables must be non-empty in the process environment for planned network actions. If the backbone is already staged, or if HF_TOKEN is unset while a staged file is present, pre-flight uses the staged file and skips the download, so HF_TOKEN is optional when every asset is pre-staged. The approved credential files are ~/.tao/secrets.env, ~/.config/tao/.env, or a file that you point the run at, loaded with set -a; source /path/to/.env; set +a. The run never creates that file, writes a credential value into it, or prints its contents.

Pre-flight resolves the network mode before dependencies and reads exactly one branch: references/air-gap.md for air-gapped mode or references/network-bootstrap.md for network-enabled mode. It never loads or executes the network bootstrap in an air-gapped run and never issues a network command to infer the mode. Air-gapped mode activates when AIR_GAPPED=1 is present in the process environment, when you explicitly request air-gapped or offline execution, or when the harness reports restricted networking; every other case uses the network-enabled path. Setting HF_HUB_OFFLINE=1 disables Hugging Face access only and does not activate global air-gap mode. Pre-flight records both the mode and its activation source, then runs the ordered checks in the following table.

Step	Check
1	Locate the workspace root, specifications, CSV files, checkpoints, and augmentation assets; derive `RESULTS_DIR` as `<workspace>/results/run_$(date +%Y%m%d_%H%M%S)`, or reuse an existing `results/run_*/deft_state.json` on resume; resolve `IMAGES_DIR`, preferring `<workspace>/images` with a legacy fallback of `<workspace>/kpi/images`; run `scripts/validate_training_csv.py` against each of the three base CSV files; resolve the mining source independently from its images; and check the host Python dependencies.
2	Read the relevant `references/` files for command syntax and output contracts.
3	Perform the credential presence check.
4	Log in to the registry only after the approval gate and only in network-enabled mode; air-gapped mode neither logs in nor pulls.
5	Resolve and export `TAO_PYT_IMAGE`, `TAO_DS_IMAGE`, and `AG_IMAGE` with `scripts/resolve_versions_key.py` against `versions.yaml`, and hard-stop on any resolver error.
6	Verify that the images are present locally with `docker image inspect`, run an architecture compatibility check, and run a GPU architecture runnability probe; `no kernel image is available` is a hard stop, and air-gapped mode uses `docker run --pull=never`.
7	Apply the path rule: pre-create the iteration directories under `${RESULTS_DIR}/iter${ITER}/` and mount `<workspace>` at the same absolute path.
8	Verify the GPU count, record the exact GPU model and memory, probe the three AnomalyGen override slots, stage the backbone with `scripts/stage_backbone.py --workspace <workspace>`, and gate the AnomalyGen Guardrail safety model file check before synthetic data generation. Staging is mandatory, a non-zero exit is a hard stop, and there is no URL fallback.
9	Run the GPU memory sanity check.
10	Run the train and validation leakage check before resuming any prior run.

Pre-flight then prints a summary with the sections ### Run config, ### Dataset, ### Augmentation, ### Docker Images, and ### Runtime Estimate. Air-gapped runs add evidence rows for the network mode, activation source, container images, models and datasets, and credentials. The following screenshot shows a typical launch prompt and the response that starts the workflow.

Example prompt for running the DEFT loop and the agent's initial response

The next screenshot shows the pre-flight output with the dataset, augmentation, and tool resolution rows populated.

DEFT loop pre-flight stage output showing dataset, augmentation, and tool resolution

There is exactly one user gate in the whole workflow: the pre-flight confirmation. The run prints the summary, stops, and waits for explicit approval such as go, yes, or looks good. No side-effecting step runs before approval, which rules out docker run, training, synthetic data generation, and any mutation under ${RESULTS_DIR}/. Reading specifications, listing files, running docker image inspect, and populating the summary table are allowed before approval. Autonomy describes behavior after this gate, not before it: after approval the run does not pause between steps and does not ask whether to continue, so enable auto-accept or bypass-permissions mode before you approve. A fix applied during the run is not another user gate, but any change to a run parameter after the summary requires a fresh pre-flight and an updated summary. The gate persists in air-gapped mode. Pre-flight resolves the container images from versions.yaml at the 7.2.0 tag.

Environment variable	Image	Used by
`TAO_PYT_IMAGE`	`nvcr.io/nvidia/tao/tao-toolkit:7.2.0-pyt`	`train`, `evaluate`, and `rca`
`TAO_DS_IMAGE`	`nvcr.io/nvidia/tao/tao-toolkit:7.2.0-data-services`	`data_mining`
`AG_IMAGE`	`nvcr.io/nvidia/paidf-anomalygen:1.0.1`	`anomalygen`

Note

versions.yaml is authoritative even when a reference, a cached transcript, or a previously installed plugin mentions a different tag. Do not treat these image tags as skill-owned constants.

The workflow asks one consolidated question for missing required inputs and never asks about a parameter that has a default. The only required input is max_iterations, which has no default: the run asks for it when you do not supply it and does not proceed past pre-flight without it. If you give a time limit instead, the run converts it using the runtime estimate and surfaces the estimate for confirmation. The documented defaults are the following.

Parameter	Default
`training_epochs`	`num_epochs` from `specs/baseline_spec.yaml`; the template ships 10, raised toward 20 only in the low thousands of rows
`num_SDG`	20
`min_similarity`	Read from `config.mining_filter.min_similarity`, with 0.9 as the fallback
`top_k_per_target`	5
Workspace root	The path in your prompt, otherwise `~/workspace`
Pretrained backbone	The first staged weight under `augmentation/backbone/`
AnomalyGen checkpoint, dataset, and Cosmos base models	The staged `augmentation/anomalygen/` paths
`init_deft_state.py` command line	`--batch-size 16`, `--top-k-per-target 5`, `--knn-metric cosine` with the choices `cosine`, `euclidean`, and `manhattan`, and `--min-similarity` unset

The runtime estimate in the summary is a heuristic rather than a measurement of your hardware. It derives from a measured RTX 6000 Ada (48 GB) run at about 200 training rows with default epochs, which took about 33 minutes per iteration, so 10 iterations amount to about 6.5 hours of wall time plus a one-time asset pull of roughly 22 GB.

Data Layout#

This loop trains on your own AOI inspection data. There is no public AOI dataset to download, and the NV_PCB_Siamese paths that appear throughout the skill are a naming convention for the mount layout rather than a fetchable dataset. The following paths are required, and the loop can neither fabricate nor download them.

Path	Contents
`specs/baseline_spec.yaml`	Visual ChangeNet train and evaluation specification. Copy the bundled template at `references/baseline_spec.yaml` and adjust it; this file is the source of truth for architecture, lighting, and image size.
`train/base/training_set.csv`	Seed training rows using the four mandatory Visual ChangeNet columns; about 200 rows is a normal first-run size.
`train/base/validation_set.csv`	Held-out rows with the same schema; these rows must not overlap the training rows, and leakage is a hard stop.
`kpi/testing_set.csv`	KPI test rows with the same schema; the workflow measures false accept rate (FAR) and recall on these rows.
`images/`	Canonical image root referenced by every CSV file above, holding both real captures and golden references.
`NGC_KEY` and `HF_TOKEN`	Process environment credentials, used for approved network-enabled actions only.

The input tree looks like this.

<workspace>/
  images/                                  canonical real-image root shared by train, validation, KPI, and mining CSV files
    golden/images/                         golden reference component crops
  specs/baseline_spec.yaml                 Visual ChangeNet train and evaluation specification
  train/base/
    training_set.csv
    validation_set.csv
  kpi/
    testing_set.csv
  augmentation/
    mining_pool/
      mining_pool.csv
    anomalygen/                            optional user override slots
      checkpoints/<project>/
      base_checkpoints/                    manifest-pinned Cosmos base models cache, about 22 GB, Predict2 Text2Image 2B
      datasets/<project>/                  defect_spec.jsonl and per-texture image and mask subdirectories
  results/run_<YYYYMMDD_HHMMSS>/           created or resumed by this workflow, equal to ${RESULTS_DIR}

All three CSV files share one schema with four mandatory columns.

Column	Required	Meaning
`input_path`	Yes	Directory, not a file, holding the component crop, relative to `images_dir`
`golden_path`	Yes	Directory of the golden reference image for the same component; a row without it is unusable, because this is a siamese change detector
`label`	Yes	`PASS` in exact case, which is the class-0 sentinel of the dataloader, or a defect string such as `Missing` or `Shift`
`object_name`	Yes	Component identifier or filename stem, for example `C1018@1`, with no lighting suffix and no filename extension

Extra production metadata columns are optional. The loop can preserve them when they are present, but they are not part of the required Visual ChangeNet CSV contract. One example is boardname, which the mid-iteration leakage difference includes in its comparison tuple together with the four required columns. TAO constructs each image path as {images_dir}/{input_path}/{object_name}_{light}{image_ext}, where {light} is a key in dataset.classify.input_map, for example SolderLight, and {image_ext} is dataset.classify.image_ext, for example .jpg. The specification defines both, not the CSV file. The header and one example row look like this.

input_path,golden_path,label,object_name
690-5G190-0510-001P1/AOI_B/FXLH_..._AOI_B_20230317130332/PerComponent,golden/images/690-5G190-0510-001P1BOT/,PASS,C1018@1

Warning

The dataloader performs case-sensitive equality against the literal PASS. If you lowercase that label, every row lands in class 1, and fpratio_sampling fails at the start of training with RuntimeError: invalid multinomial distribution (sum of probabilities <= 0) and RuntimeError: Please call iter(combined_loader) first within about 30 seconds. Apply the rule row["label"] = row["label"] if row["label"] == "PASS" else row["label"].lower().strip(). Because that rule lowercases every label other than PASS, defect-label casing elsewhere in this page is illustrative rather than normative.

The real-image pool at augmentation/mining_pool/mining_pool.csv is optional. Provide it when you have a production-line image stream to mine from; the loop runs without it and falls back to synthetic-only augmentation. The pool is append-only and grows from day one to day N. The override slots augmentation/anomalygen/checkpoints/<project>/, augmentation/anomalygen/base_checkpoints/, and augmentation/anomalygen/datasets/<project>/ are also optional bring-your-own inputs.

Important

The mining pool must contain no-good (NG) samples for every defect type listed in the KPI testing set, not only PASS samples. When a defect type has no NG sample in the pool, no candidate crosses min_similarity, and that defect type silently receives no real-image augmentation. Past production pools have been missing SHIFT, LIFTED_LEAD, UPSIDE_DOWN, TOMBSTONE, and POLARITY at the same time, which left five of eight KPI defect types with no augmentation path.

Auto-Fetched Assets#

The workflow fetches four assets on first use in network-enabled mode, and you pre-stage them only for an air-gapped run: the ChangeNet backbone nvidia/C-RADIOv2-B at about 393 MB, the AnomalyGen fine-tuned checkpoint nvidia/Cosmos-AnomalyGen-PCB-2B at about 5 GB, the Cosmos and AnomalyGen base checkpoints at about 22 GB for the 2B workflow set, and the AnomalyGen printed circuit board reference dataset nvidia/Cosmos-AnomalyGen-PCB-Dataset. The HF_TOKEN variable gates all four. The mining embedding model is the google/siglip-base-patch16-224 snapshot at SIGLIP_MODEL_PATH.

Important

The AnomalyGen printed circuit board reference dataset is a generator fine-tuning set that holds a clean image, a mask, and a defect specification. It is not your AOI training data, and it cannot substitute for it.

DEFT AOI is always a printed circuit board workflow, so the loop selects --uc pcb and reads step=14000 from ${checkpoint_dir}/checkpoints/latest_checkpoint.txt. The loop creates everything under results/run_<YYYYMMDD_HHMMSS>/, the per-iteration synthetic_iter* staged images, and the combined training CSV files, and you never hand-author any of them.

Loop Stages#

The state file records a fixed set of stage identifiers: train, evaluate, rca, routing, anomalygen, data_mining, data_merge, and loop_stop. The iteration labels are baseline, iter1, iter2, and so on. Inference is a task name rather than a stage: when the workflow calls the training skill it passes a separate task name of train, inference, or evaluate, but the committed stage value is still only train or evaluate.

The baseline runs once before the loop, in the order train, inference, evaluate, and then rca. The train sub-step is skipped when deft_state.json arrives with iterations.baseline.stage_completed equal to train and a best_ckpt_path that points at an existing file. The AutoML pipeline skill pre-seeds those fields from its winning AutoML configuration so that DEFT does not retrain at the same hyperparameters. In that case the baseline picks up at inference against the pre-seeded checkpoint, evaluates the configured customer metric, and then runs RCA. Each iteration afterward executes seven steps in order.

Step	What it does	What it produces
1	Resolves the prior RCA input from disk: iteration 1 reads `state["iterations"]["baseline"]["rca_gaps_parquet"]`, and iteration N greater than 1 reads `state["iterations"][f"iter{N-1}"]["rca_gaps_parquet"]`.	`state["current_iteration"]` set to N on the first `iterN` `routing` event
2	Routes weak samples with the routing skill.	`routing_mining_parquet` and `routing_anomalygen_parquet`
3	Runs adaptive mask placement (AMP) and synthetic defect generation, followed by an inline sub-step for the synthetic training contribution.	Staged synthetic Visual ChangeNet pairs
4	Mines the real-image pool and runs a mid-iteration leakage check.	Mined real-image rows
5	Assembles the training CSV file with monotonic growth.	`train_combined_iter${ITER}.csv` and its provenance file
6	Runs pre-train validation, which is an existence check plus a train and validation leakage check, then commits the `data_merge` stage.	`merge_validation.json`
7	Fine-tunes and evaluates the model.	A new checkpoint and a metric result

Routing#

Routing splits the RCA gap rows into a synthetic branch and a mining branch. Its inputs are the gaps parquet file, whose required columns are filepath and label, and source_pool_csv, which must be the Visual ChangeNet formatted <workspace>/augmentation/mining_pool/mining_pool.csv with a label column. Never pass an empty string for that input and never substitute KPI rows. The anomalygen_supported_labels set defaults to PASS, EXCESS_SOLDER, MISSING, and BRIDGE, and you override it only when AnomalyGen generator coverage has changed. Routing writes mining_gaps.parquet, anomalygen_gaps.parquet, and routing_summary.txt under ${RESULTS_DIR}/iter${N}/routing_results/<timestamp>/, and it always writes both parquet paths even when one subset is empty. If both subsets are empty because every label was dropped, the phase status becomes failed, the run stops, it logs status=error, and it surfaces the list of dropped labels.

Synthetic Defect Generation#

Each iteration runs the generation skill in mode=inference_only and needs only two of its phases. Phase 2 runs prep_testcase.sh and AMP routing to produce testcase.jsonl in about 10 seconds with no GPU. Phase 3 runs run_sdg.sh, whose diffusion pass generates reconstructed_image/ and original_image/ in 1 to 3 minutes on Blackwell. Setting num_search_run=0 and nn_threshold=0 skips phases 4 through 7. The stage also takes checkpoint_dir; step, an integer parsed from checkpoints/latest_checkpoint.txt with iter_ and .pt stripped; dataset_dir, set to <workspace>/augmentation/anomalygen/datasets/<project>/ and passed verbatim, with clean_dir set to the same value and defect_spec set to ${dataset_dir}/defect_spec.jsonl; num_SDG, the per-iteration budget read from deft_state.json and allocated across defect types in proportion to mask count; num_gpus=1; model_size=2b; output_dir=${RESULTS_DIR}/iter${N}/anomalygen/sdg/; and cosmos_models_dir=${COSMOS_MODELS_DIR}.

Four invariants gate the stage: cad_mask RGB values survive unchanged, text entries carry prompts, clean and CAD pairs match by filename stem, and semantic_segmentation_labels.json exists. The commit requires both SDG_result.csv and the Phase 2 file allocation.json, which is the canonical proof of what AMP allocated, and commit_stage.py sums the allocation counts to derive the allocated count. If routing_anomalygen_parquet has zero rows, the workflow does not launch the GPU generator: it sets anomalygen_skipped=true, advances stage_completed to anomalygen, and commits the stage with status=skipped and a skip reason. This is a documented branch skip and not a fabricated artifact. The scripts/changenet_data_pair_prepare.py script converts synthetic rows into Visual ChangeNet pairs and staged under ${RESULTS_DIR}/iter${ITER}/dataset/images/synthetic_iter${ITER}_ng/ and the matching _ok directory. Synthetic rows skip k-nearest-neighbor filtering, because only real-image mining applies the cosine threshold. The Guardrail check is tri-state with the values passed, failed, and not_run. A missing Guardrail checkpoint at $COSMOS/nvidia/Cosmos-Guardrail1/video_content_safety_filter/safety_filter.pt, or a generation log containing post-generation image checks are DISABLED, is a hard stop recorded as guardrail=not_run.

Real-Image Mining#

Mining retrieves real production images that resemble the routed weak samples, using the TAO data services embedding and mining stack described in Data Mining, Generating Embeddings, and Targeted Mode Matching. The embedding model can be CLIP, SigLIP, or a TAO .pth or .ckpt checkpoint, and the default is google/siglip-base-patch16-224. Never point SigLIP at a DINO or C-RADIO cache, and make sure that the embedding and mining steps use an identical model and model_path.

Mining keeps only entries whose cosine similarity is greater than or equal to state.config.mining_filter.min_similarity, and it falls back to 0.9 only when that field is unset or null. It always logs the value actually used into the similarity_threshold column of knn_summary.csv. The knn_metric value defaults to cosine, topn defaults to 5, and filter_by_label defaults to false. Mining is history-aware and runs in four ordered steps: it embeds the targets, embeds the source pool, mines k-nearest neighbors and applies the cosine floor to produce mined_raw.parquet, mined_candidates.parquet, and knn_summary.csv, and finally drops samples that earlier iterations already mined with scripts/filter_mined_history.py. That last step produces the final mined.parquet, a per-iteration mining_history_summary.json, and the run-level ledger ${RESULTS_DIR}/mining_history.json. The first three steps run inside $TAO_DS_IMAGE, and the history filter runs on the host.

A zero-row routing_mining_parquet sets data_mining_skipped=true and commits the stage with status=skipped and a skip reason. That is the only legal mining skip: an invalid source CSV file or a failed mining command with routed rows still outstanding is a hard stop. When mining yields zero novel rows, the loop continues only if AnomalyGen contributed new synthetic rows; if both producers add zero new rows, the loop hard-stops rather than retraining on an unchanged dataset. A pre-mine yield precheck warns instead of stopping, and a post-mine yield monitor warns when the current kept count falls below half the previous kept count, because a collapse from 30 kept rows to 5 in the second iteration has happened in past runs with no signal reaching you. When mined file paths are converted into CSV rows, input_path becomes the directory of the file, object_name combined with {light} and {image_ext} reconstructs the basename, and golden_path becomes the paired golden directory relative to the workspace root. Both input_path and golden_path need this collapse from file to directory.

Retraining#

Training data grows monotonically. Iteration 1 combines train/base/training_set.csv with the current iter${ITER}/mining_filter/mining_pool.csv, which is the per-iteration merged file and not the user-supplied augmentation/mining_pool/mining_pool.csv source pool. Iteration N and any resume combine the previous train_combined_iter${N-1}.csv with the current iter${ITER}/mining_filter/mining_pool.csv, and the run never re-adds the base training file when it uses a previous combined file. The provenance source values are base_train and mining_pool for iteration 1, and previous_iter_train and mining_pool for iteration N greater than 1, where every prior row must reappear unchanged as previous_iter_train.

For every iteration N of at least 1, initialize training from the previous best checkpoint using train.pretrained_model_path and not train.resume_training_checkpoint_path. The latter makes Lightning inherit current_epoch and exit with Trainer.fit stopped: max_epochs=N reached after zero training steps while still printing Execution status: PASS. The command commit_stage.py --stage train rejects a checkpoint outside ${RESULTS_DIR}/<iter-label>/train/ and requires the exact training specification, so do not copy an old checkpoint into the new directory. The per-iteration specification is asymmetric: only train_dataset.images_dir points at /data/workspace, while validation_dataset, test_dataset, and infer_dataset stay at /data/datasets/NV_PCB_Siamese/images. A bulk sed over the specification catches all four keys and breaks the latter three.

Evaluation#

Evaluation runs inference on two checkpoints rather than one: the best-validation checkpoint with the lowest val_loss and the latest checkpoint with the highest epoch. Validation loss and the deployment metric of the customer can diverge, so the workflow runs the evaluator of the metric contract on every candidate. It selects the lowest result for the < and <= operators and the highest result for the > and >= operators, and it never infers the direction from the metric name. Only checkpoints from a training run that exited with status zero and whose TAO status JSONL file has no FAILURE entry are candidates.

Root Cause Analysis#

RCA identifies the weak samples that feed the next iteration, so it runs only when another iteration follows. The RCA that feeds iteration N is already committed before iteration N starts: baseline.rca_gaps_parquet feeds iter1, and iterN.rca_gaps_parquet feeds iteration N plus 1. Do not run the baseline RCA a second time under iter1. When max_iterations is reached, the run executes the loop-end sequence with the best-iteration report and adds no post-loop RCA event, because the terminal evaluate stage transitions directly to loop_stop. The underlying analysis service is the TAO Data Services gap_analysis command.

The RCA stage passes Hydra overrides to the gap_analysis vcn_aoi container. The inference_results_dir value must be a directory that contains inference.csv, whose required columns are input_path, object_name, label, and siamese_score. The other overrides are train_config, kpi_media_path, results_dir, min_recall with a default of 1.0 for zero-miss operation, and top_k_per_label with a default of 50. Every RCA stage must complete the spot check and save rca_images/, then write the seven-section RCA_Report.md file, before it commits. If min_recall is unreachable at every threshold, the container writes unreachable_kpi.txt instead of kpi_gaps.parquet; the workflow then skips the spot check, writes the abridged report with no image directory, and does not attempt routing or mining.

Metric Contract#

The metric contract defines what success means for your run. It is stored once at state["metric_contract"] and drives both the stop condition and the best-iteration selection. There is no repository-level default contract, and init_deft_state.py requires the contract either through --kpi-target "<expr>" or through all three of --metric-name, --metric-operator, and --metric-target; mixing the two forms raises an error. The target expression grammar is <display name> <op> <number><unit> [at <context>], for example quality_score >= 0.9.

Field	Allowed values
`name`	A string matching `[a-z][a-z0-9_]*`
`display_name`	A non-empty string; defaults to `name`
`operator`	Exactly one of `<`, `<=`, `>`, or `>=`
`target`	A finite number
`unit`	A string that can be empty; the recorder forces the unit to `%` for `far_pct`
`evaluator.type`	Exactly one of `builtin`, `command`, or `artifact`
`constraints`	A list of secondary gates, each with `name`, `operator`, and `target`, plus optional `display_name` and `unit`

The three evaluator types differ in who computes the metric. The builtin type accepts exactly one identifier, far_at_recall; any other value raises unsupported builtin metric evaluator; use far_at_recall or configure a command/artifact evaluator. Its parameter is recall_target_pct, which defaults to 100.0, and the bundled implementation is scripts/analyze_kpi.py, a threshold-sweep evaluator that emits the standard metric_result.json file plus diagnostic CSV files and plots. The command type runs an absolute, customer-owned executable as an argument vector and never through eval or an interpolated shell string, in the form <path> <configured args...> --inference-csv <absolute CSV> --output-json <absolute JSON>; the path must be absolute, must exist, and must be executable. The artifact type waits for an external producer to supply the result JSON at an exact absolute path template that must contain {iter_label}, for example /home/user/workspace/metrics/{iter_label}/metric_result.json. The recorder rejects a different path, and the workflow must not substitute another metric. A custom metric must have a command or artifact evaluator before the pre-flight summary, because the evaluator is part of the approved run configuration and changing it requires a revised summary.

Constraints are secondary gates that determine both success and best-iteration selection. The workflow first keeps the candidates that satisfy every secondary constraint, then selects the lowest primary result for < and <= or the highest primary result for > and >=. If no candidate satisfies all constraints, it retains the direction-aware primary best and labels it clearly as a non-passing result. It never infers the direction from the name of the metric. A complete contract looks like this.

{
  "name": "weighted_escape_cost",
  "display_name": "Weighted escape cost",
  "operator": "<=",
  "target": 0.02,
  "unit": "cost/board",
  "evaluator": {
    "type": "command",
    "path": "/home/user/workspace/metrics/evaluate_cost.py",
    "args": ["--policy", "/home/user/workspace/metrics/policy.json"]
  },
  "constraints": [
    {"name": "recall_pct", "display_name": "Recall", "operator": ">=", "target": 99.5, "unit": "%"}
  ]
}

The run canonicalizes a false accept rate goal before it starts. The requirement FAR < 10% at Recall=100% becomes a primary metric far_pct with operator <, target 10, unit %, the bundled far_at_recall evaluator, and a secondary constraint recall_pct >= 100%. A display name of far combined with the unit % maps to far_pct, and a compact expression such as FAR < 10% at Recall=100 auto-attaches the evaluator and auto-injects the recall constraint when you supply no constraints of your own. The resulting KPI target row in the pre-flight summary reads FAR < X% at Recall=100%. The evaluator result must place the achieved recall in constraints.recall_pct, because a diagnostic-only recall value cannot satisfy the gate. Never compare a fractional false accept rate value against a percentage target. The canonical form looks like this.

{
  "name": "far_pct",
  "display_name": "FAR",
  "operator": "<",
  "target": 10,
  "unit": "%",
  "evaluator": {
    "type": "builtin",
    "id": "far_at_recall",
    "parameters": {"recall_target_pct": 100.0}
  },
  "constraints": [
    {"name": "recall_pct", "display_name": "Recall", "operator": ">=", "target": 100.0, "unit": "%"}
  ]
}

Every evaluator returns a result JSON document with the same shape. The required fields are name, a finite numeric value, a unit that matches the contract exactly, and a value for every configured constraint. The threshold and diagnostics fields are optional. The bundled recorder does not trust a passed field from the evaluator; it recomputes the outcome from the approved contract.

{
  "name": "weighted_escape_cost",
  "value": 0.018,
  "unit": "cost/board",
  "threshold": 0.31,
  "constraints": {"recall_pct": 99.8},
  "diagnostics": {"precision_pct": 94.0, "sample_count": 12480}
}

The evaluate commit passes --metric-result, --best-ckpt, --inference-csv, --training-spec, --threshold, --duration-sec, and --summary to commit_stage.py, which invokes the bundled metric recorder internally and appends an evaluate event to deft_state.json. If validation fails, it restores the original state. A result can be structurally valid and still miss the target, and the loop continues until the metric passes or until max_iterations is reached. The per-iteration status line follows the format [iter <N>/<max> · <stage>] <primary metric> · <duration> · next: <stage>.

Note

Reaching the target is not guaranteed, and the false accept rate can regress between iterations. When a run misses the target, the report renders the direction-aware absolute gap, in percentage points when the unit is %. It does not emit a failure banner or failure styling.

Outputs#

Every artifact of a run lands under one timestamped results directory. The layout below is relative to <workspace>.

results/run_<YYYYMMDD_HHMMSS>/               equal to ${RESULTS_DIR}
  deft_state.json                            resume snapshot and ordered events; single source of truth
  DEFT_Loop_Report.html                      atomically refreshed by the commit_stage.py report hook
  best_model.json                            inference handoff metadata
  best_model_inference_spec.yaml             ready-to-run TAO inference specification built from the training configuration
  iter${ITER}_summary.md                     per-iteration summary of at most 300 words
  baseline/
    train/                                   model_epoch_<EEE>_step_<SSS>.pth files, status.json, experiment.yaml, train.log
    inference/best_val/ and inference/latest/   per-checkpoint inference.csv and KPI plots from scripts/analyze_kpi.py
    rca_results/<TS>/                        kpi_gaps.parquet, threshold.txt, weak_samples_breakdown.txt
  iter${ITER}/
    routing_results/<TS>/                    mining_gaps.parquet, anomalygen_gaps.parquet, routing_summary.txt
    anomalygen/
      amp/                                   AMP testcase intermediates
      testcase.jsonl                         built by prep_testcase.sh, consumed by run_sdg.sh
      sdg/                                   SDG_result.csv, reconstructed_image/, original_image/, original_mask/, cropped_image/, cropped_mask/, annotated_image/
    ag_config_sdg.yaml                       sanitized configuration, job and model only
    mining_filter/
      mining_pool.csv                        combined synthetic rows and real mined rows used for training
      sdg_rows.csv
      knn_summary.csv                        candidate_count, kept_count, rejected_count, similarity_threshold
      source_embeddings.parquet
      target_embeddings.parquet
      mining_summary.txt
    dataset/
      train_combined_iter${ITER}.csv
      train_combined_iter${ITER}_provenance.csv   source in base_train, previous_iter_train, mining_pool
      images/synthetic_iter${ITER}_ng/ and images/synthetic_iter${ITER}_ok/
    train/
    inference/best_val/ and inference/latest/
    rca_results/<TS>/

The following artifacts are the ones you read most often.

Artifact	Purpose
`deft_state.json`	The only persistent loop record. `init_deft_state.py` writes version 4 state one time, atomically; it never runs on resume and requires `--force` to overwrite an existing file. Only `commit_stage.py` mutates the state. Never edit it with inline Python, `jq`, heredocs, or an editor.
`DEFT_Loop_Report.html`	The rendered loop report, built from the bundled template by `scripts/render_report.py`, which is the only renderer. All images are embedded as base64 data URIs so that the file opens offline.
`best_model.json`	Inference handoff metadata holding `checkpoint`, `threshold`, `metric_contract`, `metric_result`, `iteration`, `backbone`, `backbone_container_path`, `backbone_type`, `backbone_frozen`, `images_dir`, and `training_spec`.
`best_model_inference_spec.yaml`	A ready-to-run TAO inference specification built from the winning training configuration.
`iter${ITER}_summary.md`	A per-iteration narrative summary of at most 300 words.
RCA report files	Per stage, `kpi_gaps.parquet` with the columns `filepath`, `label`, `siamese_score`, and `weakness`, plus `threshold.txt` holding a single float and `weak_samples_breakdown.txt`. The agent adds the seven-section `RCA_Report.md` file and the `rca_images/` directory, which must hold at least one non-empty image and can be absent only beside `unreachable_kpi.txt`. The 7.2 container generates none of `metrics.json`, `RCA_Report.md`, or `rca_images/`.
`merge_validation.json`	The mandatory merge proof under `${RESULTS_DIR}/iter${ITER}/dataset/`, carrying `rows_checked`, `missing_file_count` of 0, and `train_val_leakage_overlap_count` of 0.

The script scripts/finalize_run.py produces the handoff artifacts. It generates the Visual ChangeNet handoff first, validates the stop reason against the final metric evidence, commits loop_stop, and records the final artifacts. It takes --results-dir, --iter-label, --stop-reason with a value of metric_met or max_iterations, and --duration-sec, and it invokes scripts/prepare_inference_spec.py --results-dir PATH, which writes best_model.json and best_model_inference_spec.yaml. It refuses the terminal commit if either file is absent. Downstream inference consumes those two files and never reads deft_state.json or the training specification directly.

The HTML report is never hand-authored. init_deft_state.py renders it once at loop start, and commit_stage.py runs the renderer as a post-commit hook after every valid commit, including failed-stage and loop_stop commits. A hook failure prints report hook failed but never rolls back a valid stage commit. You can rebuild the report manually with render_report.py --results-dir "${RESULTS_DIR}" [--require-terminal]. Completion evidence is a fresh read of deft_state.json that shows status equal to complete, iterations.baseline.status equal to complete, and the final iteration status equal to complete. A checkpoint, an inference CSV file, a report, or an assistant message is not completion evidence by itself, and a hard-stop path that leaves status equal to failed must be reported as failed rather than relabeled as a completed loop.

Execution Platforms and Air-Gapped Operation#

After you confirm that you want to run the workflow, the skill asks which supported platform you intend to use. It discovers the execution platforms from the installed platform skills, which cover Docker, Slurm, Kubernetes, and Brev, plus any external platform skill. On a runtime that surfaces only the core router skills, it reads the frontmatter of skills/platform/tao-run-on-*/SKILL.md, and it then reads the credentials section of the platform skill you choose. All documented stage invocations in the reference files are direct docker run commands, which the skill calls the direct-container fallback. Refer to Getting Started with NVIDIA TAO Toolkit for host setup and to Reference Workflows for the other workflows in this family.

The workflow supports air-gapped and offline runs with pre-staged assets. Initialize the state with --network-mode airgap and its activation source. After initialization, run every external command through scripts/deft_exec.py, which re-reads the execution policy, injects the offline variables, and adds --pull=never to direct Docker and Podman runs. Do not run package managers, registry login or pull, downloads, APIs, or network probes. Convert every fetch or pull instruction into a local presence check, because a missing image, model, dataset, or required host dependency is a hard stop. Never run pip, not even as a probe and not even after a failed bare-Python call: a package-manager attempt invalidates the air-gap run and requires a fresh run directory rather than an explanation after the fact. Use only images already present in the local Docker daemon and assets at the documented staged paths, and pass HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 to the stages that consume Hugging Face backed caches. Image checks use docker image inspect --format '{{.Architecture}}' and docker run --pull=never, and they never query a registry manifest. The normal pre-flight approval gate still applies, because air-gap mode changes asset resolution and network behavior rather than user authorization or stage ordering. An air-gapped run requires that the following assets are already present:

All three container images in the local Docker daemon.
The host Python interpreter with all six required modules.
The Visual ChangeNet backbone file, and the exact DINOv3 profile file when you use a DINOv3 backbone, because the staging helper rejects a missing or mismatched checkpoint.
The AnomalyGen fine-tuned checkpoint, the printed circuit board reference dataset, and the Cosmos base checkpoints under augmentation/anomalygen/, where every required slot must already be non-empty and a missing asset is a hard stop.
The AnomalyGen Guardrail safety model at Cosmos-Guardrail1/video_content_safety_filter/safety_filter.pt.
A local google/siglip-base-patch16-224 snapshot that contains config.json at SIGLIP_MODEL_PATH, bind-mounted at the identical absolute path.

The AnomalyGen base cache must be fully pre-staged along with the Guardrail safety model, and you must verify the base cache offline with the check built into the container before synthetic defect generation runs.

Example Prompts#

The skill lists the following prompts as representative ways to start the workflow:

“Run the DEFT loop”
“Fine-tune until the configured quality metric meets its target”
“Optimize a customer-defined metric while preserving its constraints”
“Improve my AOI ChangeNet model using RCA and synthetic defects”
“Iterate training until the deployment KPI meets the target”

Source of Truth#

The skill definition and its reference files are the executable contract for this workflow. The following sources are published at the 7.2.0 tag: