Data Enhanced Fine-tuning (DEFT)#

Data Enhanced Fine-tuning (DEFT) is an agent-run, disk-backed, iterative model-improvement loop that TAO ships as application workflow skills in the TAO Skill Bank. Instead of tuning a training recipe against a fixed dataset, a DEFT loop measures where the current model is weak, adds data that targets those weaknesses, retrains, and measures again. The agent treats the loop as a disk-backed state machine rather than as a prose recipe, so a run can be interrupted, resumed, and audited.

TAO 7.2.0 documents two of these loops:

DEFT is a family, and the TAO Skill Bank ships further loops beyond these two. This release documents only the two loops above, and every statement on this page is scoped to them. Each skill’s SKILL.md file and references/ files remain the executable contract; this page summarizes them so that you can compare the loops and choose one. For hyperparameter optimization, refer to AutoML. For the wider set of agent-driven pipelines, refer to Reference Workflows.

Loop Anatomy#

Every DEFT loop follows the same general shape: establish a baseline, analyze where the model fails, enhance the data to close those gaps, retrain, and apply a gate that decides whether to continue. The two documented loops implement that shape with different stage names, so treat the stage sequence as workflow-specific. The analysis step in both loops is backed by the TAO Data Services gap_analysis command.

Workflow

Stage sequence

DEFT for Automated Optical Inspection

train, evaluate, rca, routing, anomalygen, data_mining, data_merge, and loop_stop. The baseline runs a single time as train, inference, evaluate, and then rca, where inference is a task name rather than a committed stage.

DEFT for Object Detection

prep, inference, kpi_analyze, gap_analysis, embed, mine, stage, train, and loop_stop. The prep stage runs a single time, and the agent skips it when its artifacts already exist. The baseline runs inference and then kpi_analyze.

The baseline convention differs between the two loops, and that difference changes how you read the first result. The AOI loop trains a model at baseline. The Object Detection loop evaluates the supplied zero-shot or pretrained checkpoint as iteration 0 and only trains from iteration 1 onward, so its baseline number is a zero-shot number.

Iteration Budgets#

The max_iterations parameter sets how many enhancement and retraining passes the loop runs after the baseline. The two documented loops resolve it differently, so confirm the value in the Pre-Flight Summary before you approve a run.

The AOI loop has no default: the value is a required input, and the agent refuses to proceed past pre-flight without it. If you supply a time limit instead, the agent converts it into an estimated max_iterations and surfaces that estimate for your confirmation.

The Object Detection loop defaults max_iterations to 1, which is one mine, train, and score pass, the smallest run that yields a comparison against the baseline. An unattended run takes that default rather than stopping to ask you.

Data Enhancement Flavors#

DEFT loops enhance data in two flavors: mining-based selection, which pulls images from an existing pool, and synthetic data generation, which creates new images. A workflow can use one flavor or both. For background on the underlying data services, refer to Data Mining, Embedding Generation, and Targeted Mode Matching (TMM) Mining.

The Object Detection loop uses mining only and performs no synthetic generation. It runs tmm unique-neighbor matching over SigLIP embeddings against a prebuilt source pool whose labels come from Co-DETR pseudo-labeling during a separate one-time preparation run, so every image it adds is a real image that already existed in that pool.

The AOI loop is hybrid and combines both flavors. AnomalyGen synthetic generation contributes defect-type diversity, and the loop keeps all generated images without filtering. Real images from augmentation/mining_pool/ contribute a real-distribution anchor, and the loop keeps only entries whose cosine similarity to the current iteration’s weak-sample embeddings is greater than or equal to state.config.mining_filter.min_similarity, then applies history-aware filtering. Synthetic rows skip the similarity filter, so the cosine threshold applies to real-image mining only.

A routing stage splits the root cause analysis gaps into routing_mining_parquet and routing_anomalygen_parquet, which feed the two branches. The loop skips either branch when its routed parquet file has zero rows. Skipping both is not permitted: if mining yields zero novel rows and AnomalyGen contributed nothing, the loop takes a hard stop.

Data Enhancement Modules#

The two documented workflows share a recognizable structure in how they acquire and label new data. That structure is an observed pattern across the shipped workflows, not a published extension interface. The TAO Skill Bank ships no extension-contract document at 7.2.0, so treat what follows as a description of the existing loops rather than as a supported interface for building your own. The pattern has two parts. The first is a data source, which is either a generator or a source pool paired with a miner. The second is an annotator that emits the target model’s training format, because the retraining stage consumes that format directly.

Workflow

Data source

Annotator

Training format

DEFT for Automated Optical Inspection

AnomalyGen generation plus AMP synthetic data generation, and real images from augmentation/mining_pool/ selected by SigLIP k-nearest-neighbor search

scripts/changenet_data_pair_prepare.py, which maps the NG image to input_path and the OK image to golden_path

Visual ChangeNet four-column paired CSV file with the columns input_path, golden_path, label, and object_name, assembled into train_combined_iter${N}.csv

DEFT for Object Detection

An unlabeled raw pool plus tmm unique-neighbor mining

Co-DETR pseudo-labels the pool, the agent folds the labels to the target classes, converts KITTI to COCO to Object Detection Visual Grounding (ODVG), and then runs stage_mined_odvg.py

ODVG: tmm_odvg.jsonl plus labelmap.json, appended as one new entry to the dataset.train_data_sources list

One invariant holds across the workflows. Every stage commits its outputs through commit_stage.py, which verifies the stage’s required artifacts before it writes state, so a stage that produced nothing cannot record itself as complete.

Metric Contracts#

The metric contract is an AOI concept. The AOI loop optimizes one primary customer metric and can enforce zero or more secondary constraints. The state.metric_contract object, stored once at state["metric_contract"], selects the evaluator and the comparison, and metrics are not restricted to a fixed name or unit. The contract carries the metric name, display_name, operator, target, and unit, an evaluator binding, and a constraints array whose entries repeat the same comparison fields.

The supported operators are <, <=, >, and >=. They determine both success and best-iteration selection. The agent first keeps the candidates that satisfy every secondary constraint, then selects the lowest primary result for < and <= or the highest primary result for > and >=. If no candidate satisfies all constraints, the agent retains the direction-aware primary best and reports it as a clearly non-passing result. The agent never infers direction from the metric’s name.

Canonicalization turns a customer sentence into contract fields. The requirement FAR < 10% at Recall=100%, which reads as a false alarm rate (FAR) below 10 percent at 100 percent recall, becomes a primary metric far_pct with operator <, target 10, unit %, and the bundled far_at_recall evaluator, plus a secondary constraint recall_pct with operator >=, target 100, and unit %. Never compare a fractional FAR value to a percentage target.

The contract binds one of three evaluator types: a builtin evaluator shipped with the skill, a command evaluator that names a customer-owned executable, or an artifact evaluator that consumes a result file an external system produces. A custom metric must have a command or artifact evaluator before the Pre-Flight Summary, because the evaluator is part of the approved run configuration and changing it requires a revised summary. The bundled recorder does not trust passed from the evaluator. It recomputes passed from the approved contract. For the field-by-field contract, the evaluator invocation forms, and the result file schema, refer to DEFT for Automated Optical Inspection.

The contract drives stopping, best-iteration selection, and the report headline. A result can be structurally valid while missing the target, and the loop continues until the metric passes or the loop reaches max_iterations.

The Object Detection loop has no metric contract and does not gate. It reports mean average precision (mAP) rather than gating on it, the loop does not exit early on a metric target, and a regression does not stop the loop.

State and Resume#

Both documented loops use deft_state.json as the single resume record. The agent initializes the file a single time with init_deft_state.py and then mutates it only through commit_stage.py. Never hand-edit it, reinitialize it, or write it with inline Python, jq, or heredocs. The AOI state file also holds the immutable execution_policy, the resolved mining paths, and optional terminal artifacts.

Both loops use the same run directory form and resume-detection rule. A fresh run always creates a new timestamped results/run_<YYYYMMDD_HHMMSS>/ directory and preserves prior runs under their own directories. The agent detects a resumable run by checking for results/run_*/deft_state.json, and it does not create the run directory before the user gate.

Both loops record the same event field set with different stage enumerations. Each event carries seq, ts, iter, stage, status, summary, skip_reason (only when the agent skips a stage), duration_sec, context_tokens, and tokens. AOI status values are ok, error, and skipped, while Object Detection uses only ok and error. AOI context_tokens starts at 0 and align_token_usage.py backfills it at loop end; the Object Detection field is reserved and always 0.

Where the ordered event stream lives differs between the loops. The Object Detection run directory carries loop_log.jsonl as an append-only event stream with one JSON line per stage, alongside .deft_commit.lock, which enforces one commit_stage.py writer at a time, and .deft_commit.journal, which exists only while a commit is in flight so that the next commit_stage.py run can undo an interrupted one. The AOI loop has no loop_log.jsonl: its ordered events live inside deft_state.json.

Disk is the source of truth, and neither loop infers progress from conversation history. The AOI skill runs deft_context.py on startup, after context compaction, before every stage, and before any completion claim, then resumes from its durable next_stage together with the recorded state. The Object Detection skill runs its audit before every stage and uses the last_committed, next_action, and read_before_action output. Neither skill trusts assistant prose or an artifact that state does not record. On startup or resume each one shows the last five events so that you can see where the run stands.

There is exactly one user gate: pre-flight confirmation. The agent prints the Pre-Flight Summary, then stops and waits for you to type go, yes, looks good, or a similar explicit approval. It launches no side-effecting step before that approval, which rules out docker run, training, synthetic data generation, and mutations under ${RESULTS_DIR}/. Reading specifications, listing files, running docker image inspect, and populating the summary table remain permitted. Autonomous describes the behavior after the gate, not before it.

After the gate the skill runs the entire loop without asking for confirmation, stopping only if a step fails with an unrecoverable error or a hard-stop gate fires. The loop requires auto-mode, because it fires constant side-effecting calls after the gate and stalls on the first prompt without auto-accept or bypass-permissions mode. If any run parameter changes after the agent shows the original summary, the agent re-runs pre-flight and shows an updated summary. The summary preserves every explicit value you supply and labels the source of every run parameter as user, spec, or default.

Outputs by Workflow#

The two documented loops do not produce a uniform artifact set, so read the outputs per workflow rather than as a shared contract. The most visible difference is the report format: the AOI loop renders an HTML report, and the Object Detection loop renders a Markdown report with no HTML template.

Workflow

Report artifact

Distinctive artifacts

DEFT for Automated Optical Inspection

DEFT_Loop_Report.html

deft_state.json, best_model.json for inference handoff metadata, best_model_inference_spec.yaml as a ready-to-run TAO inference specification built from the training configuration, iter${ITER}_summary.md, a baseline/ tree, a per-iteration iter${ITER}/ tree, dataset/merge_validation.json as mandatory proof of the merge and leakage checks, RCA_Report.md with rca_images/ per root cause analysis stage, and allocation.json under the anomalygen stage

DEFT for Object Detection

DEFT_Loop_Report.md

deft_state.json, loop_log.jsonl, .deft_commit.lock, .deft_commit.journal, train_grounding_dino.yaml seeded at baseline from your template, a prep/ tree, iter${N}_summary.md, a baseline/ tree, and a per-iteration iter${N}/ tree

Both reports are generated, never hand-authored. The AOI loop refreshes its HTML report through a post-commit hook after every successful commit, and the Object Detection loop renders its Markdown report after each completed iteration and again at loop end. The Object Detection loop produces no best_model.json and no best_model_inference_spec.yaml; those artifacts are AOI only. For the full output trees and the report repair commands, refer to DEFT for Automated Optical Inspection and DEFT for Object Detection.

Execution Platforms#

Neither loop hardcodes a compute backend. Both discover the execution platforms from the installed platform skills and name Docker, SLURM, Kubernetes, and Brev; neither lists virtualenv. The AOI skill asks which supported platform you intend to use and accepts an external platform skill in addition to the four it names. Both skills require Docker and the NVIDIA Container Toolkit, and the Object Detection skill additionally requires one or more CUDA GPUs.

Air-gapped support is not symmetric. The AOI loop has a first-class air-gapped mode with a dedicated references/air-gap.md file, and its front matter advertises support for air-gapped and offline runs with pre-staged assets. The mode activates when you set AIR_GAPPED=1 in the environment, when you explicitly request it, or when the harness reports restricted networking; the agent never runs a network command to infer the mode. In this mode the agent performs no registry login or pull, converts every fetch instruction into a local presence check, and treats a missing asset as a hard stop. Setting HF_HUB_OFFLINE=1 alone disables Hugging Face access only and does not activate global air-gap mode. Air-gap mode changes asset resolution and network behavior, not user authorization or stage ordering. For the enforcement details and the full pre-staging list, refer to DEFT for Automated Optical Inspection.

The Object Detection loop has no air-gapped mode. It ships no references/air-gap.md file and has no --network-mode option. Its only accommodation is at the checkpoint level: on an air-gapped host, or any time the download cannot run, you supply --zero-shot-checkpoint directly.

Installing the Workflows#

The DEFT workflows ship inside the tao-skills plugin in the TAO Skill Bank, which bundles every model, data, platform, and application skill in one package. Pin the marketplace to the 7.2.0 tag, which is the latest release build. In Claude Code, run these two commands:

/plugin marketplace add NVIDIA-TAO/tao-skill-bank@7.2.0
/plugin install tao-skills@tao-skill-bank

For the full installation matrix and a first successful run, refer to Getting Started with NVIDIA TAO Toolkit.

Note

Release tags are stable, while main moves continuously and can reference prerelease container builds. Every link in this documentation therefore points at the 7.2.0 tag.

Choosing a Workflow#

Choose a workflow by the model you want to improve and the data you can supply. If you have a pool of unlabeled real images and an object detector, choose the Object Detection loop. If you are improving a paired-image defect classifier and want synthetic defects in addition to mined real images, choose the AOI loop.

Workflow

Domain

Model

Data enhancement flavor

Page

DEFT for Automated Optical Inspection

Automated optical inspection

Visual ChangeNet Classification

Hybrid: AnomalyGen synthetic generation plus real-image k-nearest-neighbor mining

AOI loop

DEFT for Object Detection

Object detection

Grounding DINO

Mining only, using tmm unique-neighbor matching over SigLIP embeddings

Object Detection loop

Source of Truth#

The skill files are the executable contract, and this page summarizes them. Read the skill sources at the 7.2.0 tag when you need the exact behavior: