DINOv3#

Introduction#

DINOv3 is a self-supervised learning (SSL) family in TAO, exposed through the dinov3 task. It is built for continual pre-training: rather than training a vision transformer from scratch, you start from public DINOv3 weights and keep training on your own unlabeled images to adapt the backbone to a new domain. Training combines the DINO (image-level self-distillation), iBOT (masked-patch), and KoLeo (feature-spreading) objectives with an optional Gram-anchoring term that stabilizes dense features during long or high-resolution schedules.

DINOv3 inherits its training flow from Nv-DINOv2: the multi-crop dataloader, the projection heads, the EMA teacher, and the warmup-cosine schedulers are shared. The DINOv3-specific differences are:

  • 2D axial RoPE (rotary position embedding) in place of a learnable absolute positional embedding. Position is encoded by rotating the query/key vectors of patch tokens inside attention; the [CLS] and register tokens receive an identity rotation.

  • a GELU MLP feed-forward network for the ViT-S/B/L sizes (the larger ViT-S+/H+/7B use SwiGLU).

  • Sinkhorn-Knopp (SwAV) centering of the teacher’s DINO/iBOT head outputs, rather than softmax centering.

  • the optional Gram-anchoring loss, off by default for the ViT-B baseline.

TAO 7.2.0 adds an optional LoRA (Low-Rank Adaptation) path for continual pre-training that freezes the backbone and trains small low-rank adapters instead. This lowers optimizer and gradient memory and limits drift away from the pretrained features. Refer to LoRA Continual Pre-Training.

Benefits#

  • No labels required: adapts a strong ViT backbone to your domain using only unlabeled images.

  • Continual pre-training: starts from public DINOv3 weights, so it reaches a domain-adapted backbone in a short schedule instead of a full from-scratch pre-training.

  • Downstream reuse: the domain-adapted backbone is converted to the timm/backbone_v2 layout with dinov3 convert and consumed by downstream supervised tasks through pretrained_backbone_path (refer to Downstream Use).

  • Parameter-efficient option: LoRA freezes the backbone and trains low-rank adapters, so a continual pre-training run needs less optimizer and gradient memory and stays closer to the pretrained features (refer to LoRA Continual Pre-Training).

Data Input for DINOv3#

DINOv3 expects input data to be RGB images stored in a single directory. Supported image formats include: .jpg, .jpeg, .png, .ppm, .bmp, .pgm, .tif, .tiff, and .webp. DINOv3 reuses the Nv-DINOv2 multi-crop dataloader, so no labels or annotation files are required.

Creating a Training Experiment Specification File#

The top-level structure of a DINOv3 training experiment spec mirrors the other TAO tasks:

Parameter

Data Type

Default

Description

Supported Values

model

dict config

–

The configuration of the model architecture.

dataset

dict config

–

The configuration of the dataset.

train

dict config

–

The configuration of the training task.

inference

dict config

–

The configuration of the inference task.

export

dict config

–

The configuration of the ONNX export task.

convert

dict config

–

The configuration of the backbone-conversion task.

encryption_key

string

None

The encryption key to encrypt and decrypt model files.

results_dir

string

/results

The directory where experiment results are saved.

model#

Parameter

Data Type

Default

Description

Supported Values

centering_method

string

sinkhorn

Teacher-output centering for the DINO/iBOT heads. DINOv3 uses Sinkhorn-Knopp (SwAV); softmax is the DINOv2 fallback. If training shows instability or collapse, try softmax.

sinkhorn, softmax

distill

dict config

–

Configuration for the distillation module (reused from Nv-DINOv2).

backbone

dict config

–

Configuration for the DINOv3 backbone.

head

dict config

–

Configuration for the DINOv3 projection and prediction head.

gram

dict config

–

Configuration for Gram anchoring.

lora

dict config

–

Configuration for LoRA parameter-efficient continual pre-training.

preservation

dict config

–

Configuration for CLS-token preservation against the frozen anchor teacher.

model.backbone#

Parameter

Data Type

Default

Description

Supported Values

teacher_type

string

vit_b

The teacher backbone type.

vit_s, vit_s_plus, vit_b, vit_l, vit_h_plus, vit_7b

student_type

string

vit_b

The student backbone type.

vit_s, vit_s_plus, vit_b, vit_l, vit_h_plus, vit_7b

num_register_tokens

int

4

Number of register tokens.

≥ 0

drop_path_rate

float

0.4

Drop-path rate for stochastic-depth regularization.

[0, 1)

patch_size

int

16

Size of input patches. DINOv3 is a patch-16 architecture.

16

img_size

int

256

Input image size used in the backbone.

256, 512, 768

rope_theta

float

100.0

Frequency base for the 2D axial RoPE. Must match the timm DINOv3 reference; changing it breaks feature parity.

> 0

The available backbone sizes and their parameters (patch-16) are:

Backbone

Embed dim

Depth

Heads

FFN

MLP ratio

vit_s

384

12

6

MLP

4.0

vit_s_plus

384

12

6

SwiGLU

4.0

vit_b

768

12

12

MLP

4.0

vit_l

1024

24

16

MLP

4.0

vit_h_plus

1280

32

20

SwiGLU

4.0

vit_7b

4096

40

32

SwiGLU

2.0

vit_b is the bring-up target; vit_l is also supported (refer to the train_dinov3_vitl.yaml example spec). The larger sizes scale the same architecture.

model.head#

Parameter

Data Type

Default

Description

Supported Values

num_layers

int

3

Number of layers in the projection head.

≥ 1

hidden_dim

int

2048

Dimension of the head’s hidden layers.

≥ 1

bottleneck_dim

int

384

Dimension of the head’s bottleneck layer.

≥ 1

model.gram#

Gram anchoring keeps the student’s patch-token feature geometry close to a frozen reference by matching cosine Gram matrices of patch tokens. It is a high-resolution, large-backbone, long-schedule tool and is off by default; it is not used for the ViT-B baseline. Enable it (and set w_gram) only for the high-resolution phase. Refer to the train_dinov3_vitb_highres.yaml example spec.

Parameter

Data Type

Default

Description

Supported Values

enable

bool

False

If True, adds the Gram-anchoring loss term.

True, False

w_gram

float

0.0

Weight of the Gram-anchoring loss term.

≥ 0

start_step

int

0

Global step at which the Gram term activates.

≥ 0

teacher_source

string

pretrained

Source of the frozen Gram teacher: pretrained anchors to the loaded DINOv3 weights; ema uses an early-EMA snapshot of the current run’s teacher, refreshed every refresh_interval steps.

pretrained, ema

refresh_interval

int

0

Steps between refreshing the Gram teacher from the EMA teacher (only when teacher_source=ema). 0 = never refresh.

≥ 0

teacher_scale

float

1.0

Resolution multiple at which the Gram teacher runs relative to the student. 1.0 = same resolution (memory-safe). Higher values are memory-heavy at high resolution.

> 0

model.lora#

LoRA performs parameter-efficient adaptation of the backbone. When you enable it, TAO replaces the targeted projections of both the student and the EMA-teacher backbones with key-preserving low-rank layers, freezes the rest of the backbone, and trains only the lora_A and lora_B matrices, the DINO and iBOT heads, and mask_token. The low-rank update is dW = (alpha / rank) * B @ A. TAO folds the adapters back into the base weights during dinov3 convert and dinov3 export, so the shipped artifact is a stock DINOv3 backbone. Refer to LoRA Continual Pre-Training for the workflow, its constraints, and the merged-artifact guarantee.

Parameter

Data Type

Default

Description

Supported Values

enable

bool

False

If True, applies LoRA adapters to the targeted backbone projections.

True, False

rank

int

8

Inner dimension of the low-rank update matrices lora_A and lora_B.

≥ 1

alpha

float

16.0

Scaling factor for the low-rank update; the applied scale is alpha / rank.

–

dropout

float

0.05

Dropout applied only to the LoRA branch input; it does not affect the frozen base path.

[0, 1]

target_modules

list[str]

[qkv, proj]

Projections to adapt inside each targeted ViT block. qkv and proj are the attention projections; fc1 and fc2 additionally adapt the feed-forward network. SwiGLU backbones (ViT-S+/H+/7B) expose a fused fc1, and TAO 7.2.0 defers validation of that target.

qkv, proj, fc1, fc2

num_last_blocks

int

0

Number of trailing transformer blocks to adapt. 0, or any value greater than or equal to the backbone depth, adapts every block.

≥ 0

model.preservation#

CLS-token preservation keeps the global [CLS] embedding close to a frozen anchor teacher while the rest of the backbone adapts. Gram anchoring preserves patch-token geometry, which dense consumers such as segmentation depend on; preservation instead protects the global embedding that k-NN, linear-probe, and retrieval consumers depend on. Both terms read the same frozen anchor-teacher forward pass, so enabling preservation costs no additional teacher pass. Enabling model.preservation or model.gram builds that anchor teacher, which holds one extra frozen backbone in GPU memory. Consider enabling preservation alongside LoRA for continual pre-training.

Parameter

Data Type

Default

Description

Supported Values

enable

bool

False

If True, adds the CLS-token preservation loss terms against the frozen anchor teacher.

True, False

cls_mse_weight

float

0.05

Weight of the CLS-token mean-squared-error preservation term. 0 disables this term only.

≥ 0

cls_cosine_weight

float

0.05

Weight of the CLS-token cosine preservation term. 0 disables this term only.

≥ 0

dataset#

Parameter

Data Type

Default

Description

Supported Values

train_dataset

dict config

None

Configuration for the training dataset. images_dir is the path to the training images directory.

test_dataset

dict config

None

Configuration for the testing dataset. images_dir is the path to the testing images directory.

batch_size

int

4

The batch size for training.

≥ 1

pin_memory

bool

True

If True, enables page-locked memory for faster CPU-GPU transfer.

True, False

workers

int

8

Number of parallel workers used in data loading.

≥ 1

transform

dict config

–

Configuration parameters for data transformation (refer to the following table).

dataset.transform#

DINOv3 uses single-resolution 256 with patch-16-friendly crop sizes.

Parameter

Data Type

Default

Description

Supported Values

n_global_crops

int

2

Number of global crops to generate.

≥ 1

global_crops_scale

list[float]

[0.32, 1.0]

Scale range for global crops.

Range: (0.0, 1.0]

global_crops_size

int

256

Size (in pixels) of global crops.

≥ 1

n_local_crops

int

8

Number of local crops to generate.

≥ 1

local_crops_scale

list[float]

[0.05, 0.32]

Scale range for local crops.

Range: (0.0, 1.0]

local_crops_size

int

112

Size (in pixels) of local crops (a multiple of the patch size).

≥ 1

train#

Parameter

Data Type

Default

Description

Supported Values

pretrained_model_path

string

None

Path to the public DINOv3 weights used to initialize continual pre-training. Can be a directory, in which case TAO looks for model.safetensors, pytorch_model.bin, and model.pth, in that order, or a direct .safetensors, .pth, .bin, or .tlt file. Both the timm and Hugging Face layouts are supported; refer to Running the Model.

–

resume_training_checkpoint_path

string

None

Path to a checkpoint to resume training from.

–

num_gpus

int

1

The number of GPUs to use.

≥ 1

num_nodes

int

1

The number of nodes to use.

≥ 1

num_epochs

int

10

The number of training epochs.

≥ 1

checkpoint_interval

int

1

The interval (in epochs) at which checkpoints are saved.

≥ 1

layerwise_decay

float

1.0

Layerwise learning-rate decay factor.

(0, 1]

clip_grad_norm

float

3.0

Maximum gradient norm for gradient clipping.

> 0

num_prototypes

int

131072

Number of prototypes used in the DINO/iBOT heads.

> 0

precision

string

16-mixed

Mixed-precision setting for training.

–

distributed_strategy

string

auto

Lightning distributed strategy. auto keeps the Nv-DINOv2 behaviour (single-device or DDP); fsdp shards params/grads/optimizer (FULL_SHARD) for high-resolution and large-backbone multi-GPU training.

auto, ddp, fsdp

cudnn

dict config

–

cuDNN determinism controls (refer to the following note).

optim

dict config

–

Optimizer configuration (optim: adamw).

adamw

schedulers

dict config

–

Learning-rate / weight-decay / momentum / teacher-temperature schedulers (shared with Nv-DINOv2).

Note

Determinism caveat. The DINOv3 example specs set train.cudnn.deterministic: False and train.cudnn.benchmark: True on purpose. The SSL backbone uses xformers memory_efficient_attention, whose backward pass has no deterministic kernel, so the global determinism default (True; refer to Deterministic Training) is turned off for this family, because it is required for the xformers attention backward to run. As a result, DINOv3 SSL training is not bit-for-bit reproducible across repeated runs.

The schedulers sub-configuration (learning_rate, last_layer_learning_rate, weight_decay, momentum, teacher_temperature) is identical in structure to Nv-DINOv2; refer to that page for the per-scheduler fields.

Example Specification File#

The baseline ViT-B continual pre-training spec (train_dinov3_vitb.yaml):

encryption_key: tlt_encode
results_dir: /path/to/experiment_results
model:
  distill:
    enable: False
    disable_masking: False
    pretrained_non_distill_pl_model_path: null
  centering_method: "sinkhorn"
  backbone:
    teacher_type: "vit_b"
    student_type: "vit_b"
    num_register_tokens: 4
    drop_path_rate: 0.4
    patch_size: 16
    img_size: 256
    rope_theta: 100.0
  head:
    num_layers: 3
    hidden_dim: 2048
    bottleneck_dim: 384
  gram:
    enable: False
    w_gram: 0.0
    start_step: 0
    teacher_source: "pretrained"
  lora:
    enable: False
dataset:
  train_dataset:
    images_dir: /path/to/img_dir
  batch_size: 16
  workers: 10
  transform:
    n_global_crops: 2
    global_crops_scale: [0.32, 1.0]
    global_crops_size: 256
    n_local_crops: 8
    local_crops_scale: [0.05, 0.32]
    local_crops_size: 112
train:
  resume_training_checkpoint_path: null
  # Public DINOv3 ViT-B weights (timm format); a directory or a file.
  pretrained_model_path: /path/to/dinov3/vitb16
  # xformers memory_efficient_attention has no deterministic backward kernel.
  cudnn:
    deterministic: False
    benchmark: True
  num_nodes: 1
  num_gpus: 1
  num_epochs: 10
  checkpoint_interval: 1
  layerwise_decay: 1.0
  clip_grad_norm: 3.0
  optim:
    optim: "adamw"
  schedulers:
    learning_rate:
      val_base: "${eval: '5e-5 * (${dataset.batch_size} * ${train.num_gpus} * ${train.num_nodes} / 1024) ** (1/2)'}"
      val_final: 1e-6
      warm_up_steps: 10000
    last_layer_learning_rate:
      val_base: "${eval: '5e-5 * (${dataset.batch_size} * ${train.num_gpus} * ${train.num_nodes} / 1024) ** (1/2)'}"
      val_final: 1e-6
      warm_up_steps: 10000
      freeze_steps: 1250
    weight_decay:
      val_base: 0.04
      val_final: 0.2
    momentum:
      val_base: 0.9999
      val_final: 1.0
    teacher_temperature:
      val_base: 0.07
      val_final: 0.07
      val_start: 0.04
      warm_up_steps: 37500
  num_prototypes: 131072
  results_dir: "${results_dir}/train"

Two additional example specs ship alongside the baseline:

  • train_dinov3_vitb_highres.yaml: a high-resolution (img_size: 768) phase that turns Gram anchoring on (model.gram.enable: True, w_gram: 2.0, teacher_source: ema) and seeds from a 256-adapted checkpoint.

  • train_dinov3_vitl.yaml: the ViT-L (teacher_type/student_type: vit_l) baseline at img_size: 256 with Gram off.

Running the Model#

The dinov3 task exposes the train, inference, export, convert, and default_specs subtasks. default_specs prints a generated reference spec from the config schema.

Note

The public DINOv3 ViT-B weights are license-gated on Hugging Face. Accept the license at facebook/dinov3-vitb16-pretrain-lvd1689m first, then download the weights in either the timm or the Hugging Face format:

# timm release layout
hf download timm/vit_base_patch16_dinov3.lvd1689m --local-dir <weights_dir>

# Hugging Face DINOv3ViTModel export layout
hf download facebook/dinov3-vitb16-pretrain-lvd1689m --local-dir <weights_dir>

Point train.pretrained_model_path at <weights_dir> (a directory) or at a specific weight file.

train.pretrained_model_path accepts either a directory or a single file:

  • In a directory, TAO looks for model.safetensors, pytorch_model.bin, and model.pth, in that order, and loads the first one it finds.

  • A file must carry a .safetensors, .pth, .bin, or .tlt extension.

TAO supports both the timm release layout and the Hugging Face DINOv3ViTModel export layout. TAO detects the Hugging Face layout from its embeddings.* and layer.N.* keys and remaps it onto the TAO backbone. The remap fuses the separate q_proj, k_proj, and v_proj tensors into the single qkv projection. The Hugging Face export also supplies mask_token, which the timm release omits. TAO applies two additional loading behaviors:

  • TAO rejects DINOv2 and Nv-DINOv2 checkpoints. It detects the absolute pos_embed tensor that DINOv3 replaces with RoPE and reports an error instead of loading a mismatched backbone.

  • TAO automatically merges a checkpoint that still carries LoRA adapters when it loads that checkpoint into a plain backbone, for example when you point export or a fresh training run at it.

# Continual pre-training from the public DINOv3 weights.
dinov3 train -e /path/to/train_dinov3_vitb.yaml \
    train.pretrained_model_path=<weights_dir> \
    results_dir=<output_dir>

# Inference / ONNX export from an SSL checkpoint.
dinov3 inference -e /path/to/train_dinov3_vitb.yaml inference.checkpoint=<ckpt>
dinov3 export    -e /path/to/train_dinov3_vitb.yaml export.checkpoint=<ckpt>

# Print a reference spec.
dinov3 default_specs

For multi-GPU runs on hosts without working PCIe peer-to-peer, set NCCL_P2P_DISABLE=1.

LoRA Continual Pre-Training#

Full continual pre-training updates every backbone weight, which for a ViT-B backbone means roughly 86 million parameters. LoRA (Low-Rank Adaptation) instead freezes those weights and learns a low-rank update dW = (alpha / rank) * B @ A on the targeted projections of each transformer block. TAO injects the adapters into the student and the EMA-teacher backbones identically, so only the lora_A and lora_B matrices, the DINO and iBOT projection heads, and mask_token receive gradients. TAO logs the resulting trainable-parameter count and fraction when it injects the adapters, so you can confirm the run is parameter-efficient from the training log.

Choose LoRA when your target domain is small, when you want to limit how far the backbone drifts from the pretrained features, or when optimizer and gradient memory is constrained. Choose full continual pre-training when the domain shift is large and you have enough unlabeled data to support it.

Consider enabling model.preservation alongside LoRA. The anchor teacher never receives LoRA adapters: it stays pinned to the frozen pretrained weights, which is what makes it a stable reference.

No shipped experiment specification enables LoRA, so add the blocks yourself. The following overlay applies to train_dinov3_vitb.yaml and spells out every model.lora and model.preservation field:

model:
  lora:
    enable: True
    rank: 8
    alpha: 16.0
    dropout: 0.05
    target_modules: ["qkv", "proj"]
    num_last_blocks: 0
  preservation:
    enable: True
    cls_mse_weight: 0.05
    cls_cosine_weight: 0.05

Every value other than the two enable flags is the shipped default, so you can leave it as it is. The overlay lists the fields in full so that the available knobs are discoverable in one place, not because each one needs setting.

Then launch training the same way as a full continual pre-training run:

dinov3 train -e /path/to/train_dinov3_vitb.yaml \
    train.pretrained_model_path=<weights_dir> \
    results_dir=<output_dir>

Note

TAO 7.2.0 enforces the following LoRA constraints when training starts:

  • You cannot combine LoRA with model.distill.enable. Distillation builds an additional student_ema backbone whose adapter lifecycle this release does not cover.

  • LoRA requires model.gram.teacher_source: pretrained. An EMA-refreshed anchor tracks the drifting teacher, which defeats preservation. The shipped train_dinov3_vitb_highres.yaml specification sets teacher_source: ema, so you cannot enable LoRA on it as shipped.

  • LoRA does not support train.distributed_strategy: fsdp. Use auto or ddp. The LoRA optimizer state is small, so DDP is sufficient.

In addition, SwiGLU backbones (ViT-S+/H+/7B) expose a fused fc1. TAO 7.2.0 defers validation of that target, so keep to the default qkv and proj targets on those backbones. TAO does not reject the fc1 and fc2 targets there, so nothing warns you beyond this note.

Checkpoints saved during a LoRA run keep the stock DINOv3 keys and add lora_A, lora_B, and lora_scaling entries, so nothing is renamed. Both dinov3 convert and dinov3 export fold W + (alpha / rank) * B @ A back into the base weights and drop every adapter key, which makes the shipped artifact a stock DINOv3 backbone: timm validation passes, the ONNX graph contains no LoRA operations, and downstream tasks cannot distinguish it from a fully fine-tuned backbone.

TAO stores the alpha / rank scale in the lora_scaling buffer inside the checkpoint, so the merge is self-contained and does not read the experiment specification. Changing model.lora.alpha at convert or export time therefore has no effect on the merge, and a checkpoint with a missing or partial adapter set fails with an error rather than merging at the wrong scale.

Downstream Use: dinov3 convert#

An SSL run produces a domain-adapted backbone in TAO’s SSL layout. To consume it in downstream supervised tasks, the dinov3 convert subtask remaps that backbone to the timm/backbone_v2 layout that the dinov3_vitb16 registry entry loads. The conversion is a pure key remap (no retraining) and runs on CPU.

If the run used LoRA, dinov3 convert folds the adapters into the base weights before it remaps them and drops every lora_* key, so the converted file is a stock DINOv3 backbone and the default validate: True check still passes. dinov3 export performs the same merge before it traces the ONNX graph. Downstream tasks therefore need no configuration change. Refer to LoRA Continual Pre-Training.

Parameter

Data Type

Default

Description

Supported Values

checkpoint

string

–

SSL DINOv3 checkpoint to convert: a stripped backbone file (student_*.pth / teacher_*.pth) or a full Lightning .pth/.ckpt.

–

output_path

string

–

Output path for the timm-format backbone (.safetensors or .pth). Defaults to <results_dir>/dinov3_<arch>_backbone.safetensors.

–

source

string

teacher

Which SSL sub-model’s backbone to export. The EMA teacher is the recommended feature extractor for continual pre-training.

student, teacher, student_ema

validate

bool

True

Validate the converted state dict against a fresh timm DINOv3 model (key + shape match) before writing.

True, False

results_dir

string

None

Directory for convert results/logs.

–

# Convert the EMA-teacher backbone into timm layout.
dinov3 convert -e /path/to/train_dinov3_vitb.yaml \
    convert.checkpoint=<results_dir>/train/teacher_epoch_XXX_step_YYYYY.pth \
    convert.output_path=<out>/dinov3_vitb_backbone.safetensors

The converted file loads through the standard pretrained_backbone_path field of the downstream supervised tasks that consume the dinov3_vitb16 backbone, for example Image Classification, SegFormer, and Visual ChangeNet. Refer to those pages for how each task wires up pretrained_backbone_path; this page does not duplicate the downstream backbone documentation.