DINOv3#

Introduction#

DINOv3 is a self-supervised learning (SSL) family in TAO, exposed through the dinov3 task. It is built for continual pre-training: rather than training a vision transformer from scratch, you start from public DINOv3 weights and keep training on your own unlabeled images to adapt the backbone to a new domain. Training combines the DINO (image-level self-distillation), iBOT (masked-patch), and KoLeo (feature-spreading) objectives with an optional Gram-anchoring term that stabilizes dense features during long or high-resolution schedules.

DINOv3 inherits its training flow from Nv-DINOv2: the multi-crop dataloader, the projection heads, the EMA teacher, and the warmup-cosine schedulers are shared. The DINOv3-specific differences are:

2D axial RoPE (rotary position embedding) in place of a learnable absolute positional embedding. Position is encoded by rotating the query/key vectors of patch tokens inside attention; the [CLS] and register tokens receive an identity rotation.
a GELU MLP feed-forward network for the ViT-S/B/L sizes (the larger ViT-S+/H+/7B use SwiGLU).
Sinkhorn-Knopp (SwAV) centering of the teacher’s DINO/iBOT head outputs, rather than softmax centering.
the optional Gram-anchoring loss, off by default for the ViT-B baseline.

Benefits#

No labels required: adapts a strong ViT backbone to your domain using only unlabeled images.
Continual pre-training: starts from public DINOv3 weights, so it reaches a domain-adapted backbone in a short schedule instead of a full from-scratch pre-training.
Downstream reuse: the domain-adapted backbone is converted to the timm/backbone_v2 layout with dinov3 convert and consumed by downstream supervised tasks through pretrained_backbone_path (refer to Downstream Use).

Data Input for DINOv3#

DINOv3 expects input data to be RGB images stored in a single directory. Supported image formats include: .jpg, .jpeg, .png, .ppm, .bmp, .pgm, .tif, .tiff, and .webp. DINOv3 reuses the Nv-DINOv2 multi-crop dataloader, so no labels or annotation files are required.

Creating a Training Experiment Specification File#

The top-level structure of a DINOv3 training experiment spec mirrors the other TAO tasks:

Parameter	Data Type	Default	Description
model	dict config	–	The configuration of the model architecture.
dataset	dict config	–	The configuration of the dataset.
train	dict config	–	The configuration of the training task.
inference	dict config	–	The configuration of the inference task.
export	dict config	–	The configuration of the ONNX export task.
convert	dict config	–	The configuration of the backbone-conversion task.
encryption_key	string	None	The encryption key to encrypt and decrypt model files.
results_dir	string	/results	The directory where experiment results are saved.

model#

Parameter	Data Type	Default	Description	Supported Values
centering_method	string	sinkhorn	Teacher-output centering for the DINO/iBOT heads. DINOv3 uses Sinkhorn-Knopp (SwAV); `softmax` is the DINOv2 fallback. If training shows instability or collapse, try `softmax`.	sinkhorn, softmax
distill	dict config	–	Configuration for the distillation module (reused from Nv-DINOv2).
backbone	dict config	–	Configuration for the DINOv3 backbone.
head	dict config	–	Configuration for the DINOv3 projection and prediction head.
gram	dict config	–	Configuration for Gram anchoring.
lora	dict config	–	Disabled LoRA stub (forward-compatibility).

model.backbone#

Parameter	Data Type	Default	Description	Supported Values
teacher_type	string	vit_b	The teacher backbone type.	vit_s, vit_s_plus, vit_b, vit_l, vit_h_plus, vit_7b
student_type	string	vit_b	The student backbone type.	vit_s, vit_s_plus, vit_b, vit_l, vit_h_plus, vit_7b
num_register_tokens	int	4	Number of register tokens.	≥ 0
drop_path_rate	float	0.4	Drop-path rate for stochastic-depth regularization.	[0, 1)
patch_size	int	16	Size of input patches. DINOv3 is a patch-16 architecture.	16
img_size	int	256	Input image size used in the backbone.	256, 512, 768
rope_theta	float	100.0	Frequency base for the 2D axial RoPE. Must match the timm DINOv3 reference; changing it breaks feature parity.	> 0

The available backbone sizes and their parameters (patch-16) are:

Backbone	Embed dim	Depth	Heads	FFN	MLP ratio
vit_s	384	12	6	MLP	4.0
vit_s_plus	384	12	6	SwiGLU	4.0
vit_b	768	12	12	MLP	4.0
vit_l	1024	24	16	MLP	4.0
vit_h_plus	1280	32	20	SwiGLU	4.0
vit_7b	4096	40	32	SwiGLU	2.0

vit_b is the bring-up target; vit_l is also supported (refer to the train_dinov3_vitl.yaml example spec). The larger sizes scale the same architecture.

model.head#

Parameter	Data Type	Default	Description	Supported Values
num_layers	int	3	Number of layers in the projection head.	≥ 1
hidden_dim	int	2048	Dimension of the head’s hidden layers.	≥ 1
bottleneck_dim	int	384	Dimension of the head’s bottleneck layer.	≥ 1

model.gram#

Gram anchoring keeps the student’s patch-token feature geometry close to a frozen reference by matching cosine Gram matrices of patch tokens. It is a high-resolution, large-backbone, long-schedule tool and is off by default; it is not used for the ViT-B baseline. Enable it (and set w_gram) only for the high-resolution phase. Refer to the train_dinov3_vitb_highres.yaml example spec.

Parameter	Data Type	Default	Description	Supported Values
enable	bool	False	If `True`, adds the Gram-anchoring loss term.	True, False
w_gram	float	0.0	Weight of the Gram-anchoring loss term.	≥ 0
start_step	int	0	Global step at which the Gram term activates.	≥ 0
teacher_source	string	pretrained	Source of the frozen Gram teacher: `pretrained` anchors to the loaded DINOv3 weights; `ema` uses an early-EMA snapshot of the current run’s teacher, refreshed every `refresh_interval` steps.	pretrained, ema
refresh_interval	int	0	Steps between refreshing the Gram teacher from the EMA teacher (only when `teacher_source=ema`). `0` = never refresh.	≥ 0
teacher_scale	float	1.0	Resolution multiple at which the Gram teacher runs relative to the student. `1.0` = same resolution (memory-safe). Higher values are memory-heavy at high resolution.	> 0

model.lora#

A disabled forward-compatibility stub for parameter-efficient SSL. Keep it disabled for TAO 7.1.0.

Parameter	Data Type	Default	Description	Supported Values
enable	bool	False	If `True`, applies LoRA adapters (disabled).	True, False
rank	int	8	LoRA rank.	≥ 1
alpha	float	16.0	LoRA scaling alpha.	> 0

dataset#

Parameter	Data Type	Default	Description	Supported Values
train_dataset	dict config	None	Configuration for the training dataset. `images_dir` is the path to the training images directory.
test_dataset	dict config	None	Configuration for the testing dataset. `images_dir` is the path to the testing images directory.
batch_size	int	4	The batch size for training.	≥ 1
pin_memory	bool	True	If `True`, enables page-locked memory for faster CPU-GPU transfer.	True, False
workers	int	8	Number of parallel workers used in data loading.	≥ 1
transform	dict config	–	Configuration parameters for data transformation (refer to the following table).

dataset.transform#

DINOv3 uses single-resolution 256 with patch-16-friendly crop sizes.

Parameter	Data Type	Default	Description	Supported Values
n_global_crops	int	2	Number of global crops to generate.	≥ 1
global_crops_scale	list[float]	[0.32, 1.0]	Scale range for global crops.	Range: (0.0, 1.0]
global_crops_size	int	256	Size (in pixels) of global crops.	≥ 1
n_local_crops	int	8	Number of local crops to generate.	≥ 1
local_crops_scale	list[float]	[0.05, 0.32]	Scale range for local crops.	Range: (0.0, 1.0]
local_crops_size	int	112	Size (in pixels) of local crops (a multiple of the patch size).	≥ 1

train#

Parameter	Data Type	Default	Description	Supported Values
pretrained_model_path	string	None	Path to the public DINOv3 weights (timm format) used to initialize continual pre-training. May be a directory (the loader finds `model.safetensors`) or a direct `.safetensors`/`.pth`/`.bin` file.	–
resume_training_checkpoint_path	string	None	Path to a checkpoint to resume training from.	–
num_gpus	int	1	The number of GPUs to use.	≥ 1
num_nodes	int	1	The number of nodes to use.	≥ 1
num_epochs	int	10	The number of training epochs.	≥ 1
checkpoint_interval	int	1	The interval (in epochs) at which checkpoints are saved.	≥ 1
layerwise_decay	float	1.0	Layerwise learning-rate decay factor.	(0, 1]
clip_grad_norm	float	3.0	Maximum gradient norm for gradient clipping.	> 0
num_prototypes	int	131072	Number of prototypes used in the DINO/iBOT heads.	> 0
precision	string	16-mixed	Mixed-precision setting for training.	–
distributed_strategy	string	auto	Lightning distributed strategy. `auto` keeps the Nv-DINOv2 behaviour (single-device or DDP); `fsdp` shards params/grads/optimizer (FULL_SHARD) for high-resolution and large-backbone multi-GPU training.	auto, ddp, fsdp
cudnn	dict config	–	cuDNN determinism controls (refer to the following note).
optim	dict config	–	Optimizer configuration (`optim: adamw`).	adamw
schedulers	dict config	–	Learning-rate / weight-decay / momentum / teacher-temperature schedulers (shared with Nv-DINOv2).

Note

Determinism caveat. The DINOv3 example specs set train.cudnn.deterministic: False and train.cudnn.benchmark: True on purpose. The SSL backbone uses xformers memory_efficient_attention, whose backward pass has no deterministic kernel, so the global determinism default (True; refer to Deterministic Training) is turned off for this family, because it is required for the xformers attention backward to run. As a result, DINOv3 SSL training is not bit-for-bit reproducible across repeated runs.

The schedulers sub-configuration (learning_rate, last_layer_learning_rate, weight_decay, momentum, teacher_temperature) is identical in structure to Nv-DINOv2; refer to that page for the per-scheduler fields.

Example Specification File#

The baseline ViT-B continual pre-training spec (train_dinov3_vitb.yaml):

encryption_key: tlt_encode
results_dir: /path/to/experiment_results
model:
  distill:
    enable: False
    disable_masking: False
    pretrained_non_distill_pl_model_path: null
  centering_method: "sinkhorn"
  backbone:
    teacher_type: "vit_b"
    student_type: "vit_b"
    num_register_tokens: 4
    drop_path_rate: 0.4
    patch_size: 16
    img_size: 256
    rope_theta: 100.0
  head:
    num_layers: 3
    hidden_dim: 2048
    bottleneck_dim: 384
  gram:
    enable: False
    w_gram: 0.0
    start_step: 0
    teacher_source: "pretrained"
  lora:
    enable: False
dataset:
  train_dataset:
    images_dir: /path/to/img_dir
  batch_size: 16
  workers: 10
  transform:
    n_global_crops: 2
    global_crops_scale: [0.32, 1.0]
    global_crops_size: 256
    n_local_crops: 8
    local_crops_scale: [0.05, 0.32]
    local_crops_size: 112
train:
  resume_training_checkpoint_path: null
  # Public DINOv3 ViT-B weights (timm format); a directory or a file.
  pretrained_model_path: /path/to/dinov3/vitb16
  # xformers memory_efficient_attention has no deterministic backward kernel.
  cudnn:
    deterministic: False
    benchmark: True
  num_nodes: 1
  num_gpus: 1
  num_epochs: 10
  checkpoint_interval: 1
  layerwise_decay: 1.0
  clip_grad_norm: 3.0
  optim:
    optim: "adamw"
  schedulers:
    learning_rate:
      val_base: "${eval: '5e-5 * (${dataset.batch_size} * ${train.num_gpus} * ${train.num_nodes} / 1024) ** (1/2)'}"
      val_final: 1e-6
      warm_up_steps: 10000
    last_layer_learning_rate:
      val_base: "${eval: '5e-5 * (${dataset.batch_size} * ${train.num_gpus} * ${train.num_nodes} / 1024) ** (1/2)'}"
      val_final: 1e-6
      warm_up_steps: 10000
      freeze_steps: 1250
    weight_decay:
      val_base: 0.04
      val_final: 0.2
    momentum:
      val_base: 0.9999
      val_final: 1.0
    teacher_temperature:
      val_base: 0.07
      val_final: 0.07
      val_start: 0.04
      warm_up_steps: 37500
  num_prototypes: 131072
  results_dir: "${results_dir}/train"

Two additional example specs ship alongside the baseline:

train_dinov3_vitb_highres.yaml: a high-resolution (img_size: 768) phase that turns Gram anchoring on (model.gram.enable: True, w_gram: 2.0, teacher_source: ema) and seeds from a 256-adapted checkpoint.
train_dinov3_vitl.yaml: the ViT-L (teacher_type/student_type: vit_l) baseline at img_size: 256 with Gram off.

Running the Model#

The dinov3 task exposes the train, inference, export, convert, and default_specs subtasks. default_specs prints a generated reference spec from the config schema.

Note

The public DINOv3 ViT-B weights are license-gated on Hugging Face. Accept the license at facebook/dinov3-vitb16-pretrain-lvd1689m first, then download the timm-format weights:

hf download timm/vit_base_patch16_dinov3.lvd1689m --local-dir <weights_dir>

Point train.pretrained_model_path at <weights_dir> (a directory) or at a specific weight file.

# Continual pre-training from the public DINOv3 weights.
dinov3 train -e /path/to/train_dinov3_vitb.yaml \
    train.pretrained_model_path=<weights_dir> \
    results_dir=<output_dir>

# Inference / ONNX export from an SSL checkpoint.
dinov3 inference -e /path/to/train_dinov3_vitb.yaml inference.checkpoint=<ckpt>
dinov3 export    -e /path/to/train_dinov3_vitb.yaml export.checkpoint=<ckpt>

# Print a reference spec.
dinov3 default_specs

For multi-GPU runs on hosts without working PCIe peer-to-peer, set NCCL_P2P_DISABLE=1.

Downstream Use: `dinov3 convert`#

An SSL run produces a domain-adapted backbone in TAO’s SSL layout. To consume it in downstream supervised tasks, the dinov3 convert subtask remaps that backbone to the timm/backbone_v2 layout that the dinov3_vitb16 registry entry loads. The conversion is a pure key remap (no retraining) and runs on CPU.

Parameter	Data Type	Default	Description	Supported Values
checkpoint	string	–	SSL DINOv3 checkpoint to convert: a stripped backbone file (`student_.pth` / `teacher_.pth`) or a full Lightning `.pth`/`.ckpt`.	–
output_path	string	–	Output path for the timm-format backbone (`.safetensors` or `.pth`). Defaults to `<results_dir>/dinov3_<arch>_backbone.safetensors`.	–
source	string	teacher	Which SSL sub-model’s backbone to export. The EMA `teacher` is the recommended feature extractor for continual pre-training.	student, teacher, student_ema
validate	bool	True	Validate the converted state dict against a fresh timm DINOv3 model (key + shape match) before writing.	True, False
results_dir	string	None	Directory for convert results/logs.	–

# Convert the EMA-teacher backbone into timm layout.
dinov3 convert -e /path/to/train_dinov3_vitb.yaml \
    convert.checkpoint=<results_dir>/train/teacher_epoch_XXX_step_YYYYY.pth \
    convert.output_path=<out>/dinov3_vitb_backbone.safetensors

The converted file loads through the standard pretrained_backbone_path field of the downstream supervised tasks that consume the dinov3_vitb16 backbone, for example Image Classification, SegFormer, and Visual ChangeNet. Refer to those pages for how each task wires up pretrained_backbone_path; this page does not duplicate the downstream backbone documentation.