Skip to navigation

nemo_automodel.components.models.deepseek_v41.vision

View as MarkdownOpen in Claude

DeepSeek vision encoder and spatial merger with independent batched images.

Module Contents

Classes

NameDescription
DeepseekV41VisionAlignerThe released padded spatial merger and GELU projection into text width.
DeepseekV41VisionTransformerThe released full-attention 2D-RoPE ViT, configured by nested V4.1 fields.

Functions

NameDescription
_vision_configTranslate the typed V4.1 config into the existing vision implementation’s config.

API

class nemo_automodel.components.models.deepseek_v41.vision.DeepseekV41VisionAligner(
)

Bases: DeepseekV4VisionAligner

The released padded spatial merger and GELU projection into text width.

Parameters:

config
DeepseekV41Config

Top-level V4.1 configuration containing vision and text widths.

nemo_automodel.components.models.deepseek_v41.vision.DeepseekV41VisionAligner.forward(
x: torch.Tensor,
n_h: int | None = None,
n_w: int | None = None,
image_grid_hws: torch.Tensor | None = None
) -> torch.Tensor

Merge each image independently, then batch the projection GEMMs.

Parameters:

x
torch.Tensor

Concatenated unmerged image features [patches, vision_dim].

n_h
int | NoneDefaults to None

Patch rows for a single-image call.

n_w
int | NoneDefaults to None

Patch columns for a single-image call.

image_grid_hws
torch.Tensor | NoneDefaults to None

Integer patch grids [images, 2] in feature order.

Returns: torch.Tensor

Features [sum(ceil(height / ratio) * ceil(width / ratio)), hidden].

class nemo_automodel.components.models.deepseek_v41.vision.DeepseekV41VisionTransformer(
)

Bases: DeepseekV4VisionTransformer

The released full-attention 2D-RoPE ViT, configured by nested V4.1 fields.

Parameters:

config
DeepseekV41Config

Top-level V4.1 configuration containing the vision dimensions.

config
= config.vision_config
spatial_merge_size
int = 1
nemo_automodel.components.models.deepseek_v41.vision.DeepseekV41VisionTransformer.forward(
patches: torch.Tensor,
n_h: int | None = None,
n_w: int | None = None,
grid_thw: torch.Tensor | None = None,
return_dict: bool = False
) -> torch.Tensor | transformers.modeling_outputs.BaseModelOutputWithPooling

Encode independent images with one call per FSDP/checkpointed block.

Parameters:

patches
torch.Tensor

Normalized RGB patches [patches, 3, patch_size, patch_size] or flattened [patches, 3 * patch_size**2], in image order.

n_h
int | NoneDefaults to None

Patch rows for the backward-compatible single-image call.

n_w
int | NoneDefaults to None

Patch columns for the backward-compatible single-image call.

grid_thw
torch.Tensor | NoneDefaults to None

Integer grids [images, 3], each (frames, height, width). Frames are independent images, including frame-sharder dummies.

return_dict
boolDefaults to False

Return the pooler-output contract used by frame sharding.

Returns: torch.Tensor | BaseModelOutputWithPooling

Unmerged features [patches, vision_dim], or a ModelOutput containing

Translate the typed V4.1 config into the existing vision implementation’s config.