nemo_automodel.components.models.glm5_next.vision

GLM-5.3 image encoder with checkpoint-compatible module names.

Module Contents

Classes

Name	Description
`Glm5NextVisionAttention`	Bidirectional per-image attention over `[vision_tokens, hidden]`.
`Glm5NextVisionBlock`	Pre-norm bidirectional vision transformer block.
`Glm5NextVisionMLP`	Clamped SwiGLU vision feed-forward block.
`Glm5NextVisionModel`	Patch encoder returning image features in the language hidden size.
`Glm5NextVisionOutput`	Raw and merged vision token representations.
`Glm5NextVisionPatchEmbed`	Conv3d patch projection from `[patches, CTP*P]` to vision hidden.
`Glm5NextVisionPatchMerger`	Post-downsample projection in the text hidden dimension.
`Glm5NextVisionRotaryEmbedding`	Two-axis rotary frequencies used by the vision transformer.

Functions

Name	Description
`_rotate_half`	-
`_vision_cu_seqlens`	Return per-frame attention boundaries `[segments + 1]`.
`_vision_position_ids`	Return block-major H/W positions `[vision_tokens, 2]`.

Data

__all__

API

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionAttention(
    config: nemo_automodel.components.models.glm5_next.config.Glm5NextVisionConfig,
    dtype: torch.dtype
)

Bases: Module

Bidirectional per-image attention over [vision_tokens, hidden].

dropout

= config.attention_dropout

head_dim

= config.hidden_size // config.num_heads

k_norm

num_heads

= config.num_heads

proj

q_norm

qkv

scaling

= self.head_dim ** -0.5

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionAttention.forward(
    hidden_states: torch.Tensor,
    cu_seqlens: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor]
) -> torch.Tensor

Attend within each cu-seqlens segment and return [tokens, hidden].

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionBlock(
    config: nemo_automodel.components.models.glm5_next.config.Glm5NextVisionConfig,
    dtype: torch.dtype
)

Bases: Module

Pre-norm bidirectional vision transformer block.

attn

= Glm5NextVisionAttention(config, dtype)

mlp

= Glm5NextVisionMLP(config, dtype)

norm1

norm2

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionBlock.forward(
    hidden_states: torch.Tensor,
    cu_seqlens: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor]
) -> torch.Tensor

Transform [vision_tokens, hidden] without cross-image attention.

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionMLP(
    config: nemo_automodel.components.models.glm5_next.config.Glm5NextVisionConfig,
    dtype: torch.dtype
)

Bases: Module

Clamped SwiGLU vision feed-forward block.

down_proj

gate_proj

swiglu_limit

= config.swiglu_limit

up_proj

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionMLP.forward(
    hidden_states: torch.Tensor
) -> torch.Tensor

Transform [tokens, vision_hidden] with clamped SwiGLU.

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionModel(
    config: nemo_automodel.components.models.glm5_next.config.Glm5NextVisionConfig
)

Bases: Module

Patch encoder returning image features in the language hidden size.

blocks

downsample

dtype

Return the patch embedding dtype.

merger

= Glm5NextVisionPatchMerger(config, dtype)

patch_embed

= Glm5NextVisionPatchEmbed(config, dtype)

post_layernorm

rotary_pos_emb

= Glm5NextVisionRotaryEmbedding(head_dim // 2)

spatial_merge_size

= config.spatial_merge_size

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionModel.forward(
    pixel_values: torch.Tensor,
    grid_thw: torch.Tensor
) -> nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionOutput

Encode flattened patches using grid metadata [images, (t,h,w)].

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionModel.init_weights(
    buffer_device: torch.device,
    init_std: float
) -> None

Initialize vision parameters without materializing outside buffer_device.

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionOutput(
    last_hidden_state: torch.Tensor,
    pooler_output: torch.Tensor
)

Dataclass

Raw and merged vision token representations.

last_hidden_state

Tensor

pooler_output

Tensor

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionPatchEmbed(
    config: nemo_automodel.components.models.glm5_next.config.Glm5NextVisionConfig,
    dtype: torch.dtype
)

Bases: Module

Conv3d patch projection from [patches, C*T*P*P] to vision hidden.

in_channels

= config.in_channels

patch_size

= config.patch_size

proj

temporal_patch_size

= config.temporal_patch_size

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionPatchEmbed.forward(
    pixel_values: torch.Tensor
) -> torch.Tensor

Project flattened patches to [vision_tokens, vision_hidden].

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionPatchMerger(
    config: nemo_automodel.components.models.glm5_next.config.Glm5NextVisionConfig,
    dtype: torch.dtype
)

Bases: Module

Post-downsample projection in the text hidden dimension.

down_proj

= nn.Linear(context, dim, bias=False, dtype=dtype)

gate_proj

= nn.Linear(dim, context, bias=False, dtype=dtype)

post_projection_norm

= nn.LayerNorm(dim, dtype=dtype)

proj

= nn.Linear(dim, dim, bias=False, dtype=dtype)

swiglu_limit

= config.swiglu_limit

up_proj

= nn.Linear(dim, context, bias=False, dtype=dtype)

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionPatchMerger.forward(
    hidden_states: torch.Tensor
) -> torch.Tensor

Project downsampled image tokens [tokens, text_hidden].

class nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionRotaryEmbedding(
    dim: int,
    theta: float = 10000.0
)

Bases: Module

Two-axis rotary frequencies used by the vision transformer.

nemo_automodel.components.models.glm5_next.vision.Glm5NextVisionRotaryEmbedding.forward(
    position_ids: torch.Tensor
) -> torch.Tensor

Map H/W ids [tokens, 2] to frequencies [tokens, head_dim/2].

nemo_automodel.components.models.glm5_next.vision._rotate_half(
    hidden_states: torch.Tensor
) -> torch.Tensor

nemo_automodel.components.models.glm5_next.vision._vision_cu_seqlens(
    grid_thw: torch.Tensor
) -> torch.Tensor

Return per-frame attention boundaries [segments + 1].

nemo_automodel.components.models.glm5_next.vision._vision_position_ids(
    grid_thw: torch.Tensor,
    merge_size: int
) -> torch.Tensor

Return block-major H/W positions [vision_tokens, 2].

nemo_automodel.components.models.glm5_next.vision.__all__ = ['Glm5NextVisionModel', 'Glm5NextVisionOutput']