nemo_automodel.components.models.muse_glimmer.vision

Native MuseGlimmer vision path copied from the canonical Transformers implementation.

Module Contents

Classes

Name	Description
`MuseGlimmerVisionAdapter`	Canonical two-layer visual adapter.
`MuseGlimmerVisionAttention`	Canonical packed bidirectional vision attention.
`MuseGlimmerVisionBlock`	Canonical vision encoder layer.
`MuseGlimmerVisionEncoder`	Canonical processor-patch vision encoder.
`MuseGlimmerVisionMLP`	Canonical vision MLP.
`MuseGlimmerVisionRotaryEmbedding`	Canonical independent-frequency two-axis vision RoPE.

Functions

Name	Description
`apply_rotary_pos_emb_vision`	Apply the canonical split-half vision RoPE in float32.
`get_vision_bilinear_indices_and_weights`	Copy the checkpoint’s grid-sample-equivalent position interpolation.
`get_vision_cu_seqlens`	Return one packed-attention segment per frame.
`get_vision_position_ids`	Build canonical block-major two-dimensional vision positions.
`get_vision_window_index`	Copy the canonical window-attention permutation and segment lengths.
`rotate_half`	Rotate the two halves of the hidden dimension.

API

class nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionAdapter(
    config: nemo_automodel.components.models.muse_glimmer.config.MuseGlimmerConfig
)

Bases: Module

Canonical two-layer visual adapter.

act

= ACT2FN[config.projector_hidden_act]

c_fc

c_proj

nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionAdapter.forward(
    x: torch.Tensor
) -> torch.Tensor

class nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionAttention(
    config
)

Bases: Module

Canonical packed bidirectional vision attention.

dim

= config.hidden_size

head_dim

= self.dim // self.num_heads

k_proj

= nn.Linear(self.dim, self.dim, bias=True)

num_heads

= config.num_attention_heads

o_proj

= nn.Linear(self.dim, self.dim, bias=True)

q_proj

= nn.Linear(self.dim, self.dim, bias=True)

scaling

= self.head_dim ** -0.5

v_proj

= nn.Linear(self.dim, self.dim, bias=True)

nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionAttention.forward(
    hidden_states: torch.Tensor,
    cu_seqlens: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor]
) -> torch.Tensor

class nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionBlock(
    config
)

Bases: GradientCheckpointingLayer

Canonical vision encoder layer.

attn

= MuseGlimmerVisionAttention(config)

ln_1

= nn.LayerNorm(config.hidden_size, eps=1e-05)

ln_2

= nn.LayerNorm(config.hidden_size, eps=1e-05)

mlp

= MuseGlimmerVisionMLP(config)

nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionBlock.forward(
    hidden_states: torch.Tensor,
    cu_seqlens: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor]
) -> torch.Tensor

class nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionEncoder(
    config: nemo_automodel.components.models.muse_glimmer.config.MuseGlimmerConfig
)

Bases: Module

Canonical processor-patch vision encoder.

conv1_linear

ln_post

ln_pre

positional_embedding_vlm

rotary_emb

= MuseGlimmerVisionRotaryEmbedding(vision_config)

transformer

nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionEncoder._pixel_shuffle(
    hidden_states: torch.Tensor,
    grid_thw: torch.Tensor
) -> torch.Tensor

nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionEncoder.forward(
    pixel_values: torch.Tensor,
    grid_thw: torch.Tensor
) -> torch.Tensor

class nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionMLP(
    config
)

Bases: Module

Canonical vision MLP.

act

= ACT2FN[config.hidden_act]

c_fc

c_proj

nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionMLP.forward(
    x: torch.Tensor
) -> torch.Tensor

class nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionRotaryEmbedding(
    config
)

Bases: Module

Canonical independent-frequency two-axis vision RoPE.

nemo_automodel.components.models.muse_glimmer.vision.MuseGlimmerVisionRotaryEmbedding.forward(
    x: torch.Tensor,
    position_ids: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor]

nemo_automodel.components.models.muse_glimmer.vision.apply_rotary_pos_emb_vision(
    q: torch.Tensor,
    k: torch.Tensor,
    cos: torch.Tensor,
    sin: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor]

Apply the canonical split-half vision RoPE in float32.

nemo_automodel.components.models.muse_glimmer.vision.get_vision_bilinear_indices_and_weights(
    grid_thw: torch.Tensor,
    num_grid_per_side: int,
    spatial_merge_size: int
) -> tuple[torch.Tensor, torch.Tensor]

Copy the checkpoint’s grid-sample-equivalent position interpolation.

nemo_automodel.components.models.muse_glimmer.vision.get_vision_cu_seqlens(
    grid_thw: torch.Tensor
) -> torch.Tensor

Return one packed-attention segment per frame.

nemo_automodel.components.models.muse_glimmer.vision.get_vision_position_ids(
    grid_thw: torch.Tensor,
    spatial_merge_size: int
) -> torch.Tensor

Build canonical block-major two-dimensional vision positions.

nemo_automodel.components.models.muse_glimmer.vision.get_vision_window_index(
    grid_thw: torch.Tensor,
    spatial_merge_size: int,
    window_size: int,
    patch_size: int
) -> tuple[torch.Tensor, torch.Tensor]

Copy the canonical window-attention permutation and segment lengths.

nemo_automodel.components.models.muse_glimmer.vision.rotate_half(
    x: torch.Tensor
) -> torch.Tensor

Rotate the two halves of the hidden dimension.