> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.kimi_k25_vl.model

KimiK25VL model with backend-aware DeepseekV3 language model.

This is a self-contained implementation that includes all necessary components:

* Configuration classes
* Vision tower (MoonViT3d with temporal dimension)
* Multi-modal projector (PatchMergerMLP)
* Language model backend (DeepseekV3)

## Module Contents

### Classes

| Name                                                                                                                             | Description                                                                          |
| -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| [`DeepSeekV3RotaryEmbeddingAdapter`](#nemo_automodel-components-models-kimi_k25_vl-model-DeepSeekV3RotaryEmbeddingAdapter)       | Callable adapter that wraps DeepseekV3's freqs\_cis-based RoPE.                      |
| [`KimiK25VLConfig`](#nemo_automodel-components-models-kimi_k25_vl-model-KimiK25VLConfig)                                         | Configuration for KimiK25VL model.                                                   |
| [`KimiK25VLForConditionalGeneration`](#nemo_automodel-components-models-kimi_k25_vl-model-KimiK25VLForConditionalGeneration)     | KimiK25VL model with backend-aware DeepseekV3 language model.                        |
| [`KimiK25VLLanguageModelBackend`](#nemo_automodel-components-models-kimi_k25_vl-model-KimiK25VLLanguageModelBackend)             | Backend-aware language model wrapper using DeepseekV3 architecture.                  |
| [`KimiK25VLModel`](#nemo_automodel-components-models-kimi_k25_vl-model-KimiK25VLModel)                                           | KimiK25VL multimodal backbone with a DeepseekV3 text decoder.                        |
| [`KimiK25VLMultiModalProjector`](#nemo_automodel-components-models-kimi_k25_vl-model-KimiK25VLMultiModalProjector)               | Projects vision features to language model dimension using patch merger MLP.         |
| [`Learnable2DInterpPosEmbDividedFixed`](#nemo_automodel-components-models-kimi_k25_vl-model-Learnable2DInterpPosEmbDividedFixed) | Learnable 2D interpolatable position embedding with fixed temporal sincos embedding. |
| [`MoonViT3dConfig`](#nemo_automodel-components-models-kimi_k25_vl-model-MoonViT3dConfig)                                         | Configuration for MoonViT3d vision encoder with temporal support.                    |
| [`MoonViT3dEncoder`](#nemo_automodel-components-models-kimi_k25_vl-model-MoonViT3dEncoder)                                       | MoonViT3d encoder with temporal support.                                             |
| [`MoonViT3dEncoderLayer`](#nemo_automodel-components-models-kimi_k25_vl-model-MoonViT3dEncoderLayer)                             | Single encoder layer for MoonViT3d.                                                  |
| [`MoonViT3dMLP`](#nemo_automodel-components-models-kimi_k25_vl-model-MoonViT3dMLP)                                               | MLP for MoonViT3d.                                                                   |
| [`MoonViT3dPretrainedModel`](#nemo_automodel-components-models-kimi_k25_vl-model-MoonViT3dPretrainedModel)                       | MoonViT3d vision encoder with temporal support.                                      |
| [`MoonVision3dPatchEmbed`](#nemo_automodel-components-models-kimi_k25_vl-model-MoonVision3dPatchEmbed)                           | Patch embedding for MoonViT3d.                                                       |
| [`Rope2DPosEmbRepeated`](#nemo_automodel-components-models-kimi_k25_vl-model-Rope2DPosEmbRepeated)                               | 2D rotary position embedding repeated for temporal dimension.                        |

### Functions

| Name                                                                                                                                     | Description                                                         |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| [`_apply_rope_vision`](#nemo_automodel-components-models-kimi_k25_vl-model-_apply_rope_vision)                                           | Apply rotary position embedding for vision.                         |
| [`_register_kimi_k25_vl_with_transformers`](#nemo_automodel-components-models-kimi_k25_vl-model-_register_kimi_k25_vl_with_transformers) | Register KimiK25VLConfig and model with transformers Auto classes.  |
| [`compute_expanded_seq_length`](#nemo_automodel-components-models-kimi_k25_vl-model-compute_expanded_seq_length)                         | Compute the expanded sequence length after image token insertion.   |
| [`get_1d_sincos_pos_embed`](#nemo_automodel-components-models-kimi_k25_vl-model-get_1d_sincos_pos_embed)                                 | Generate 1D sinusoidal positional embedding for temporal dimension. |
| [`get_1d_sincos_pos_embed_from_grid`](#nemo_automodel-components-models-kimi_k25_vl-model-get_1d_sincos_pos_embed_from_grid)             | Generate 1D sinusoidal positional embedding from grid positions.    |
| [`tpool_patch_merger`](#nemo_automodel-components-models-kimi_k25_vl-model-tpool_patch_merger)                                           | Merge patches with temporal pooling.                                |
| [`vision_attention_flash`](#nemo_automodel-components-models-kimi_k25_vl-model-vision_attention_flash)                                   | Flash attention for vision.                                         |
| [`vision_attention_sdpa`](#nemo_automodel-components-models-kimi_k25_vl-model-vision_attention_sdpa)                                     | SDPA attention for vision.                                          |

### Data

[`FLASH_ATTN_AVAILABLE`](#nemo_automodel-components-models-kimi_k25_vl-model-FLASH_ATTN_AVAILABLE)

[`LOGGER`](#nemo_automodel-components-models-kimi_k25_vl-model-LOGGER)

[`ModelClass`](#nemo_automodel-components-models-kimi_k25_vl-model-ModelClass)

### API

```python
class nemo_automodel.components.models.kimi_k25_vl.model.DeepSeekV3RotaryEmbeddingAdapter(
    parent_module: torch.nn.Module,
    rope_fusion: bool = False
)
```

Callable adapter that wraps DeepseekV3's freqs\_cis-based RoPE.

```python
nemo_automodel.components.models.kimi_k25_vl.model.DeepSeekV3RotaryEmbeddingAdapter.__call__(
    hidden_states: torch.Tensor,
    position_ids: torch.Tensor
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLConfig(
    vision_config: typing.Optional[typing.Union[typing.Dict, nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dConfig]] = None,
    text_config: typing.Optional[typing.Union[typing.Dict, transformers.models.deepseek_v3.configuration_deepseek_v3.DeepseekV3Config]] = None,
    ignore_index: int = -100,
    media_placeholder_token_id: int = 163605,
    pad_token_id: int = 0,
    tie_word_embeddings: bool = False,
    mm_projector_type: str = 'patchmerger',
    mm_hidden_size: typing.Optional[int] = None,
    projector_hidden_act: str = 'gelu',
    projector_ln_eps: float = 1e-05,
    architectures: typing.Optional[typing.List[str]] = None,
    kwargs = {}
)
```

**Bases:** `PretrainedConfig`

Configuration for KimiK25VL model.

Supports both 'kimi\_k25\_vl' and 'kimi\_k25' model types for compatibility
with original checkpoints.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLConfig.to_dict() -> typing.Dict[str, typing.Any]
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration(
    config,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), `Module`, [MoEFSDPSyncMixin](/nemo-automodel/nemo_automodel/components/moe/fsdp_mixin#nemo_automodel-components-moe-fsdp_mixin-MoEFSDPSyncMixin)

KimiK25VL model with backend-aware DeepseekV3 language model.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.forward(
    input_ids = None,
    attention_mask = None,
    position_ids = None,
    past_key_values = None,
    inputs_embeds = None,
    labels = None,
    use_cache = None,
    output_attentions = None,
    output_hidden_states: typing.Optional[bool] = None,
    return_dict = None,
    pixel_values = None,
    grid_thws = None,
    padding_mask = None,
    target_seq_length = None,
    logits_to_keep: typing.Union[int, torch.Tensor] = 0,
    kwargs = {}
)
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.from_config(
    config,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
```

classmethod

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args = (),
    kwargs = {}
)
```

classmethod

Load model from pretrained path.

Creates the model structure. Weights are loaded by DCP which calls the
state\_dict\_adapter.to\_hf() to get checkpoint-format keys (including
*\_packed/*\_scale/\*\_shape for INT4), then from\_hf() to dequantize.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.get_input_embeddings()
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.get_output_embeddings()
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.initialize_weights(
    buffer_device = None,
    dtype = torch.bfloat16
)
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.set_input_embeddings(
    value
)
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLForConditionalGeneration.set_output_embeddings(
    new_embeddings
)
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLLanguageModelBackend(
    config,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None
)
```

**Bases:** `Module`

Backend-aware language model wrapper using DeepseekV3 architecture.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLLanguageModelBackend.forward(
    input_ids = None,
    inputs_embeds = None,
    attention_mask = None,
    position_ids = None,
    padding_mask = None,
    kwargs = {}
)
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLLanguageModelBackend.get_input_embeddings()
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLLanguageModelBackend.init_weights(
    buffer_device = None
)
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLLanguageModelBackend.set_input_embeddings(
    value
)
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLModel(
    config,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
)
```

**Bases:** `Module`

KimiK25VL multimodal backbone with a DeepseekV3 text decoder.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLModel._compute_num_image_tokens_from_grid(
    grid_thws: torch.Tensor
) -> typing.List[int]
```

Pre-compute number of image tokens from grid\_thws without running vision tower.

For 1 image per sample: num\_tokens = (h // merge\_h) \* (w // merge\_w)
With default merge\_kernel\_size=(2,2): num\_tokens = (h // 2) \* (w // 2)

**Parameters:**

Tensor of shape (batch\_size, 3) with \[t, h, w] per sample

**Returns:** `List[int]`

List of expected image token counts per sample

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLModel._extract_image_features(
    pixel_values,
    grid_thws
)
```

Extract and project image features.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLModel._merge_input_ids_with_image_features(
    image_features: typing.List[torch.Tensor],
    inputs_embeds: torch.Tensor,
    input_ids: torch.Tensor,
    attention_mask: torch.Tensor,
    labels: typing.Optional[torch.Tensor] = None,
    target_seq_length: typing.Optional[int] = None
)
```

Merge image features into input embeddings.

Supports two modes:

1. Pre-expanded (PP mode): input\_ids already has N placeholder tokens per image,
   where N = number of image features. Does simple 1:1 replacement.
2. Dynamic expansion: input\_ids has 1 placeholder per image, expands to N tokens.

**Parameters:**

List of image feature tensors, one per image

Text embeddings (batch\_size, seq\_len, embed\_dim)

Token IDs (batch\_size, seq\_len)

Attention mask (batch\_size, seq\_len)

Optional labels for training

Optional fixed output length for pipeline parallelism.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLModel.forward(
    input_ids = None,
    attention_mask = None,
    position_ids = None,
    inputs_embeds = None,
    pixel_values = None,
    grid_thws = None,
    labels = None,
    padding_mask = None,
    target_seq_length = None,
    kwargs = {}
)
```

Forward pass with optional fixed sequence length for pipeline parallelism.

**Parameters:**

If provided, the output after image token expansion will be
padded to this fixed length. Required for PP with varying image sizes.
Can be pre-computed as: max\_text\_len - 1 + max\_image\_tokens
where max\_image\_tokens = (h // 2) \* (w // 2) for each image.

```python
class nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLMultiModalProjector(
    config
)
```

**Bases:** `Module`

Projects vision features to language model dimension using patch merger MLP.

```python
nemo_automodel.components.models.kimi_k25_vl.model.KimiK25VLMultiModalProjector.forward(
    image_features: typing.List[torch.Tensor]
) -> typing.List[torch.Tensor]
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.Learnable2DInterpPosEmbDividedFixed(
    height: int,
    width: int,
    num_frames: int,
    dim: int,
    interpolation_mode: str = 'bicubic'
)
```

**Bases:** `Module`

Learnable 2D interpolatable position embedding with fixed temporal sincos embedding.

```python
nemo_automodel.components.models.kimi_k25_vl.model.Learnable2DInterpPosEmbDividedFixed.forward(
    x: torch.Tensor,
    grid_thws: torch.Tensor
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dConfig(
    patch_size: int = 14,
    init_pos_emb_height: int = 64,
    init_pos_emb_width: int = 64,
    init_pos_emb_time: int = 4,
    pos_emb_type: str = 'divided_fixed',
    num_attention_heads: int = 16,
    num_hidden_layers: int = 27,
    hidden_size: int = 1152,
    intermediate_size: int = 4304,
    merge_kernel_size: typing.Tuple[int, int] = (2, 2),
    video_attn_type: str = 'spatial_temporal',
    merge_type: str = 'sd2_tpool',
    kwargs = {}
)
```

**Bases:** `PretrainedConfig`

Configuration for MoonViT3d vision encoder with temporal support.

```python
class nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dEncoder(
    hidden_dim: int,
    num_layers: int,
    block_cfg: dict
)
```

**Bases:** `Module`

MoonViT3d encoder with temporal support.

```python
nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dEncoder.forward(
    hidden_states: torch.Tensor,
    grid_thws: torch.Tensor
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dEncoderLayer(
    num_heads: int,
    hidden_dim: int,
    mlp_dim: int,
    activation = F.gelu,
    attn_bias: bool = False,
    attn_implementation: str = 'flash_attention_2'
)
```

**Bases:** `Module`

Single encoder layer for MoonViT3d.

```python
nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dEncoderLayer.forward(
    hidden_states: torch.Tensor,
    cu_seqlens: torch.Tensor,
    max_seqlen: int,
    rope_freqs_cis: torch.Tensor
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dMLP(
    dims: typing.List[int],
    activation,
    bias: bool = True
)
```

**Bases:** `Module`

MLP for MoonViT3d.

```python
nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dMLP.forward(
    x: torch.Tensor
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dPretrainedModel(
    config
)
```

**Bases:** `Module`

MoonViT3d vision encoder with temporal support.

```python
nemo_automodel.components.models.kimi_k25_vl.model.MoonViT3dPretrainedModel.forward(
    pixel_values: torch.Tensor,
    grid_thws: torch.Tensor
) -> typing.List[torch.Tensor]
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.MoonVision3dPatchEmbed(
    out_dim: int,
    in_dim: int = 3,
    patch_size: int = 14,
    pos_emb_height: int = 64,
    pos_emb_width: int = 64,
    pos_emb_time: int = 4
)
```

**Bases:** `Module`

Patch embedding for MoonViT3d.

```python
nemo_automodel.components.models.kimi_k25_vl.model.MoonVision3dPatchEmbed.forward(
    x: torch.Tensor,
    grid_thws: torch.Tensor
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.kimi_k25_vl.model.Rope2DPosEmbRepeated(
    dim: int,
    max_height: int,
    max_width: int,
    theta_base: float = 10000
)
```

**Bases:** `Module`

2D rotary position embedding repeated for temporal dimension.

```python
nemo_automodel.components.models.kimi_k25_vl.model.Rope2DPosEmbRepeated._precompute_freqs_cis(
    device: torch.device
) -> torch.Tensor
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.Rope2DPosEmbRepeated.get_freqs_cis(
    grid_thws: torch.Tensor,
    device: torch.device
) -> torch.Tensor
```

```python
nemo_automodel.components.models.kimi_k25_vl.model._apply_rope_vision(
    xq: torch.Tensor,
    xk: torch.Tensor,
    freqs_cis: torch.Tensor
) -> typing.Tuple[torch.Tensor, torch.Tensor]
```

Apply rotary position embedding for vision.

```python
nemo_automodel.components.models.kimi_k25_vl.model._register_kimi_k25_vl_with_transformers()
```

Register KimiK25VLConfig and model with transformers Auto classes.

```python
nemo_automodel.components.models.kimi_k25_vl.model.compute_expanded_seq_length(
    text_seq_length: int,
    grid_thws: torch.Tensor,
    merge_kernel_size: typing.Tuple[int, int] = (2, 2),
    num_images: int = 1
) -> int
```

Compute the expanded sequence length after image token insertion.

For pipeline parallelism, this can be used to pre-compute the target\_seq\_length
parameter needed for fixed-shape outputs.

**Parameters:**

Original text sequence length (including 1 placeholder per image)

Tensor of shape (num\_images, 3) with \[t, h, w] per image

Vision tower's patch merge kernel size, default (2, 2)

Number of images (placeholders) in the sequence

**Returns:** `int`

Expected sequence length after image features are inserted

```python
nemo_automodel.components.models.kimi_k25_vl.model.get_1d_sincos_pos_embed(
    embed_dim: int,
    t_size: int,
    cls_token: bool = False
) -> numpy.ndarray
```

Generate 1D sinusoidal positional embedding for temporal dimension.

```python
nemo_automodel.components.models.kimi_k25_vl.model.get_1d_sincos_pos_embed_from_grid(
    embed_dim: int,
    pos: numpy.ndarray
) -> numpy.ndarray
```

Generate 1D sinusoidal positional embedding from grid positions.

```python
nemo_automodel.components.models.kimi_k25_vl.model.tpool_patch_merger(
    x: torch.Tensor,
    grid_thws: torch.Tensor,
    merge_kernel_size: typing.List[int] | None = None
) -> typing.List[torch.Tensor]
```

Merge patches with temporal pooling.

```python
nemo_automodel.components.models.kimi_k25_vl.model.vision_attention_flash(
    q,
    k,
    v,
    q_cu_seqlens,
    k_cu_seqlens,
    max_seqlen_q = None,
    max_seqlen_k = None
)
```

Flash attention for vision.

```python
nemo_automodel.components.models.kimi_k25_vl.model.vision_attention_sdpa(
    q,
    k,
    v,
    q_cu_seqlens,
    k_cu_seqlens,
    kwargs = {}
)
```

SDPA attention for vision.

```python
nemo_automodel.components.models.kimi_k25_vl.model.FLASH_ATTN_AVAILABLE = True
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.LOGGER = logging.getLogger(__name__)
```

```python
nemo_automodel.components.models.kimi_k25_vl.model.ModelClass = KimiK25VLForConditionalGeneration
```