nemo_automodel.components.models.inkling2.multimodal

Image/text entry point sharing Inkling2’s native decoder and checkpoint loader.

Module Contents

Classes

Name	Description
`Inkling2ForConditionalGeneration`	Insert unnormalized image features after the text embedding normalization.

API

class nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration(
    config: nemo_automodel.components.models.inkling2.config.Inkling2Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    revision: str | None = None
)

Bases: Inkling2ForCausalLM

Insert unnormalized image features after the text embedding normalization.

_vlm_chunk_idx

int | None = None

_vlm_image_grid_hws_chunks

list[Tensor] | None = None

_vlm_pixel_values_chunks

list[Tensor] | None = None

state_dict_adapter

= Inkling2StateDictAdapter(config)

visual

nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration.customize_pipeline_stage_modules(
    module_names_per_stage: list[list[str]],
    layers_prefix: str,
    text_model: torch.nn.Module | None = None
) -> list[list[str]]

Keep the vision encoder and projector exclusively on the first stage.

Parameters:

module_names_per_stage

list[list[str]]

Generated stage FQNs; caller lists are unchanged.

layers_prefix

str

Empty for the native model’s top-level decoder layers.

text_model

nn.Module | NoneDefaults to None

Backbone resolved by the shared splitter.

Returns: list[list[str]]

Copied stage FQNs with one owner for vision, embedding normalization

nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration.forward(
    input_ids: torch.Tensor,
    pipeline_tensors: torch.Tensor = (),
    inputs_embeds: torch.Tensor | None = None,
    vision_patches_bthwc: torch.Tensor | None = None,
    vision_grid_thw: torch.Tensor | None = None,
    pixel_values: torch.Tensor | None = None,
    image_grid_thw: torch.Tensor | None = None,
    mm_token_type_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    output_hidden_states: bool = False
) -> torch.Tensor | tuple[torch.Tensor, ...] | transformers.modeling_outputs.CausalLMOutputWithPast

Fuse images into unpacked image/text sequences.

Parameters:

input_ids

torch.Tensor

Integer [batch, sequence] tokens, including image markers and bodies.

pipeline_tensors

torch.TensorDefaults to ()

Non-first-stage int64 [batch, sequence, 2] layout, paired with residual history [batch, sequence, sources, hidden] in input_ids. Later stages use only these transported inputs.

inputs_embeds