> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/llama_nemotron_vl/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # nemo_automodel.components.models.llama_nemotron_vl Llama Nemotron VL model for multimodal embedding and retrieval tasks. ## Submodules * **[`nemo_automodel.components.models.llama_nemotron_vl.model`](/nemo-automodel/nemo_automodel/components/models/llama_nemotron_vl/model)** * **[`nemo_automodel.components.models.llama_nemotron_vl.processor`](/nemo-automodel/nemo_automodel/components/models/llama_nemotron_vl/processor)** ## Package Contents ### Classes | Name | Description | | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | [`LlamaNemotronVLConfig`](#nemo_automodel-components-models-llama_nemotron_vl-model-LlamaNemotronVLConfig) | Base configuration for vision-language models combining vision and language components. | | [`LlamaNemotronVLModel`](#nemo_automodel-components-models-llama_nemotron_vl-model-LlamaNemotronVLModel) | LlamaNemotron VL model for vision-language reranking. | | [`LlamaNemotronVLProcessor`](#nemo_automodel-components-models-llama_nemotron_vl-processor-LlamaNemotronVLProcessor) | Processor for LlamaNemotronVL model. | ### API ```python class nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLConfig( vision_config = None, llm_config = None, use_backbone_lora = 0, use_llm_lora = 0, select_layer = -1, force_image_size = None, downsample_ratio = 0.5, template = None, dynamic_image_size = False, use_thumbnail = False, min_dynamic_patch = 1, max_dynamic_patch = 6, mlp_checkpoint = True, pre_feature_reduction = False, keep_aspect_ratio = False, vocab_size = -1, q_max_length: int | None = 512, p_max_length: int | None = 10240, query_prefix: str = 'query:', passage_prefix: str = 'passage:', pooling: str = 'last', bidirectional_attention: bool = False, max_input_tiles: int = 2, img_context_token_id: int = 128258, kwargs = {} ) ``` **Bases:** `PretrainedConfig` Base configuration for vision-language models combining vision and language components. This serves as the foundation for LlamaNemotronVL configurations. **`llm_config`** `= LlamaBidirectionalConfig(**llm_config)` --- **`model_type`** `= 'llama_nemotron_vl'` --- **`sub_configs`** --- **`vision_config`** `= SiglipVisionConfig(**vision_config)` --- **`vocab_size`** `= self.llm_config.vocab_size` --- ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLConfig.get_text_config( decoder: bool | None = None, encoder: bool | None = None ) -> transformers.configuration_utils.PretrainedConfig ``` Resolve the language config through the Transformers composite contract. **Parameters:** **`decoder`** `bool | None` — default: None Select the text-output configuration when true. --- **`encoder`** `bool | None` — default: None Select the text-input configuration when true. --- **Returns:** `PretrainedConfig` The language decoder config for decoder or unrestricted requests; ```python class nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel( config: nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLConfig, vision_model: transformers.PreTrainedModel | None = None, language_model: transformers.PreTrainedModel | None = None ) ``` **Bases:** `PreTrainedModel` LlamaNemotron VL model for vision-language reranking. Combines a vision encoder (SigLIP) with a bidirectional language model (LLaMA) for cross-modal reranking tasks. **`_no_split_modules`** `= ['LlamaDecoderLayer']` --- **`downsample_ratio`** `= config.downsample_ratio` --- **`main_input_name`** `= 'pixel_values'` --- **`mlp1`** --- **`num_image_token`** `= int((grid_size * config.downsample_ratio) ** 2)` --- **`patch_size`** `= 14` --- **`processor`** --- **`select_layer`** `= config.select_layer` --- **`template`** `= config.template` --- ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel._embed_batch( inputs: typing.Dict[str, typing.Any], pool_type: str | None = None ) ``` Encodes the inputs into a tensor of embeddings. Args: inputs: A dictionary of inputs to the model. You can prepare the inputs using the processor.process\_queries and processor.process\_documents methods. pool\_type: The type of pooling to use. If None, the pooling type is set to the pooling type configured in the model. Returns: A tensor of embeddings. ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.build_collator( processor = None, kwargs = {} ) ``` ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.encode_documents( images: typing.List[typing.Any] | None = None, texts: typing.List[str] | None = None, kwargs = {} ) ``` Encodes the input document images and texts into a tensor of embeddings. Args: images: A list of PIL.Image of document pages images. texts: A list of document page texts. Returns: A tensor of embeddings. ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.encode_queries( queries: typing.List[str], kwargs = {} ) ``` Encodes the input queries into a tensor of embeddings. Args: queries: A list of queries. Returns: A tensor of embeddings. ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.extract_feature( pixel_values ) ``` Extract and project vision features to language model space. ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.forward( pixel_values: torch.FloatTensor = None, input_ids: torch.LongTensor = None, attention_mask: torch.Tensor | None = None, position_ids: torch.LongTensor | None = None, image_flags: torch.LongTensor | None = None, past_key_values: typing.List[torch.FloatTensor] | None = None, labels: torch.LongTensor | None = None, use_cache: bool | None = None, output_attentions: bool | None = None, output_hidden_states: bool | None = None, return_dict: bool | None = None, num_patches_list: typing.List[torch.Tensor] | None = None, run_dummy_vision: bool | None = None ) -> typing.Union[typing.Tuple, transformers.modeling_outputs.CausalLMOutputWithPast] ``` ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.get_decoder() -> transformers.PreTrainedModel ``` Return the language tower used as the retrieval text decoder. ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.get_input_embeddings() ``` ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.get_output_embeddings() ``` ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.pixel_shuffle( x, scale_factor = 0.5 ) ``` ```python nemo_automodel.components.models.llama_nemotron_vl.model.LlamaNemotronVLModel.post_loss( loss, inputs ) ``` ```python class nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessor( tokenizer: typing.Any, config: nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessorConfig | None = None, q_max_length: int | None = None, p_max_length: int | None = None, pad_to_multiple_of: int | None = None, query_prefix: str = 'query:', passage_prefix: str = 'passage:', max_input_tiles: int = 6, num_image_token: int = 256, dynamic_image_size: bool = True, image_size: int = 512, use_thumbnail: bool = True, template: str = 'bidirectional-llama-retrie..., num_channels: int = 3, norm_type: str = 'siglip', system_message: str = '', padding: typing.Union[bool, str] = True, kwargs = {} ) ``` **Bases:** `ProcessorMixin` Processor for LlamaNemotronVL model. **`attributes`** `= ['tokenizer']` --- **`image_processor`** --- **`tokenizer_class`** `= 'AutoTokenizer'` --- ```python nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessor.__call__( text: typing.List[str] | None = None, images: typing.List[typing.Any] | None = None, text_kwargs: typing.Dict[str, typing.Any] | None = None, images_kwargs: typing.Dict[str, typing.Any] | None = None, common_kwargs: typing.Dict[str, typing.Any] | None = None, kwargs = {} ) -> typing.Dict[str, typing.Any] ``` Process text and/or image inputs into model-ready features. This method provides compatibility with the standard HuggingFace processor interface used by Sentence Transformers. For image inputs, it delegates to process\_documents. For text-only inputs, it tokenizes directly (assuming any task prefix has already been applied by the caller). Args: text: List of text strings. For text-only inputs, these should already include any task prefix (e.g. "query: " or "passage: "). images: List of PIL Images for document encoding. text\_kwargs: Keyword arguments for text processing (e.g. padding, truncation). images\_kwargs: Keyword arguments for image processing (unused, for API compat). common\_kwargs: Common keyword arguments (e.g. return\_tensors). \*\*kwargs: Additional keyword arguments (ignored). Returns: Dict with "input\_ids", "attention\_mask", and optionally "pixel\_values". ```python nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessor.add_dummy_labels( questions, merged_batch_dict ) ``` ```python nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessor.merge_batch_dict( query_batch_dict, doc_batch_dict ) ``` ```python nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessor.process_documents( documents: typing.Union[typing.Dict, typing.List[typing.Dict]], return_tensors: typing.Literal['pt', 'np'] = 'pt', padding: bool | str | None = None, truncation: bool = True, pixel_values_layout: typing.Literal['per_image', 'flat_tiles'] = 'flat_tiles', kwargs = {} ) -> typing.Dict[str, typing.Any] ``` Process documents into model inputs with tokenized text and pixel values. Args: documents: Either a dict with "images" and "texts" lists, or a list of dicts each with "image" and "text" keys. Images can be PIL Images, file paths, or None/empty string for text-only documents. return\_tensors: Output format — "pt" for PyTorch tensors, "np" for numpy arrays. padding: Padding strategy passed to the tokenizer. Defaults to the value set in the processor constructor. truncation: Whether to truncate sequences to p\_max\_length. pixel\_values\_layout: How to structure the pixel values output: * "flat\_tiles": All image tiles concatenated into a single tensor of shape (total\_tiles, C, H, W). Different images may contribute different numbers of tiles. None if no images are present. This is the format expected by the model's forward() method. * "per\_image": A list aligned with the input documents, where each entry is either a tensor of shape (num\_tiles, C, H, W) or None. Returns: Dict with "input\_ids", "attention\_mask", and "pixel\_values". ```python nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessor.process_queries( queries: typing.List[str], return_tensors: typing.Literal['pt', 'np'] = 'pt', padding: bool | str | None = None, truncation: bool = True, kwargs = {} ) -> transformers.BatchEncoding ``` Process queries into model inputs with tokenized text. Args: queries: List of query strings. return\_tensors: Output format — "pt" for PyTorch tensors, "np" for numpy arrays. padding: Padding strategy passed to the tokenizer. Defaults to the value set in the processor constructor. truncation: Whether to truncate sequences to q\_max\_length. Returns: Dict with "input\_ids" and "attention\_mask". ```python nemo_automodel.components.models.llama_nemotron_vl.processor.LlamaNemotronVLProcessor.process_queries_documents_biencoder( features: typing.Dict, kwargs = {} ) -> typing.Dict[str, typing.Any] ``` (Pdb) features \[\{'image': \[\, \, \], 'text': \['passage: ', 'passage: ', 'passage: '], 'question': "query: What change did Carl Rey suggest for the Strategic Plan's website objective deadline?"}, \{'image': \[\, \, \], 'text': \['passage: ', 'passage: ', 'passage: '], 'question': 'query: What are the name and TIN requirements for individuals with real estate transactions?'}, \{'image': \[\, \, \], 'text': \['passage: ', 'passage: ', 'passage: '], 'question': 'query: How does Richard Hooker view human inclinations?'}] ## Docs - [nemo_automodel.components.models.llama_nemotron_vl.model](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/llama_nemotron_vl/model.md) - [nemo_automodel.components.models.llama_nemotron_vl.processor](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/llama_nemotron_vl/processor.md)