NeVA

Emerging from the roots of LLaVA (Large Language and Vision Assistant) [MM-MODELS2], NeVA stands as a pioneering model in the NeMo Multimodal ecosystem. It adeptly fuses large language-centric models, such as NVGPT or LLaMA, with a vision encoder. The training utilizes machine-generated multimodal language-image instruction-following data. Remarkably, even with a limited dataset, NeVA exhibits profound capabilities in deciphering images and adeptly answering queries about them. Its prowess is especially evident in tasks requiring intricate visual comprehension and instruction-following. Intriguingly, NeVA mirrors the capabilities of advanced multimodal models like GPT-4, even when faced with novel images and instructions.

Building upon LLaVA’s foundational principles, NeVA amplifies its training efficiency by harnessing the NeMo LLM framework’s features, including model parallelism, activation checkpointing, AMP O2, Flash Attention, and more.

Main Language Model

The original LLaVA model incorporates the LLaMA architecture, renowned for its prowess in open-source, language-only instruction-tuning endeavors. LLaMA refines textual input through a process of tokenization and embedding. To these token embeddings, positional embeddings are integrated, and the combined representation is channeled through multiple transformer layers. The output from the concluding transformer layer, associated with the primary token, is designated as the text representation.

In NeMo, the text encoder is anchored in the MegatronGPTModel class. This class is versatile, supporting not only NVGPT models but also LLaMA, LLaMA-2 and other community models, complete with a checkpoint conversion script. Concurrently, the vision model and projection layers enhance the primary language model’s word embedding component. For a comprehensive understanding of the implementation, one can refer to the MegatronNevaModel class.

Vision Model

For visual interpretation, NeVA harnesses the power of the pre-trained CLIP visual encoder, ViT-L/14, recognized for its visual comprehension acumen. Images are first partitioned into standardized patches, for instance, 16x16 pixels. These patches are linearly embedded, forming a flattened vector that subsequently feeds into the transformer. The culmination of the transformer’s processing is a unified image representation. In the NeMo framework, the NeVA vision model, anchored on the CLIP visual encoder ViT-L/14, can either be instantiated via the CLIPVisionTransformer class or initiated through the transformers package from Hugging Face.

Projection and Integration

The encoder retrieves visual features from images and intertwines them with language embeddings using a modifiable projection matrix. This intricate projection translates visual cues into language embedding tokens, seamlessly merging text and imagery. LLaVA-1.5 [MM-MODELS1] introduces two pivotal enhancements. The integration of an MLP vision-language connector amplifies the system’s prowess. Building on the triumphs of MLPs in self-supervised learning, LLaVA-1.5 undergoes a transformative design shift. Transitioning from a linear to a dual-layer MLP projection markedly bolsters LLaVA-1.5’s multimodal faculties, empowering the model to adeptly navigate and synergize language and visual elements.

Multimodal Configuration

            

            
mm_cfg:
  use_im_start_end: False
        

• use_im_start_end: If set to True, image start and end tokens will be used before and after image embeddings.

Language Model Configuration within Multimodal

            

            
mm_cfg:
  llm:
    from_pretrained: ${data_dir}/neva/checkpoints/llama-2-13b-chat-tp8.nemo
    freeze: False
    model_type: llama_2
        

• from_pretrained: Path to the pretrained NeMo language model checkpoint.

• freeze: If set to True, the model parameters will not be updated during training.

• model_type: Specifies the type of model, either nvgpt or llama_2.

Vision Encoder Configuration within Multimodal

            

            
mm_cfg:
  vision_encoder:
    from_pretrained: "openai/clip-vit-large-patch14"
    from_hf: True
    patch_dim: 14
    hidden_size: 1024
    vision_select_layer: -2
    class_token_length: 1
    freeze: True
        

• from_pretrained: Path or name of the pretrained vision encoder.

• from_hf: If set to True, the model will be loaded from the Hugging Face model hub.

• patch_dim: Size of the patches the image is divided into.

• hidden_size: Dimensionality of the hidden layers.

• vision_select_layer: Specifies which layer to select from the vision model.

• class_token_length: Length of the classification token.

Main Language Model Configuration

            

            
mcore_gpt: False
encoder_seq_length: 4096
position_embedding_type: rope
num_layers: 40
hidden_size: 5120
ffn_hidden_size: 13824
num_attention_heads: 40
hidden_dropout: 0.0
attention_dropout: 0.0
ffn_dropout: 0.0
normalization: rmsnorm
bias: False
activation: 'fast-swiglu'
        

• mcore_gpt: If set to True, the GPTModel from megatron.core will be used.

• encoder_seq_length: Sequence length for the main language model encoder.

• position_embedding_type: Type of position embedding used.

• num_layers, hidden_size, ffn_hidden_size, num_attention_heads: Parameters defining the architecture of the main language model. The ffn_hidden_size is typically 4 times the hidden_size.

• hidden_dropout, attention_dropout, ffn_dropout: Dropout probabilities for the hidden state, attention, and feed-forward layers in the transformer respectively.

• normalization: Type of normalization layers used.

• bias: If set to True, bias terms will be used in all weight matrices.

• activation: Activation function used in the model.

Optimizations

NeVA’s training encompasses two essential stages that enhance its capacity to comprehend user instructions, understand both language and visual content, and generate accurate responses:

1. Pre-training for Feature Alignment: In this initial stage, NeVA aligns visual and language features to ensure compatibility.

2. Fine-tuning End-to-End: The second training stage focuses on fine-tuning the entire model, end-to-end. While the visual encoder’s weights remain unchanged, both the projection layer’s pre-trained weights and the LLM’s parameters become subjects of adaptation. This fine-tuning can be tailored to different application scenarios, yielding versatile capabilities.

[MM-MODELS1]

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023.

[MM-MODELS2]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023. arXiv:arXiv:2304.08485.