Configure the VLM#

VSS is designed to be configurable with many VLMs, such as:

3rd-party Endpoints
Community Models

VSS supports integrating custom VLM models. Depending on the model to be integrated, some configurations must be updated or the interface code is implemented. The model can ONLY be selected at initialization time.

The following segments explain those approaches in details.

Third Party VLM Endpoints#

You have the option to utilize externally hosted third party VLMs which follow the OpenAI API standard. Access to these endpoints are provided through the third party.

Supported Model	Developer
GPT4o	OpenAI

OpenAI (GPT-4o)#

To use GPT-4o as the VLM model in VSS, refer to VLM Model to Use and modify the env variable VLM_MODEL_TO_USE in the .env file.

Pre-Requisite: API key from https://platform.openai.com/api-keys

Export the following variables in the .env file.

export VLM_MODEL_TO_USE=openai-compat
export OPENAI_API_KEY=<your-openai-api-key>

Community Models#

We support multiple community models that are open source, developed through research, or offered by third parties. If the VLM model provides an OpenAI compatible REST API, refer to Configuration Options. Here is a list of models tested within VSS and steps:

Supported Model	Developer	Size (Parameters)
Cosmos-Reason1	NVIDIA	7b
Qwen2.5-VL-7B-Instruct / Other Qwen2.5 VLM models	Alibaba Cloud	7b
NEVA	NVIDIA	22b
Fuyu	NVIDIA	8b

Note

Qwen2.5 VL based models are supported as drop-in replacements for Cosmos-Reason1 since Cosmos-Reason1 is based on Qwen2.5 VL. Use VLM_MODEL_TO_USE=cosmos-reason1 for Qwen2.5 VL based models as well.

Auto-Download Models (Cosmos-Reason1)#

Set the following env variables in the .env file before starting the VSS container:

Cosmos-Reason1 7b

export VLM_MODEL_TO_USE=cosmos-reason1
export MODEL_PATH="ngc:nim/nvidia/cosmos-reason1-7b:1.1-fp8-dynamic"

Local Models (Cosmos-Reason1)#

Follow the steps below to use VLM weights that have been downloaded to a local filepath. This can be used as an alternative way to deploy the Cosmos-Reason1 model.

Install the NGC CLI & git-lfs which will be able to download the models to a specified location.

# Download NGC CLI

# For x86
wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/3.169.4/files/ngccli_linux.zip -O ngccli_linux.zip && unzip ngccli_linux.zip

# For arm64
wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/3.169.4/files/ngccli_arm64.zip -O ngccli_arm64.zip && unzip ngccli_arm64.zip

chmod u+x ngc-cli/ngc
export PATH="$PATH:$(pwd)/ngc-cli"

# Install git-lfs
sudo apt install git-lfs

Download the model weights you wish to store locally.

Cosmos-Reason1 7b

# Download the Cosmos-Reason1 weights
export NGC_API_KEY=<your-legacy-api-key>
export NGC_CLI_ORG=nim
export NGC_CLI_TEAM=nvidia
ngc registry model download-version "nim/nvidia/cosmos-reason1-7b:1.1-fp8-dynamic"
chmod a+w cosmos-reason1-7b_v1.1-fp8-dynamic

Cosmos-Reason1 7b FP16 (Hugging Face)

# Download the Cosmos-Reason1 weights
git clone https://huggingface.co/nvidia/Cosmos-Reason1-7B
chmod a+w Cosmos-Reason1-7B

To deploy VSS with a locally downloaded Cosmos-Reason1 checkpoint, set the following env variables in the .env file:
```
export VLM_MODEL_TO_USE=cosmos-reason1
export MODEL_PATH=</path/to/local/cosmos-reason1-checkpoint>
export MODEL_ROOT_DIR=<MODEL_ROOT_DIR_ON_HOST>
```
Note

Make sure </path/to/local/cosmos-reason1-checkpoint> is a directory under <MODEL_ROOT_DIR_ON_HOST>. <MODEL_ROOT_DIR_ON_HOST> is a parent directory on the host machine for all the models.

OpenAI Compatible REST API#

If the VLM model provides an OpenAI compatible REST API, set the following env variables in the .env file:

export VLM_MODEL_TO_USE=openai-compat
export OPENAI_API_KEY=<openai-api-key> # Optional. Can be set if using OpenAI endpoint
export VIA_VLM_API_KEY=<>  # Optional. Can be set if using a custom endpoint
export VIA_VLM_OPENAI_MODEL_DEPLOYMENT_NAME=<>
export VIA_VLM_ENDPOINT=<>
export OPENAI_API_VERSION=<>  # Optional

For more details on the above environment variables, refer to VLM Configuration.

vLLM Served OpenAI API Compatible VLM#

VSS supports dropping in VLMs that are OpenAI API compatible.

The below example shows how to drop in a VLM served through vLLM, a popular high-throughput and memory-efficient inference and serving engine. Many community models on HuggingFace can be served through vLLM.

Download the model, run vLLM serve, and test the local endpoint.

Example steps to download + serve “Qwen/Qwen2.5-VL-7B-Instruct” below:
1. Instructions to install vLLM can be found here: QwenLM/Qwen2.5-VL
  
  More details available at QwenLM/ and https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct.
2. Serve the model using vLLM:
```
vllm serve "Qwen/Qwen2.5-VL-7B-Instruct" --port 38011 --host 0.0.0.0 --dtype bfloat16 --limit-mm-per-prompt image=10,video=1 --served-model-name Qwen2.5VL-7B-instruct
```
  Note
  
  vLLM serve errors were observed for Qwen model with latest transformers version (July 2025). Follow issue and workaround at: vllm-project/vllm-ascend#1470. Workaround: pip install "transformers<4.53.0"

Set the following env variables in the .env file:

export VLM_MODEL_TO_USE=openai-compat
export OPENAI_API_KEY="empty" #random value; unused
export VIA_VLM_ENDPOINT="http://<host-IP>:38011/v1" #match vllm --port and the host-IP
export VIA_VLM_OPENAI_MODEL_DEPLOYMENT_NAME="Qwen2.5VL-7B-instruct" #match vllm --served-model-name

For more details on the above environment variables, refer to VLM Configuration.

Install the Helm Chart.

Other Custom Models#

VSS allows you to drop in your own models to the model directory by providing the pre-trained weight of the model or a model with REST API endpoint and implementing an interface to bridge to the VSS pipeline.

The interface includes an inference.py file and a manifest.yaml.

The manifest.yaml file is used to describe the configuration of the model. An example is shown below:

input:
  number_of_frames: 1 # Number of frames to sample from each chunk.
  jpeg_encoded: false # Whether to encode the frames in JPEG format or pass as raw frame torch tensors.

The inference.py file is used to define the interface for the model. An example is shown below:

class Inference:
  def __init__(self):
     # Load and initialize the model.
     pass

  def get_embeddings(self, tensor:torch.tensor) -> tensor:torch.tensor:
     # Generate video embeddings for the chunk / file.
     # Do not implement if explicit video embeddings are not supported by model
     return tensor

  def generate(self, prompt:str, input:torch.tensor | list[np.ndarray], configs:Dict):
     # Generate summary string from the input prompt and frame/embedding input.
     # configs contains VLM generation parameters like
     # max_new_tokens, seed, top_p, top_k, temperature
     return summary

Based on the chunk size selected during summarize API call, equidistant number_of_frames of frames will be sampled from each chunk and passed to the generate method.

The generate method will be called for each chunk. It will be passed the frames sampled for that chunk along with the text prompt and generation parameters that is, seed, top_k, top_p, and temperature if set by the user as part of the configs dictionary.

When jpeg_encoded parameter in manifest.yaml is set to true, the frames will be passed as a list of NumPy arrays containing encoded jpeg bytes. When this parameter is false or unset, the frames will be passed as a list of torch tensors in RGB HWC format.

The optional get_embeddings method is used to generate embeddings for a given set of frames wrapped in a TCHW tensor and must be removed if the model doesn’t support the feature.

The generate method can be used to implement inference using models that are executed locally on the system or use remote models with REST APIs.

Some examples are available at NVIDIA-AI-Blueprints/video-search-and-summarization

Examples include models fuyu8b and NeVA.

The VSS container image or the Blueprint Helm Chart might need to be modified to use custom VLMs.

Example:

For fuyu8b, you can export the following env variables. For fuyu8b, model weights need to be downloaded, refer to the Fuyu8b README for more details.

export VLM_MODEL_TO_USE=custom
export MODEL_PATH="</path/to/directory/with/inference.py>"
export MODEL_ROOT_DIR=<MODEL_ROOT_DIR_ON_HOST>

After model weights are downloaded using the Fuyu8b README, verify that the directory structure looks like:

ls /path/to/fuyu8b

inference.py                  fuyu8b                          model-00002-of-00002.safetensors  skateboard.png
architecture.png              generation_config.json          model.safetensors.index.json      special_tokens_map.json
bus.png                       added_tokens.json                    preprocessor_config.json          tokenizer_config.json
chart.png                     manifest.yaml                   __pycache__                       tokenizer.json
config.json                   model-00001-of-00002.safetensors README.md                        tokenizer.model

For NeVA, you can export the following env variables. For NVIDIA_API_KEY, refer to Using NIMs from build.nvidia.com.

export NVIDIA_API_KEY=<nvidia-api-key>
export VLM_MODEL_TO_USE=custom
export MODEL_PATH="</path/to/directory/with/inference.py>"
export MODEL_ROOT_DIR=<MODEL_ROOT_DIR_ON_HOST>

Directory structure for NeVA looks like:

ls /path/to/neva

inference.py                  manifest.yaml

Note

Make sure </path/to/directory/with/inference.py> is a directory under <MODEL_ROOT_DIR_ON_HOST>. <MODEL_ROOT_DIR_ON_HOST> is a parent directory on the host machine for all the models.