Is this page helpful?

DeepStream Reference Application - deepstream-vllm-plugin#

The nvvllmvlm plugin is a GStreamer plugin for NVIDIA DeepStream that integrates Vision-Language Models (VLM) using vLLM for real-time video understanding and analysis. It enables the integration of vision-language models (Cosmos Reason 2, etc.) into DeepStream pipelines, processing video frames in configurable time segments and performing batch VLM inference asynchronously.

The plugin source and reference application are available in the DeepStream mono-repo under src/apps/reference_apps/deepstream-vllm-plugin.

Key Features#

  • Segment-based processing: Collects frames into time windows for batch inference.

  • Flexible frame sampling: FPS-based or interval-based subsampling.

  • Async inference: Background worker thread for non-blocking processing.

  • Multi-stream support: One plugin instance handles multiple streams efficiently.

  • Per-stream prompts: Different prompts and settings for each stream.

  • Configurable models: Supports video-native and image-only VLM models.

  • GPU-optimized: Zero-copy GPU operations, with the model shared across streams.

  • Flexible input formats: PyTorch tensors, PIL Images, or numpy arrays.

Requirements#

  • NVIDIA GPU with CUDA support

  • NVIDIA DeepStream SDK 9.0.0 or later

  • Docker with NVIDIA Container Runtime

  • Python 3.12 or later

  • vLLM 0.21.0 with PyTorch 2.11.0 (installed by install.sh)

  • 40 GB or more of GPU memory

  • Currently supported on x86-based GPU platforms

Quick Start#

Installation#

Launch the DeepStream Triton container and install the plugin dependencies:

# Launch the DeepStream container
sudo docker run -it --rm --runtime=nvidia --gpus all --network=host \
  -v $(pwd):/home/vllm_ds_plugin \
  nvcr.io/nvidia/deepstream:9.1-triton-multiarch

# Inside the DeepStream container, install the dependencies
cd /home/vllm_ds_plugin/deepstream-vllm-plugin
chmod +x install.sh
./install.sh

Obtain a Hugging Face token#

  1. Log in at https://huggingface.co.

  2. Go to Profile → Access Tokens.

  3. Create and save the generated token.

  4. To use the Cosmos Reason 2 model, go to https://huggingface.co/nvidia/Cosmos-Reason2-8B, review and accept the NVIDIA Open Model License Agreement.

Single-stream and multi-stream processing#

Run the following inside the DeepStream container:

# Select the GPU (optional)
export CUDA_VISIBLE_DEVICES=0

# Export the Hugging Face token to download models from HF
export HF_TOKEN=<Your HF Token>

# Single stream (dry-run: results printed to the console)
python3 vllm_ds_app_kafka_publish.py <video1.mp4 or RTSP stream url> --dry-run

# For example:
python3 vllm_ds_app_kafka_publish.py /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 --dry-run

# Multi-stream with a shared model (dry-run)
python3 vllm_ds_app_kafka_publish.py <video1 url> <video2 url> --dry-run

Kafka integration#

To stream results to Kafka in real time, bring up the Kafka containers on the host (outside the DeepStream container):

docker compose -f docker-compose-kafka.yml up -d

Then run the publishing application and consumer inside the DeepStream container:

# Run with Kafka publishing (single- or multi-stream)
python3 vllm_ds_app_kafka_publish.py <video1.mp4> <video2.mp4> \
    --kafka-bootstrap localhost:9092 --topic vlm-results

# On another terminal, start the consumer test script
python3 test_consumer.py --topic vlm-results

Configuration#

Place config.yaml in the plugin directory or the current working directory. A complete example follows:

# Model Configuration
model:
  path: "nvidia/Cosmos-Reason2-8B"
  max_model_len: 20480          # Max context length
  gpu_memory_utilization: 0.7   # GPU memory fraction to use; update per platform/available GPU memory
  trust_remote_code: true
  gpu_id: 0                     # GPU device ID (-1 for auto)
  video_mode: 1                 # 1=video metadata (native video support), 0=multi-image mode
  tensor_format: "pytorch"      # pytorch/pil/numpy (image inputs)

# Segment Processing
segment:
  length_sec: 30                # Segment length in seconds
  overlap_sec: 0                # Segment overlap in seconds
  subsample_interval: 1         # Keep every Nth frame
  selection_fps: 30             # Target FPS (0 to disable FPS-based sampling)

# Inference Configuration
inference:
  # User prompt with optional placeholders: {num_frames}, {stream_id}, {timestamps}
  user_prompt: "These are {num_frames} images from stream {stream_id} sampled at timestamps {timestamps}. Describe the scene in detail."
  system_prompt: |
    Provide captions with timestamps using format:
    <start time> <end time> caption of event.
  max_tokens: 2048              # Max tokens to generate
  temperature: 0.7              # Sampling temperature (0-1)
  # top_p: 0.9                  # Top-p nucleus sampling
  # top_k: 100                  # Top-k sampling
  # repetition_penalty: 1.1     # Repetition penalty

  # Per-stream prompt overrides (multi-stream mode)
  stream_prompts:
    0:
      user_prompt: "Stream {stream_id} at {timestamps}: Monitor for security threats."
      system_prompt: "You are a security analyst."
    1:
      user_prompt: "Analyze traffic flow."

# Pipeline Configuration
pipeline:
  queue_maxsize: 20             # Max inference queue size
  max_wait_timeout: 300         # Max wait time for segment completion (seconds)

# Video Configuration
video:
  default_fps_numerator: 30     # Default FPS numerator (30/1 = 30 fps)
  default_fps_denominator: 1    # Default FPS denominator (used if the stream lacks FPS)

Key settings:

  • Video mode: video_mode: 1 uses video metadata (for models with native video support); video_mode: 0 uses multi-image mode (image-only models).

  • Tensor format (image modes only): pytorch (default), pil, or numpy.

  • System prompt: specify a value to use it; omit it for no system prompt; use an empty string to send an empty system prompt.

  • Prompt placeholders: {num_frames} (number of frames in the segment), {stream_id} (stream identifier), and {timestamps} (timestamp string). Include {timestamps} to show timestamps, or omit it to exclude them.

  • Per-stream prompts: override any inference setting for specific streams; streams without overrides use the global defaults.

Plugin properties#

Properties can override the config values at runtime:

Property

Type

Default

Description

model

string

from config

HuggingFace model ID

user-prompt

string

from config

User prompt with placeholders

system-prompt

string

None

System prompt (optional)

segment-length-sec

int

10

Segment length in seconds

overlap-sec

int

0

Segment overlap

selection-fps

int

1

Target FPS (0=disabled)

subsample-interval

int

1

Keep every Nth frame

max-tokens

int

2048

Max tokens to generate

temperature

float

0.7

Sampling temperature

top-p

float

0.9

Top-p nucleus sampling

top-k

int

100

Top-k sampling

repetition-penalty

float

1.1

Repetition penalty

max-model-len

int

20480

Max model context length

trust-remote-code

bool

true

Trust remote code

gpu-memory-utilization

float

0.7

GPU memory fraction (0.0-1.0)

gpu-id

int

0

GPU device ID (-1=auto)

video-mode

int

1

1=video, 0=multi-image

tensor-format

string

pytorch

pytorch/pil/numpy

queue-maxsize

int

20

Inference queue size

max-wait-timeout

int

300

Shutdown timeout (seconds)

default-fps-numerator

int

30

Default FPS numerator

default-fps-denominator

int

1

Default FPS denominator

Model Support#

Video-native models (video_mode: 1): Cosmos-Reason2-8B (default) and other models with native video-metadata support.

Image-only models (video_mode: 0): models that support image inputs only.

Signal-based results#

Access results via GObject signals:

def on_vlm_result(element, stream_id, start_time, end_time, text, user_data):
    print(f"Stream {stream_id} [{start_time:.2f}s-{end_time:.2f}s]: {text}")

vlm.connect("vlm-result", on_vlm_result, None)