Video Diffusion Support (Experimental)

For general TensorRT-LLM features and configuration, see the Reference Guide.

Dynamo supports video generation using diffusion models through the --modality video_diffusion flag and image generation through --modality image_diffusion flag.

Requirements

• TensorRT-LLM with visual_gen: The visual_gen module is part of TensorRT-LLM (tensorrt_llm._torch.visual_gen). Install TensorRT-LLM following the official instructions.
• dynamo-runtime with multimodal API: The Dynamo runtime must include ModelType.Videos or ModelType.Images support. Ensure you’re using a compatible version.
• VIDEO diffusion: imageio with ffmpeg: Required for encoding generated frames to MP4 video:

  $
  pip install imageio[ffmpeg]

Supported Models

The pipeline type is auto-detected from the model’s model_index.json — no --model-type flag is needed.

Quick Start

Video Diffusion

Launch worker

$
python -m dynamo.trtllm \
>
  --modality video_diffusion \
>
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
>
  --media-output-fs-url file:///tmp/dynamo_media

API Endpoint

Video generation uses the /v1/videos endpoint:

$
curl -X POST http://localhost:8000/v1/videos \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "prompt": "A cat playing piano",
>
    "model": "wan_t2v",
>
    "seconds": 4,
>
    "size": "832x480",
>
    "nvext": {
>
      "fps": 24
>
    }
>
  }'

Image Diffusion

Launch worker

$
python -m dynamo.trtllm \
>
  --modality image_diffusion \
>
  --model-path black-forest-labs/FLUX.1-dev \
>
  --media-output-fs-url file:///tmp/dynamo_media

API Endpoint

Image generation uses the /v1/images/generations endpoint:

$
curl -X POST http://localhost:8000/v1/images/generations \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "prompt": "A cat playing piano",
>
    "model": "black-forest-labs/FLUX.1-dev",
>
    "size": "256x256"
>
  }'

Configuration Options

Limitations

• Diffusion is experimental and not recommended for production use
• Only text-to-video and text-to-image is supported in this release (image-to-video planned)
• Requires GPU with sufficient VRAM for the diffusion model