Prerequisites

Hardware

Note on modes

“Optimized Mode” indicates that the full TensorRT‑LLM optimized pipeline is used for best performance. “Functional” indicates the NIM runs end-to-end but may use fallback paths without certain optimizations; expect lower throughput.

Software

• Docker
• NVIDIA Container Toolkit
• CUDA Drivers (11.8+)

NGC

1. Create an NGC account and generate an NGC API key with NGC catalog access.

2. Authenticate Docker to NGC:

   docker login nvcr.io
   Username: $oauthtoken
   Password: <NGC API key>

3. Set your API key as an environment variable:

   export NGC_API_KEY=<your NGC API key>

Supported Formats

• Text: UTF-8 strings
• Video Codecs: H.264, HEVC, AV1, VP8, VP9, VC1, MPEG4, MPEG2, and MPEG1
• Image Codecs: JPEG, JPEG 2000, TIFF, BMP, PNG, PNM, WebP
• Video Submission Methods:
  • query request type: Base64-encoded videos, presigned video URLs, or video_frames (8 frames per item, as presigned URLs or base64)
  • bulk_video request type: Presigned video URLs or video_frames (8 frames per item, as presigned URLs or base64). Base64-encoded full videos are not supported in bulk_video.
• Video Length: The recommended length is 15 seconds; the recommended maximum length is 1-2 minutes. No strict maximum is enforced by the NIM.

Disk and Cache Requirements

• On first run, the container will download model assets. Allow 5-10 GB of free disk space for cache and temporary decode buffers depending on variant and workload.
• If you enable a host cache, ensure the directory has read/write permissions for the container user.