Image, Audio, and Video Input#
NIM LLM and VLM can serve multimodal models that accept image, audio, or video input. Each container image includes either the vLLM or SGLang inference engine. Supported media types depend on the container image and model. Use OpenAI-compatible Chat Completions requests with a mixed content array that contains text and media items.
Note
This guide applies to models that support the media type you send. A text-only model cannot process image, audio, or video input. For model-specific limitations, refer to the model card and the Support Matrix.
The following table lists supported content parts. The model must support the corresponding media type:
Media Type |
Content Part |
Backend Support |
|---|---|---|
Image |
|
vLLM and SGLang |
Audio |
|
vLLM and SGLang |
Video |
|
vLLM; requires video decoding dependencies in the container image |
To confirm which engine your container runs, check its startup logs or the selected profile description. For more information, refer to Model Profiles and Selection.
Prerequisites#
Before you send image, audio, or video input, make sure you have the following:
A NIM LLM and VLM container image (vLLM or SGLang). Both backends support image and audio input, but video input requires a vLLM backend.
A model that supports image input, audio input, or video input (or some combination).
A running NIM server and the served model name from
/v1/models.Media URLs that are reachable from the NIM container.
The Python package for the examples you run:
openaifor the OpenAI Python SDK examples orlangchain-openaifor the LangChain examples.
Send an Image Request#
Send image input with a chat message content array that combines the prompt text with either a remote image URL or inline image bytes.
Use an External URL#
Use an image_url item inside the chat message content array.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<served-model-name>",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image in one sentence."
},
{
"type": "image_url",
"image_url": {
"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"
}
}
]
}
],
"max_completion_tokens": 512,
"temperature": 0.2
}'
pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used",
)
response = client.chat.completions.create(
model="<served-model-name>",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in one sentence."},
{
"type": "image_url",
"image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"},
},
],
}
],
max_completion_tokens=512,
temperature=0.2,
)
print(response.choices[0].message.content)
pip install langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="<served-model-name>",
base_url="http://localhost:8000/v1",
api_key="not-used",
max_completion_tokens=512,
temperature=0.2,
)
response = llm.invoke(
[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in one sentence."},
{
"type": "image_url",
"image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"},
},
],
}
]
)
print(response.content)
Send Inline Image Data#
To avoid downloading an image from a URL at request time, use Python’s base64
module and pathlib.Path to encode the image bytes.
import base64
from pathlib import Path
image_b64 = base64.b64encode(Path("image.jpg").read_bytes()).decode("utf-8")
Pass the encoded bytes inline by setting image_url.url to a data URL such as
data:image/jpeg;base64,<base64-data>. Match the MIME type to the file, such as
image/png for a PNG file.
Send an Audio Request#
Send audio input with a chat message content array that combines the prompt text with either a remote audio URL or inline audio bytes.
Use an External URL#
Audio input uses the audio_url content part, an extension to the OpenAI Chat
Completions schema that both the vLLM and SGLang images accept. Use an
audio_url item inside the chat message content array.
Both backends also accept the OpenAI input_audio content part for inline audio. The following examples use audio_url.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<served-model-name>",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Transcribe and summarize this audio."
},
{
"type": "audio_url",
"audio_url": {
"url": "https://assets.ngc.nvidia.com/products/api-catalog/gemma3n/speech.wav"
}
}
]
}
],
"max_completion_tokens": 512,
"temperature": 0.2
}'
pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used",
)
response = client.chat.completions.create(
model="<served-model-name>",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe and summarize this audio."},
{
"type": "audio_url",
"audio_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/gemma3n/speech.wav"},
},
],
}
],
max_completion_tokens=512,
temperature=0.2,
)
print(response.choices[0].message.content)
pip install langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="<served-model-name>",
base_url="http://localhost:8000/v1",
api_key="not-used",
max_completion_tokens=512,
temperature=0.2,
)
response = llm.invoke(
[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe and summarize this audio."},
{
"type": "audio_url",
"audio_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/gemma3n/speech.wav"},
},
],
}
]
)
print(response.content)
Send Inline Audio Data#
To avoid downloading an audio file from a URL at request time, use Python’s
base64 module and pathlib.Path to encode the audio bytes.
import base64
from pathlib import Path
audio_b64 = base64.b64encode(Path("speech.wav").read_bytes()).decode("utf-8")
Pass the encoded bytes inline by setting audio_url.url to a data URL such as
data:audio/wav;base64,<base64-data>. Match the MIME type to the file, such as
audio/mpeg for an MP3 file or audio/flac for a FLAC file.
Send a Video Request#
Important
Use a vLLM container image for these video examples. The SGLang container images do not include the video decoder required for video_url requests.
Send video input with a chat message content array that combines the prompt text with either a remote video URL or inline video bytes.
Use an External URL#
Video input uses the video_url content part, an extension to the OpenAI Chat
Completions schema. Use a video_url item inside the chat message content array.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<served-model-name>",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what is happening in this video in one sentence."
},
{
"type": "video_url",
"video_url": {
"url": "https://assets.ngc.nvidia.com/products/api-catalog/sam-2/example1/sample1.mp4"
}
}
]
}
],
"max_completion_tokens": 512,
"temperature": 0.2
}'
pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used",
)
response = client.chat.completions.create(
model="<served-model-name>",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what is happening in this video in one sentence.",
},
{
"type": "video_url",
"video_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/sam-2/example1/sample1.mp4"},
},
],
}
],
max_completion_tokens=512,
temperature=0.2,
)
print(response.choices[0].message.content)
pip install langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="<served-model-name>",
base_url="http://localhost:8000/v1",
api_key="not-used",
max_completion_tokens=512,
temperature=0.2,
)
response = llm.invoke(
[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what is happening in this video in one sentence.",
},
{
"type": "video_url",
"video_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/sam-2/example1/sample1.mp4"},
},
],
}
]
)
print(response.content)
Send Inline Video Data#
To avoid downloading a video from a URL at request time, use Python’s base64
module and pathlib.Path to encode the video bytes.
import base64
from pathlib import Path
video_b64 = base64.b64encode(Path("video.mp4").read_bytes()).decode("utf-8")
Pass the encoded bytes inline by setting video_url.url to
data:video/mp4;base64,<base64-data>.