Image, Audio, and Video Input#
NIM LLM and VLM can serve multimodal models that accept image, audio, or video input through the vLLM backend. Use OpenAI-compatible Chat Completions requests with a mixed content array that contains text and media items.
Note
This guide applies to models that support the media type you send. A text-only model cannot process image, audio, or video input. For model-specific limitations, refer to the model card and the Support Matrix.
Prerequisites#
Before you send image, audio, or video input, make sure you have the following:
A NIM LLM and VLM container with the vLLM backend.
A model that supports image input, audio input, or video input (or some combination).
A running NIM server and the served model name from
/v1/models.Media URLs that are reachable from the NIM container.
The Python package for the examples you run:
openaifor the OpenAI Python SDK examples orlangchain-openaifor the LangChain examples.
Send an Image Request#
Send image input with a chat message content array that combines the prompt text with either a remote image URL or inline image bytes.
Use an External URL#
Use an image_url item inside the chat message content array.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<served-model-name>",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image in one sentence."
},
{
"type": "image_url",
"image_url": {
"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"
}
}
]
}
],
"max_completion_tokens": 512,
"temperature": 0.2
}'
pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used",
)
response = client.chat.completions.create(
model="<served-model-name>",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in one sentence."},
{
"type": "image_url",
"image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"},
},
],
}
],
max_completion_tokens=512,
temperature=0.2,
)
print(response.choices[0].message.content)
pip install langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="<served-model-name>",
base_url="http://localhost:8000/v1",
api_key="not-used",
max_completion_tokens=512,
temperature=0.2,
)
response = llm.invoke(
[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in one sentence."},
{
"type": "image_url",
"image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"},
},
],
}
]
)
print(response.content)
Send Inline Image Data#
To avoid downloading an image from a URL at request time, use Python’s base64
module and pathlib.Path to encode the image bytes.
import base64
from pathlib import Path
image_b64 = base64.b64encode(Path("image.jpg").read_bytes()).decode("utf-8")
Pass the encoded bytes inline by setting image_url.url to a data URL such as
data:image/jpeg;base64,<base64-data>. Match the MIME type to the file, such as
image/png for a PNG file.
Send an Audio Request#
Send audio input with a chat message content array that combines the prompt text with either a remote audio URL or inline audio bytes.
Use an External URL#
Audio input uses a vLLM-specific audio_url extension. Use an audio_url
item inside the chat message content array.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<served-model-name>",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Transcribe and summarize this audio."
},
{
"type": "audio_url",
"audio_url": {
"url": "https://assets.ngc.nvidia.com/products/api-catalog/gemma3n/speech.wav"
}
}
]
}
],
"max_completion_tokens": 512,
"temperature": 0.2
}'
pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used",
)
response = client.chat.completions.create(
model="<served-model-name>",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe and summarize this audio."},
{
"type": "audio_url",
"audio_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/gemma3n/speech.wav"},
},
],
}
],
max_completion_tokens=512,
temperature=0.2,
)
print(response.choices[0].message.content)
pip install langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="<served-model-name>",
base_url="http://localhost:8000/v1",
api_key="not-used",
max_completion_tokens=512,
temperature=0.2,
)
response = llm.invoke(
[
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe and summarize this audio."},
{
"type": "audio_url",
"audio_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/gemma3n/speech.wav"},
},
],
}
]
)
print(response.content)
Send Inline Audio Data#
To avoid downloading an audio file from a URL at request time, use Python’s
base64 module and pathlib.Path to encode the audio bytes.
import base64
from pathlib import Path
audio_b64 = base64.b64encode(Path("speech.wav").read_bytes()).decode("utf-8")
Pass the encoded bytes inline by setting audio_url.url to a data URL such as
data:audio/wav;base64,<base64-data>. Match the MIME type to the file, such as
audio/mpeg for an MP3 file or audio/flac for a FLAC file.
Send a Video Request#
Send video input with a chat message content array that combines the prompt text with either a remote video URL or inline video bytes.
Use an External URL#
Video input uses a vLLM-specific video_url extension. Use a video_url item
inside the chat message content array.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<served-model-name>",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what is happening in this video in one sentence."
},
{
"type": "video_url",
"video_url": {
"url": "https://assets.ngc.nvidia.com/products/api-catalog/sam-2/example1/sample1.mp4"
}
}
]
}
],
"max_completion_tokens": 512,
"temperature": 0.2
}'
pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used",
)
response = client.chat.completions.create(
model="<served-model-name>",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what is happening in this video in one sentence.",
},
{
"type": "video_url",
"video_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/sam-2/example1/sample1.mp4"},
},
],
}
],
max_completion_tokens=512,
temperature=0.2,
)
print(response.choices[0].message.content)
pip install langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="<served-model-name>",
base_url="http://localhost:8000/v1",
api_key="not-used",
max_completion_tokens=512,
temperature=0.2,
)
response = llm.invoke(
[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what is happening in this video in one sentence.",
},
{
"type": "video_url",
"video_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/sam-2/example1/sample1.mp4"},
},
],
}
]
)
print(response.content)
Send Inline Video Data#
To avoid downloading a video from a URL at request time, use Python’s base64
module and pathlib.Path to encode the video bytes.
import base64
from pathlib import Path
video_b64 = base64.b64encode(Path("video.mp4").read_bytes()).decode("utf-8")
Pass the encoded bytes inline by setting video_url.url to
data:video/mp4;base64,<base64-data>.