Is this page helpful?

Query the Qwen3.6-35B-A3B for DGX Spark API#

This page shows how to launch the NIM container and call the Chat Completions API with curl, the OpenAI Python SDK, and LangChain. It covers image and video inputs, text-only queries, multi-turn conversations, and function calling.

This page applies to the DGX Spark only (linux/arm64 architecture, vLLM backend) release. Refer to Query the Qwen3.6 API for the general multi-architecture release.

Launch NIM#

Make sure you complete the steps in Get Started with NIM before you launch the NIM.

The command in this section launches a Docker container for this specific model. Refer to the following NGC container card for more information:

The container supports text and image input without extra dependencies. Video decoding is opt-in because the container does not distribute FFmpeg. Before sending video input, obtain or build a complete FFmpeg 8 shared library distribution for Linux ARM64. It must include the FFmpeg 8 libraries and their symlink chains, including the following libraries:

  • libavcodec.so.62

  • libavformat.so.62

  • libavutil.so.60

  • libswscale.so.9

  • libswresample.so.6

Mount the distribution’s lib directory (the directory that directly contains the .so files; use lib64 if that is the archive’s layout) at /opt/ffmpeg8. Do not set LD_LIBRARY_PATH yourself. When /opt/ffmpeg8 is mounted, the NIM container automatically prepends it to the existing LD_LIBRARY_PATH before starting the server.

No PATH override is needed because video decoding loads the shared libraries through torchcodec rather than invoking the ffmpeg executable.

Use the following commands to configure and launch the NIM container:

# Choose a container name for bookkeeping
export CONTAINER_NAME=qwen-qwen3.6-35b-a3b
# Set the path to the external ffmpeg8 binary. Only required for video input.
export FFMPEG_PATH="/absolute/path/to/external-ffmpeg8"

# The container name from the previous ngc registry image list command
Repository="qwen3.6-35b-a3b"
Latest_Tag="1.7.1-variant"

# Choose a VLM NIM Image from NGC
export IMG_NAME="nvcr.io/nim/qwen/${Repository}:${Latest_Tag}"

# Choose a path on your system to cache the downloaded models
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"

# Start the VLM NIM
# Line -v "$FFMPEG_PATH/lib:/opt/ffmpeg8:ro" is only required for video input.
docker run -it --rm --name=$CONTAINER_NAME \
--runtime=nvidia \
--gpus all \
--shm-size=16GB \
-e NGC_API_KEY=$NGC_API_KEY \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-v "$FFMPEG_PATH/lib:/opt/ffmpeg8:ro" \
-p 8000:8000 \
$IMG_NAME

Video input is supported through the /v1/chat/completions endpoint using a video_url content part. The /v1/responses endpoint does not support video input. Refer to Video Input for more information.

This NIM is built with a specialized base container and is subject to limitations. It uses the release tag 1.7.1-variant for the container image. Refer to Notes on NIM Container Variants for more information.

OpenAI Chat Completions Request#

The Chat Completions endpoint is typically used with chat or instruct-tuned models designed for a conversational approach. With the endpoint, prompts are sent as messages with roles and content, providing a natural way to keep track of a multi-turn conversation. To stream the result, set "stream": true.

Note

The snippets below use max_tokens to keep examples short. For reasoning examples, the max_tokens value is higher because reasoning output is typically longer.

For example, provide the URL of an image and query the NIM server. Add "stream": true in the request body for streaming responses.

curl -X 'POST' \
'http://0.0.0.0:8000/v1/chat/completions' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "qwen/qwen3.6-35b-a3b",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What is in this image?"
                    },
                    {
                        "type": "image_url",
                        "image_url":
                            {
                                "url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"
                            }
                    }
                ]
            }
        ],
        "max_tokens": 1024
    }'

Alternatively, use the OpenAI Python SDK:

pip install -U openai

Run the client and query the Chat Completions API:

from openai import OpenAI
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What is in this image?"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"
                }
            }
        ]
    }
]
chat_response = client.chat.completions.create(
    model="qwen/qwen3.6-35b-a3b",
    messages=messages,
    max_tokens=1024,
    stream=False,
)
assistant_message = chat_response.choices[0].message
print(assistant_message)

To stream responses, pass stream=True and iterate over the response:

# Code preceding `client.chat.completions.create` is the same.
stream = client.chat.completions.create(
    model="qwen/qwen3.6-35b-a3b",
    messages=messages,
    max_tokens=1024,
    # Take note of this param.
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        text = delta.content
        # Print immediately and without a newline to update the output as the response is
        # streamed in.
        print(text, end="", flush=True)
# Final newline.
print()

API Compatibility (vLLM Backend)#

This container uses the vLLM backend. The following notes apply:

  • Structured output: Use either the OpenAI-compatible response_format field with type set to json_schema (or vLLM’s structured_outputs field) similar to the following:

    "response_format": {
    "type": "json_schema",
    "json_schema": {
        "name": "Schema",
        "schema": { ... }
    }
}

  • The following guided decoding parameters are not supported. The guided_* family was removed upstream in vLLM 0.24:

    • guided_json

    • guided_regex

    • guided_choice

    • guided_grammar

    • guided_whitespace_pattern

  • top_logprobs is supported.

  • include_stop_str_in_output is supported.

  • continuous_usage_stats is not supported.

  • matched_stop is not emitted in responses. That field is SGLang-specific.

The following API features are not supported:

  • Reward

  • Llama API

  • nvext

Passing Images#

NIM for VLMs follows the OpenAI specification to pass images as part of the HTTP payload in a user message.

Important

The supported image formats are GIF, JPG, JPEG, and PNG.

To adjust the maximum number of images allowed per request, set the environment variable NIM_MAX_IMAGES_PER_PROMPT. The default value is 5. This variable maps to vLLM’s limit_mm_per_prompt. Requests that exceed the limit are rejected with HTTP 400. Image input works without setting this variable.

Public direct URL

Passing the direct URL of an image will cause the container to download that image at runtime.

{
    "type": "image_url",
    "image_url": {
        "url": "https://www.nvidia.com/content/dam/en-zz/Solutions/data-center/dgx-b200/dgx-b200-hero-bm-v2-l580-d.jpg"
    }
}

Base64 data

Another option, useful for images not already on the web, is to first base64-encode the image bytes and send that in your payload.

{
    "type": "image_url",
    "image_url": {
        "url": "data:image/jpeg;base64,SGVsbG8gZGVh...ciBmZWxsb3chIQ=="
    }
}

To convert images to base64, use the base64 command or the following Python code:

import base64

with open("image.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

Passing Videos#

Important

Video decoding requires an external FFmpeg 8 shared library mount at /opt/ffmpeg8. Without the mount, text and image requests are unaffected, and video requests return an actionable error. Refer to Launch NIM for setup details.

NIM for VLMs follows the OpenAI specification to pass videos as part of the HTTP payload in a user message.

Video input is supported on the /v1/chat/completions endpoint using a video_url content part. The /v1/responses endpoint does not support video input.

To adjust the maximum number of videos allowed per request, set the environment variable NIM_MAX_VIDEOS_PER_PROMPT. This variable is a limit, not an enable switch; it maps to vLLM’s limit_mm_per_prompt. Requests that carry more videos than the limit are rejected with HTTP 400. Video input works without setting this variable.

Public direct URL

Passing the direct URL of a video will cause the container to download that video at runtime.

{
    "type": "video_url",
    "video_url": {
        "url": "https://download.samplelib.com/mp4/sample-5s.mp4"
    }
}

Base64 data

For local videos, base64-encode the video bytes and send that in the payload.

{
    "type": "video_url",
    "video_url": {
        "url": "data:video/mp4;base64,SGVsbG8gZGVh...ciBmZWxsb3chIQ=="
    }
}

To convert videos to Base64, use the base64 command or the following Python code:

import base64

with open("video.mp4", "rb") as f:
    video_b64 = base64.b64encode(f.read()).decode()

Sampling and Preprocessing Parameters#

Extensions of the OpenAI API provide better control over sampling and preprocessing of images and videos at request time.

Video sampling

To control how frames are sampled from video inputs, sampling parameters are exposed using the top-level media_io_kwargs API field.

Specify either fps or num_frames. If you specify both, the model uses the option that results in the least number of frames.

"media_io_kwargs": {"video": { "fps": 3.0 }}

or

"media_io_kwargs": {"video": { "num_frames": 16 }}

As a general guideline, sampling more frames can result in better accuracy but hurts performance. The default sampling rate is 2.0 FPS, matching the recommended value.

Note

This build ships GB10-tuned media budgets. Requests that exceed these limits fail fast rather than degrading:

  • Video decode budget: 512 MiB (VLLM_VIDEO_MAX_DECODE_BYTES)

  • Max frame pixels: 2,073,600; frames are downscaled to 1080p (VLLM_VIDEO_MAX_FRAME_PIXELS)

  • Max media fetch size: 2 GiB; larger media is rejected with HTTP 400 (VLLM_MEDIA_MAX_FETCH_BYTES)

Note

Specifying values of fps or num_frames higher than the actual values for a given video results in an HTTP 400 error.

Note

The value of fps or num_frames is directly correlated to the temporal resolution of the model’s outputs. For example, at 2 FPS, timestamp precision in the generated output will be at best within +/- 0.25 seconds of the true values.

Video pixel count

To balance accuracy and performance, you can specify shortest_edge and longest_edge to control frame size after preprocessing. These parameters specify the minimum and maximum number of pixels for the ensemble of sampled frames.

Set these values using the top-level mm_processor_kwargs field:

"mm_processor_kwargs": {"size": { "shortest_edge": 1568, "longest_edge": 262144 }}

Defaults are shortest_edge=65536 and longest_edge=16777216.

Each patch of 32x32x2=2048 pixels maps to a multimodal input token.

Image frame size

Image frame size can be specified the same way:

"mm_processor_kwargs": {"size": { "shortest_edge": 1568, "longest_edge": 262144 }}

Each patch of 32x32=1024 pixels, for each image, maps to a multimodal input token.

OpenAI Python SDK

Use the extra_body parameter to pass these parameters in the OpenAI Python SDK.

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-used")
model = client.models.list().data[0].id

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant."
    },
    {
        "role": "user",
        "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": "https://download.samplelib.com/mp4/sample-5s.mp4"
                }
            },
            {
                "type": "text",
                "text": "What is in this video?"
            }
        ]
    }
]
chat_response = client.chat.completions.create(
    model=model,
    messages=messages,
    max_tokens=1024,
    stream=False,
    extra_body={
        "mm_processor_kwargs": {"size": {"shortest_edge": 1568, "longest_edge": 262144}},
        # Alternatively, this can be:
        # "media_io_kwargs": {"video": {"num_frames": some_int}},
        "media_io_kwargs": {"video": {"fps": 1.0}},
    }
)
assistant_message = chat_response.choices[0].message
print(assistant_message)

Summary table

The following table summarizes the parameters above:

Preprocessing and Sampling Parameters#

Name

Example

Default

Notes

Video FPS Sampling

"media_io_kwargs": {"video": {"fps": 3.0}}

2.0 FPS

  • Higher FPS can enhance accuracy at the cost of more computation.

  • Mutually exclusive with explicit number of frames.

Sampling N Frames in a Video

"media_io_kwargs": {"video": {"num_frames": 16}}

N/A (default is FPS sampling, not fixed number of frames)

  • A higher number of frames can enhance accuracy at the cost of more computation.

  • Mutually exclusive with explicit FPS.

Min and Max Number of Pixels for Videos and Images

"mm_processor_kwargs": {"size": { "shortest_edge": 1568, "longest_edge": 262144 }}

shortest_edge=65536, longest_edge=16777216

Higher resolutions can enhance accuracy at the cost of more computation.

Function (Tool) Calling#

You can connect NIM to external tools and services using function calling (also known as tool calling). For more information, refer to Call Functions (Tools).

Reasoning#

This model supports reasoning. It is on by default. To turn it off, add "chat_template_kwargs": { "enable_thinking": false } in the request body.

Example with reasoning turned off:

curl -X 'POST' \
'http://0.0.0.0:8000/v1/chat/completions' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "qwen/qwen3.6-35b-a3b",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What is in this image?"
                    },
                    {
                        "type": "image_url",
                        "image_url":
                            {
                                "url": "https://www.nvidia.com/content/dam/en-zz/Solutions/data-center/dgx-b200/dgx-b200-hero-bm-v2-l580-d.jpg"
                            }
                    }
                ]
            }
        ],
        "chat_template_kwargs": { "enable_thinking": false },
        "max_tokens": 4096
    }'

You can also omit the reasoning tokens from the response by setting "include_reasoning": false in the request body. The model will still reason internally. Setting "include_reasoning": false is not supported for streaming responses.

Text-only Queries#

Many VLMs such as qwen/qwen3.6-35b-a3b support text-only queries, where a VLM behaves exactly like a (text-only) LLM.

Important

Text-only capability is not available for all VLMs. Refer to the model cards in Support Matrix for support on text-only queries.

Send a text-only request with curl:

curl -X 'POST' \
'http://0.0.0.0:8000/v1/chat/completions' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "qwen/qwen3.6-35b-a3b",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant"
            },
            {
                "role": "user",
                "content": "Create a detailed itinerary for a week-long adventure trip through Southeast Asia."
            }
        ],
        "max_tokens": 4096,
        "stream": true
    }'

Using the OpenAI Python SDK:

from openai import OpenAI
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant"
    },
    {
        "role": "user",
        "content": "Create a detailed itinerary for a week-long adventure trip through Southeast Asia."
    }
]
stream = client.chat.completions.create(
    model="qwen/qwen3.6-35b-a3b",
    messages=messages,
    max_tokens=4096,
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        text = delta.content
        # Print immediately and without a newline to update the output as the response is
        # streamed in.
        print(text, end="", flush=True)
# Final newline.
print()

Multi-turn Conversation#

This model supports multi-turn conversations: send multiple messages with alternating user and assistant roles.

Important

Multi-turn capability is not available for all VLMs. Refer to the model cards for information on multi-turn conversations.

Send a multi-turn request with curl:

curl -X 'POST' \
'http://0.0.0.0:8000/v1/chat/completions' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "qwen/qwen3.6-35b-a3b",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What is in this image?"
                    },
                    {
                        "type": "image_url",
                        "image_url":
                            {
                                "url": "https://www.nvidia.com/content/dam/en-zz/Solutions/data-center/dgx-b200/dgx-b200-hero-bm-v2-l580-d.jpg"
                            }
                    }
                ]
            },
            {
                "role": "assistant",
                "content": "This image shows an **NVIDIA DGX system**, which is NVIDIA's flagship line of AI supercomputers/servers. ..."
            },
            {
                "role": "user",
                "content": "When was this system released?"
            }
        ],
        "max_tokens": 4096
    }'

Alternatively, send the request using the OpenAI Python SDK:

from openai import OpenAI
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="not-used")
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What is in this image?"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.nvidia.com/content/dam/en-zz/Solutions/data-center/dgx-b200/dgx-b200-hero-bm-v2-l580-d.jpg"
                }
            }
        ]
    },
    {
        "role": "assistant",
        "content": "This image shows an **NVIDIA DGX system**, which is NVIDIA's flagship line of AI supercomputers/servers. ..."
    },
    {
        "role": "user",
        "content": "When was this system released?"
    }
]
chat_response = client.chat.completions.create(
    model="qwen/qwen3.6-35b-a3b",
    messages=messages,
    max_tokens=4096,
    stream=False
)
assistant_message = chat_response.choices[0].message
print(assistant_message)

Using LangChain#

You can call NIM from LangChain, a framework for building applications with large language models (LLMs).

Install LangChain using the following command:

pip install -U langchain-openai langchain-core

Query the OpenAI Chat Completions endpoint using LangChain:

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

model = ChatOpenAI(
    model="qwen/qwen3.6-35b-a3b",
    openai_api_base="http://0.0.0.0:8000/v1",
    openai_api_key="not-needed"
)

message = HumanMessage(
    content=[
        {"type": "text", "text": "What is in this image?"},
        {
            "type": "image_url",
            "image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"},
        },
    ],
)

print(model.invoke([message]))