Is this page helpful?

Get Started with Qwen3.8-Flash-Next#

This guide provides the advanced deployment procedure for Qwen3.8-Flash-Next. It uses the vLLM Model-Free NIM to download the model from a remote repository.

Qwen3.8-Flash-Next is a 180B-class multimodal Mixture-of-Experts model. It supports text, image, and video input, long contexts, reasoning, and function (tool) calling.

Prerequisites#

Before deploying a NIM VLM container, ensure your environment meets the requirements identified in the Prerequisites guide.

You do not need to generate an NGC API key to download models from the remote repository (Hugging Face). To download models from Hugging Face, create a Hugging Face access token with read permissions (or higher, only if your organization or private repositories require it). This token allows Model-Free NIM to fetch models directly from Hugging Face.

Configuration#

Configure the Hugging Face token, local model cache, and model source before you start the container.

Hugging Face Access Token#

  1. Export the variable in your shell (temporary), replacing <token-value> with your actual token:

    export HF_TOKEN="<token-value>"

  2. Optional: Persist it for future terminals:

    If you use Bash:

    echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc

    If you use zsh:

    echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc

  3. Verify the variable is set:

    echo "$HF_TOKEN"

Model Cache and Source#

NIM downloads model weights and other assets to a cache directory inside the container. Map a persistent directory on your host machine to that path so files survive restarts and later deployments can reuse them.

Local Cache#

An essential variable to configure on your host system is the local cache directory. This directory is mapped from the host machine to container. Assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. Use any valid environment variable name for the local cache path.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the
# container is running as a different user.
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions#

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

To run as a custom user (for example, your host user), pass -u <uid>:0:

docker run --gpus all -u $(id -u):0 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Important

When using -u <uid>, you must include :0 to set GID 0 (for example, -u $(id -u):0). The container’s writable directories are group-owned by GID 0. Without it, the container fails with PermissionError when writing to cache, config, or log paths.

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Hugging Face Model Source#

Specify a Hugging Face model directly using the hf:// prefix. Use a model ID that points to a model repository on Hugging Face. This model ID varies depending on the weight format you want to use. For example, for BF16 weights, use Qwen/Qwen3.8-Flash-Next.

# For BF16 weights:
export MODEL_ID="Qwen/Qwen3.8-Flash-Next"
# For FP8 weights:
# export MODEL_ID="Qwen/Qwen3.8-Flash-Next-FP8"

export NIM_MODEL_PATH="hf://$MODEL_ID"
# Use the same served API model name for both weight formats.
export NIM_SERVED_MODEL_NAME="Qwen/Qwen3.8-Flash-Next"

Installation#

Use docker pull to fetch the vLLM Model-Free NIM container that includes the Qwen3.8-Flash-Next recipes used in this guide.

export NIM_MODEL_FREE_IMAGE="nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.1"
docker pull "$NIM_MODEL_FREE_IMAGE"

Important

If downloading hf://Qwen/Qwen3.8-Flash-Next fails because Hugging Face returns HTTP 429 rate-limit errors, add the following environment variables when starting the container:

-e NIM_SDK_MAX_PARALLEL_DOWNLOAD_REQUESTS=1 \
-e NIM_SDK_MAX_PARALLEL_FILE_REQUESTS=1

Run NIM#

Run the container to download the model using the Model-Free NIM container.

Online Deployment#

Use online deployment when the host has network access to Hugging Face. NIM downloads the selected weights into the mounted cache before serving requests.

# Start vLLM Model-Free NIM.
docker run --gpus=all \
  --ipc=host \
  -e NIM_MODEL_PATH="$NIM_MODEL_PATH" \
  -e NIM_SERVED_MODEL_NAME="$NIM_SERVED_MODEL_NAME" \
  -e NIM_PASSTHROUGH_ARGS="" \
  -e HF_TOKEN="$HF_TOKEN" \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  "$NIM_MODEL_FREE_IMAGE"

The built-in recipe applies the following settings automatically:

  • Enables Expert Parallelism.

  • Sets max_num_batched_tokens to 9240.

  • Sets max_num_seqs to 256.

  • Activates the qwen3 reasoning parser and the qwen3_coder tool-call parser.

  • Activates vLLM loading and communication settings validated for each GPU topology (refer to Support Matrix).

You normally do not need to repeat these settings using NIM_PASSTHROUGH_ARGS. Explicit user-supplied vLLM arguments take precedence over the recipe, so overriding them can move the deployment outside of the validated configuration.

Offline Deployment#

For offline deployment, download the FP8 or BF16 model that matches the target recipe in advance, then mount the model directory into the container as read-only. The following example assumes the model is stored at /data/models/Qwen3.8-Flash-Next:

export MODEL_DIR=/data/models/Qwen3.8-Flash-Next
export MODEL_GID="$(stat -c %g "$MODEL_DIR")"

docker run -d \
  --name qwen3-8-flash-next \
  --group-add "$MODEL_GID" \
  --ipc=host \
  --gpus all \
  -v "$MODEL_DIR:/model:ro" \
  -e NIM_DISABLE_MODEL_DOWNLOAD=1 \
  -e NIM_MODEL_PATH=/model \
  -e NIM_SERVED_MODEL_NAME="Qwen/Qwen3.8-Flash-Next" \
  -p 8000:8000 \
  "$NIM_MODEL_FREE_IMAGE"

Mounted-Path Permissions#

The container’s default nvs user, whose GID is 1000, cannot always read files mounted from the host. If so, the container can fail to start with a Permission denied error.

Run stat -c %g /data/models/Qwen3.8-Flash-Next to obtain the model directory’s group ID, then add that group to the container with --group-add. The model directory and its contents must be readable by the group, and every parent directory in the mounted path must grant the group execute permission so that the path can be traversed.

Do not use chmod 777. After confirming that the directory has the correct group ownership, grant only the required group permissions (for example, chmod -R g+rX /data/models/Qwen3.8-Flash-Next).

Interact with the API#

After the container starts, use the OpenAI-compatible API to send inference requests and verify that the service is live and ready.

Send a Chat Completion Request#

After the server is running, you can send a request to the chat completion endpoint:

curl -X 'POST' \
  'http://localhost:8000/v1/chat/completions' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3.8-Flash-Next",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What is in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 1024
  }'

The expected response resembles the following example:

{
  "id": "chatcmpl-b26abc7313f83ba9",
  "object": "chat.completion",
  "created": 1781645253,
  "model": "Qwen/Qwen3.8-Flash-Next",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The image captures a serene rural landscape with a wooden boardwalk, tall green grass, and a clear sky.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 591,
    "total_tokens": 806,
    "completion_tokens": 215,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

To avoid downloading an image from a URL at request time, use Python’s base64 module and pathlib.Path to encode the image bytes.

import base64
from pathlib import Path

image_data = base64.b64encode(Path("image.png").read_bytes()).decode("utf-8")

Pass the encoded bytes inline by setting image_url.url to a data URL such as data:image/png;base64,<base64-data>. Match the MIME type to the file, such as image/png for a PNG file.

"image_url": {
  "url": "data:image/png;base64,<base64-data>"
}

Verify Health Endpoints#

You can verify that the NIM container is running and ready to accept requests by checking its health endpoints. By default, these endpoints are served on port 8000. If you set NIM_HEALTH_PORT, use that port instead.

Live Endpoint#

Perform a liveness check to determine whether the server is running:

curl -v http://localhost:8000/v1/health/live

Example response:

GET /v1/health/live HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "live",
  "status": "live"
}

Ready Endpoint#

Perform a readiness check to determine whether the model is fully loaded and ready for inference:

curl http://localhost:8000/v1/health/ready

Example response:

GET /v1/health/ready HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 63
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "ready",
  "status": "ready"
}

Input Video#

Video input requires an external FFmpeg installation to be mounted when the container starts. FFmpeg is not included in the image. Use a recent FFmpeg 8.x build. Image input does not require FFmpeg.

Add the following options to the docker run command:

-v /path/to/external-ffmpeg:/opt/external-ffmpeg:ro \
-e PATH=/opt/external-ffmpeg/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e LD_LIBRARY_PATH=/opt/external-ffmpeg/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib

The mounted directory must provide bin/ffmpeg, bin/ffprobe, and the required FFmpeg/libav shared libraries under lib/.

When setting PATH and LD_LIBRARY_PATH, preserve the image’s existing CUDA and NVIDIA paths. Do not keep only the standard system paths, and do not rely on a trailing colon in LD_LIBRARY_PATH=/opt/external-ffmpeg/lib: to inherit the original value. Otherwise, CUDA components such as FlashInfer and DeepGemm can fail to find required tools or libraries during service startup.

Note

External FFmpeg is required only for video input. You do not need to mount it for image-only inference.

Enable Thinking#

Thinking is enabled by default for Qwen3.8-Flash-Next. Use reasoning_effort to control the amount of reasoning, or set chat_template_kwargs to {"enable_thinking": false} to use non-thinking mode. The default reasoning effort is xhigh.

Reasoning Modes#

Reasoning Mode

Characteristics

Request Configuration

Non-Thinking

Produces a direct response without a thinking trace.

"chat_template_kwargs": {"enable_thinking": false}

Low

Uses a smaller reasoning budget for lower latency.

"reasoning_effort": "low"

Medium

Balances reasoning depth and latency.

"reasoning_effort": "medium"

XHigh

Uses the largest reasoning budget and is the default.

"reasoning_effort": "xhigh"

For example, request low reasoning effort as follows:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.8-Flash-Next",
    "messages": [
      {
        "role": "user",
        "content": "Briefly explain why the sky appears blue."
      }
    ],
    "reasoning_effort": "low"
  }'

Function Calling#

Qwen3.8-Flash-Next supports OpenAI-compatible function calling. The built-in recipe configures the qwen3_coder tool-call parser. When the model returns tool calls, first append the complete assistant message to the conversation history. Then append one tool message for each call, using the matching tool_call_id, before sending the next request. Do not discard reasoning or tool_calls from the assistant message.

After a tool-call parser is configured, NIM automatically enables automatic tool choice, so requests can use tool_choice="auto". If you override the parser or any tool-calling-related vLLM options, validate the resulting configuration first.