Is this page helpful?

Get Started with Qwen3.8-Flash-Next#

Qwen3.8-Flash-Next is a vision-language Mixture-of-Experts model from Qwen with a 262,144-token context window. It accepts text, image, and video input, returns reasoning output, and supports tool calling. Audio input is not supported. On the tuned FP8 recipes for NVIDIA B200 and NVIDIA H200, the model’s built-in multi-token-prediction (MTP) head serves as the draft model for speculative decoding. You do not need a separate draft model.

Qwen3.8-Flash-Next is served by the vLLM Model-Free NIM, which includes the following for this model:

The Qwen reasoning parser (qwen3) and tool-call parser (qwen3_coder), enabled by default through NIM_PASSTHROUGH_ARGS.
Tuned FP8 recipes for NVIDIA B200 and NVIDIA H200 that use tensor parallel 2, expert parallel 2, and MTP speculative decoding across two GPUs.
Video decode support, through NVIDIA_DRIVER_CAPABILITIES=compute,utility,video.

The container does not bundle the weights. Set NIM_MODEL_PATH to the NVFP4, FP8, or BF16 checkpoint you want, and the container downloads it at startup, so one image serves all three precisions.

This page covers the deployment settings specific to this model.

Prerequisites#

Before deploying a NIM LLM and VLM container, ensure your environment meets the following requirements:

Hardware Requirements#

The following are the minimum required specifications for supported hardware components:

Requirement	Specification
CPU	AMD64, ARM64
GPU	Refer to the Support Matrix

Software Requirements#

Minimum required versions for supported software components.

Requirement	Specification
Operating System	Ubuntu 22.04 LTS or later recommended
Container Toolkit	1.14.0 or later
CUDA SDK	12.9 or later
GPU Driver	580 or later
Docker	24.0 or later

Operating System#

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK#

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers#

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker#

Docker is required to run the containerized NIM services.

Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.
Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:
```
sudo groupadd docker
sudo usermod -aG docker $USER
```
Log out and back in for the group change to take effect.

Container Toolkit#

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

Install the toolkit by following the NVIDIA Container Toolkit installation guide.
Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.
Restart the Docker daemon after configuration:
```
sudo systemctl restart docker
```

NIM Container Access#

To download and deploy NIM containers, you need one of the following:

A free NVIDIA Developer Program membership.
An NVIDIA AI Enterprise license. To request a free 90-day evaluation license, refer to Ways to Get Started With NVIDIA AI Enterprise and Activate Your NVIDIA AI Enterprise License.

Generate Access Credentials#

NGC#

An NGC Personal API key is not required to access most NVIDIA NIM containers and models hosted on NGC. An NGC API key is only required to download Production Branch (PB) models. This Get Started documentation uses a feature branch model as an example, so creating an NGC Personal API key is optional.

If you want to create an NGC Personal API key, do the following:

Generate the Personal API Key on the Setup API Keys page.
When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM. Always use a Personal API Key.

Hugging Face#

Create a Hugging Face access token with read permissions (or higher, only if your org or private repos require it). This token allows model-free NIM to fetch models directly from Hugging Face.

Verify NVIDIA Runtime Access#

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuration#

NIM LLM and VLM offers several configuration options using environment variables to control caching, logging, and model profiles.

Export a Hugging Face Access Token#

Export a Hugging Face token to download models from Hugging Face.

Export the variable in your shell (temporary), replacing <token-value> with your actual token:
```
export HF_TOKEN="<token-value>"
```

Optional: Persist the variable in your shell profile for future terminals.

If using bash:

echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc

If using zsh:

echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc

Verify the variable is set:
```
echo "$HF_TOKEN"
```

Export an NGC API Key#

Note

An NGC API key is only required to download Production Branch (PB) models. You can skip this section if you did not create an NGC API key.

Export the variable in your shell (temporary), replacing <VALUE> with your actual API key:
```
export NGC_API_KEY=<VALUE>
```

Optional: Persist the variable in your shell profile for future terminals.

If using bash:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc

If using zsh:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc

Verify the variable is set:
```
echo "$NGC_API_KEY"
```

Model Cache#

NIM downloads model weights to a cache on the host that you mount into the container. Artifacts persist across restarts, so you do not pull the full model on every run.

Local Cache#

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions#

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

To run as a custom user (for example, your host user), pass -u <uid>:0:

docker run --gpus all -u $(id -u):0 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Important

When using -u <uid>, you must include :0 to set GID 0 (for example, -u $(id -u):0). The container’s writable directories are group-owned by GID 0. Without it, the container will fail with PermissionError when writing to cache, config, or log paths.

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Authenticate#

If you created an NGC API key, authenticate with the NVIDIA Container Registry using the following command:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Use $oauthtoken as the username and NGC_API_KEY as the password. The $oauthtoken username is a special name that indicates that you will authenticate with an API key and not a user name and password.

Choose a Checkpoint#

Set NIM_MODEL_PATH to the checkpoint for the precision you want. For every supported combination of GPU, GPU count, and precision, you can find the correct path at Qwen3.8-Flash-Next in the Support Matrix.

Each combination is fixed: the precisions available depend on both the GPU type and the number of GPUs. For example, NVFP4 needs one GPU on B300 and GB300 but two on B200 and GB200. NIM selects the profile from the GPUs it detects and the precision of the checkpoint in NIM_MODEL_PATH. To override the automatic selection, refer to Model Profiles and Selection.

NIM_SERVED_MODEL_NAME sets the name that clients send in the model field. Without it, NIM derives the name from the checkpoint in NIM_MODEL_PATH, so the name changes with the precision you select. For example, the FP8 checkpoint yields Qwen/Qwen3.8-Flash-Next-FP8. Refer to Environment Variables for details.

List the Applicable Profiles#

To list the profiles available on your system, run list-model-profiles with the same NIM_MODEL_PATH that you used in the previous section. Because the container does not bundle the weights, it needs the checkpoint location to build its profile list; without NIM_MODEL_PATH, the command fails with Could not determine path of NIM manifest. The download-to-cache command needs NIM_MODEL_PATH for the same reason.

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -e HF_TOKEN \
  -e NIM_MODEL_PATH=hf://Qwen/Qwen3.8-Flash-Next-FP8 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u $(id -u):0 \
  nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-qwen3.8-flash-next list-model-profiles

To check which profile a running NIM selected, query /v1/metadata and read profile_id and profile_name.

NIM_MODEL_PROFILE accepts a profile ID from the Qwen3.8-Flash-Next support matrix or the full profile description printed by --dry-run or /v1/metadata, such as vllm-nvidia-b200-fp8-tp2-pp1-ep2-throughput-2901:10de-2-1. It rejects generic profile names, such as vllm-fp8-tp2-pp1, and the value default, with no matching profile_id or profile description is found in manifest.

Tuned FP8 Recipes on B200 and H200#

On two NVIDIA B200 or two NVIDIA H200 GPUs with the FP8 checkpoint, NIM selects a throughput-tuned recipe that uses tensor parallel 2, expert parallel 2, and MTP speculative decoding. These recipes need more shared memory than the Docker default of 64 MB, so add --shm-size=16GB to the docker run command. Without it, the container stops during startup with Insufficient space in /dev/shm.

Important

Do not set NIM_TENSOR_PARALLEL_SIZE and do not pass --tensor-parallel-size with these recipes. Either one overrides the recipe’s tensor-parallel size but keeps its expert-parallel size, and the NIM exits at startup with ValueError: expert_parallel_size (2) must be divisible by tensor_parallel_size (4) to derive data_parallel_size. Reducing the number of visible GPUs does not clear the override.

On NVIDIA H200, this recipe trades time to first token for throughput above a concurrency of 64. Refer to the known issue for measured values.

Run NIM#

To run the container and download the model, do the following:

Start the container. Set NIM_MODEL_PATH to the checkpoint listed for your GPU and GPU count in Qwen3.8-Flash-Next. The following example runs the tuned FP8 recipe on two NVIDIA B200 GPUs:
```
docker run -it --rm \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY \
  -e HF_TOKEN \
  -e NIM_MODEL_PATH=hf://Qwen/Qwen3.8-Flash-Next-FP8 \
  -e NIM_SERVED_MODEL_NAME=qwen/qwen3-8-flash-next \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u $(id -u):0 \
  -p 8000:8000 \
  nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-qwen3.8-flash-next
```
The example uses --gpus all. Some hosts have more GPUs than the profile uses, such as an eight-GPU B200 host that runs a two-GPU recipe. On those hosts, pass only the GPUs that you want NIM to use, for example --gpus '"device=0,1"'. Set NIM_MODEL_PATH to the checkpoint that Qwen3.8-Flash-Next lists for that GPU type.

The first start downloads about 133 GB (NVFP4), 186 GB (FP8), or 360 GB (BF16) of weights. The FP8 checkpoint arrives as 131 safetensors shards. To serve a checkpoint you already downloaded, refer to Model Download.

Warning

A fatal startup error, such as too little shared memory, a missing NIM_MODEL_PATH, or an unsupported tensor-parallel override, stops the container with exit status 0 rather than a non-zero status. Check the container log for ERROR lines instead of relying on the exit status. Refer to the known issue.
In another terminal, wait until the server is ready. Poll the readiness endpoint until it returns HTTP 200:
```
until [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/v1/health/ready)" = "200" ]; do sleep 30; done
curl -s http://localhost:8000/v1/health/ready
```
A ready server returns {"object":"health.response","message":"ready","status":"ready"}.

Note

Use a separate cache directory for each model. If NIM_MODEL_PATH cannot be resolved and the cache directory already holds a model, NIM logs a warning, falls back to the cached manifest, and serves the cached model under the configured NIM_SERVED_MODEL_NAME. Refer to the known issue.

Preserve the Default Engine Flags#

The image sets NIM_PASSTHROUGH_ARGS to the following:

--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-num-seqs 256 --no-enable-log-requests --disable-uvicorn-access-log --exclude-tools-when-tool-choice-none

A NIM_PASSTHROUGH_ARGS value set in the container environment replaces this whole string instead of extending it, so include every flag that you still need. Dropping the parsers returns reasoning text in content and makes requests that carry tools fail with HTTP 400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set. Dropping the two logging flags lowers image-request throughput.

To add one flag and keep the defaults, append the flag to the container command after the image reference instead of replacing the variable:

docker run ... nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-qwen3.8-flash-next --max-num-seqs 128

Run the container with --dry-run to print the effective vllm serve command and confirm the flags.

Interact with the API#

This NIM runs vLLM, so the endpoints and request fields in API Reference apply.

Send a Chat Completion Request#

Send the following request:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3-8-flash-next",
    "messages": [{"role": "user", "content": "How many prime numbers are there below 30?"}],
    "max_tokens": 8192
  }'

Thinking is on by default. Because the qwen3 reasoning parser is enabled, the response carries the reasoning in message.reasoning and the answer in message.content. Reasoning tokens count toward max_tokens. If finish_reason is length and content is empty, the model used the whole budget on reasoning: increase max_tokens.

Enable Tool Calling#

The qwen3_coder tool-call parser and --enable-auto-tool-choice are on by default. Send the following request:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3-8-flash-next",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get the current weather for a city",
      "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
    }}],
    "tool_choice": "auto",
    "max_tokens": 8192
  }'

The default flags include --exclude-tools-when-tool-choice-none, so a request with "tool_choice": "none" omits the tool definitions from the prompt.

Send an Image#

Send the following request:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3-8-flash-next",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Describe this image."},
      {"type": "image_url", "image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"}}
    ]}],
    "max_tokens": 8192
  }'

Send a Video#

Video input needs the video decode libraries that the NVIDIA Container Toolkit mounts when video is present in NVIDIA_DRIVER_CAPABILITIES. The image sets NVIDIA_DRIVER_CAPABILITIES=compute,utility,video. A deployment that replaces this variable, such as a Kubernetes manifest or Helm values, must retain video. Without it, every video request returns HTTP 500 with Failed to load NVENC library: libnvidia-encode.so.1.

Send the following request:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3-8-flash-next",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Describe this video."},
      {"type": "video_url", "video_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/sam-2/example1/sample1.mp4"}}
    ]}],
    "max_tokens": 8192
  }'