Get Started with Qwen3.8-2.4T-A95B

Qwen3.8-2.4T-A95B (also known as Qwen3.8-Max) features 2.4 trillion total parameters with 95 billion active parameters per token using a sparse Mixture-of-Experts (MoE) design, a 1-million-token context window, and open-weight availability.

This procedure uses the SGLang Model-Free NIM to download the model from a remote source. For more information, refer to Model-Free NIM.

Known Issues

This release includes the following known issues and limitations.

• Image and video input are not supported.

• Only GB300-NVL72 GPUs are supported.

• Deployment requires a minimum of four nodes.

• Deployment using the Kubernetes serviceAccountName is not supported.

• Kubernetes is the only supported deployment method.

Prerequisites

Before deploying a NIM LLM container, ensure your environment meets the following requirements:

Hardware Requirements

The following are the minimum required specifications for supported hardware components:

Requirement	Specification
CPU	AMD64, ARM64
GPU	Refer to the Support Matrix

Software Requirements

Minimum required versions for supported software components.

Requirement	Specification
Operating System	Ubuntu 22.04 LTS or later recommended
Container Toolkit	1.14.0 or later
CUDA SDK	12.9 or later
GPU Driver	580 or later
Docker	24.0 or later

Operating System

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker

Docker is required to run the containerized NIM services.

1. Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.

2. Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:

   sudo groupadd docker
   sudo usermod -aG docker $USER
   

3. Log out and back in for the group change to take effect.

Container Toolkit

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

1. Install the toolkit by following the NVIDIA Container Toolkit installation guide.

2. Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.

3. Restart the Docker daemon after configuration:

   sudo systemctl restart docker
   

NIM Container Access

To download and deploy NIM containers, you need one of the following:

• A free NVIDIA Developer Program membership.

• An NVIDIA AI Enterprise license. To request a free 90-day evaluation license, refer to Ways to Get Started With NVIDIA AI Enterprise and Activate Your NVIDIA AI Enterprise License.

Generate Access Credentials

To create an NGC Personal API key, do the following:

1. Generate the Personal API Key on the Setup API Keys page.

2. When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM. Always use a Personal API Key.

Verify NVIDIA Runtime Access

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuration

Use environment variables to control authentication and model caching.

Export NGC API Key

1. Export the variable in your shell (temporary), replacing <VALUE> with your actual API key:

   export NGC_API_KEY=<VALUE>
   

2. Persist the variable (optional):

   If using bash:

   echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc
   

   If using zsh:

   echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc
   

3. Verify the variable is set:

   echo "$NGC_API_KEY"
   

Create and Export a Token for the Remote Repository

1. Create a Hugging Face access token with read permissions (or higher, only if your org or private repos require it).

2. Export the token in your shell (temporary), replacing <token-value> with your actual token:

   export HF_TOKEN=<token-value>
   

3. Persist the variable (optional):

   If using bash:

   echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc
   

   If using zsh:

   echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc
   

4. Verify the variable is set:

   echo "$HF_TOKEN"
   

Note

If you want to serve a pre-downloaded local model or a private cloud model instead of downloading one from Hugging Face, you do not need a Hugging Face access token. Refer to Model Downloads for your workflow.

Important

For enhanced security, consider storing your token in a file and retrieving it as needed with cat. Alternatively, consider using a password manager.

Run NIM

This NIM needs to host a model with 2.4T parameters, so it can not be deployed directly on a single node. Deploy it using a service in a Kubernetes cluster and configure Ingress to provide external access.

Note

Your Kubernetes cluster nodes need to unmask imex.service. SRE should complete this step before deployment.

1. Install LeaderWorkerSet (LWS) in Kubernetes.

   kubectl apply --server-side -f https://github.com/kubernetes-sigs/lws/releases/latest/download/manifests.yaml
   

2. (Optional) Create a new namespace in Kubernetes. (Remove the -n parameter and value from following commands if you choose to not create a namespace.)

   export YOUR_NAMESPACE=<your-nim>
   kubectl create namespace ${YOUR_NAMESPACE}
   

3. Create a Kubernetes secret for the token.

   kubectl -n ${YOUR_NAMESPACE} create secret generic hf-token-secret \
     --from-literal=HF_TOKEN="$HF_TOKEN"
   

4. Authenticate the download of the SGLang Model-Free NIM from NGC.

   kubectl -n ${YOUR_NAMESPACE} create secret docker-registry ngc-secret \
     --docker-server=nvcr.io \
     --docker-username='$oauthtoken' \
     --docker-password="$NGC_API_KEY"
   

5. Create a values.yaml file for your multi-node deployment. Make sure to update <your-rwx-storage-class> to configure a persistent model cache for production deployments.

   image:
     repository: nvcr.io/nim/nvidia/sglang-model-free-nim
     tag: "2.1.1"
     pullPolicy: Always
   
   imagePullSecrets:
     - name: ngc-secret
   
   model:
     modelPath: hf://Qwen/Qwen3.8-2.4T-A95B-FP8
     hfTokenSecret: hf-token-secret
     jsonLogging: false
   
   openshift:
     enabled: false
   
   env:
     - name: NIM_SERVED_MODEL_NAME
       value: Qwen/Qwen3.8-2.4T-A95B-FP8
     - name: NIM_SDK_MAX_PARALLEL_DOWNLOAD_REQUESTS
       value: "3"
     - name: NIM_SDK_MAX_PARALLEL_FILE_REQUESTS
       value: "3"
     - name: NCCL_MNNVL_ENABLE
       value: "1"
     - name: NCCL_NVLS_ENABLE
       value: "1"
     - name: NCCL_CUMEM_ENABLE
       value: "1"
     - name: NCCL_IB_DISABLE
       value: "1"
     - name: NCCL_SOCKET_IFNAME
       value: eth0
     - name: GLOO_SOCKET_IFNAME
       value: eth0
   
   multiNode:
     enabled: true
     workers: 3
     tensorParallelSize: 16
     pipelineParallelSize: 1
   
   resources:
     requests:
       cpu: "32"
       memory: 256Gi
       nvidia.com/gpu: "4"
     limits:
       nvidia.com/gpu: "4"
   
   containerSecurityContext:
     privileged: true
     allowPrivilegeEscalation: true
     capabilities:
       add:
         - IPC_LOCK
         - SYS_PTRACE
   
   extraVolumes:
     imex-channels:
       hostPath:
         path: /dev/nvidia-caps-imex-channels
         type: Directory
   
   extraVolumeMounts:
     imex-channels:
       mountPath: /dev/nvidia-caps-imex-channels
   
   persistence:
     enabled: true
     accessMode: ReadWriteMany
     storageClass: <your-rwx-storage-class>
     size: 4Ti
   

6. Fetch the Helm chart package.

   helm fetch https://helm.ngc.nvidia.com/nim/charts/nim-llm-2.0.10.tgz \
     --username='$oauthtoken' --password="$NGC_API_KEY"
   

7. Install the Helm chart.

   helm install qwen38-max nim-llm-2.0.10.tgz \
     -f values.yaml \
     -n ${YOUR_NAMESPACE}
   

8. Find your deployment pods.

   kubectl get pods -n ${YOUR_NAMESPACE}
   mn-qwen38-max-nim-llm-0                              0/1     Running     0             91s
   mn-qwen38-max-nim-llm-0-1                            1/1     Running     0             91s
   mn-qwen38-max-nim-llm-0-2                            1/1     Running     0             91s
   mn-qwen38-max-nim-llm-0-3                            1/1     Running     0             91s
   nginx-proxy-test-84659d5c47-ccq4t                    1/1     Running     0             20d
   

Interact with the API

Because this NIM is deployed in Kubernetes, the usual approach is to expose access interfaces to developers outside the cluster by creating a Service and an Ingress. You can configure this on your own. This guide uses kubectl exec to run commands and perform tests directly inside the NIM container.

There are three main inference endpoints:

• Chat Completions: /v1/chat/completions

• Text Completions: /v1/completions

• Responses: /v1/responses

Tip

All three endpoints support streaming.

Find the Model Name

Replace the "model" value in the examples below with the model name served by your NIM container. To find it, query the models endpoint:

kubectl -n ${YOUR_NAMESPACE} exec mn-qwen38-max-nim-llm-0 -- curl -s http://localhost:8000/v1/models

The id field in the response is the model name to use in your requests. For model-free NIMs, the name is set by the NIM_SERVED_MODEL_NAME environment variable as shown in the preceding values.yaml file. If NIM_SERVED_MODEL_NAME is not set, the name is derived from the reference to the model (for example, Qwen/Qwen3.8-2.4T-A95B-FP8).

Send a Chat Completion Request

Once the server is running, you can send a request to the chat completion endpoint:

kubectl -n ${YOUR_NAMESPACE} exec mn-qwen38-max-nim-llm-0 -- curl http://localhost:8000/v1/chat/completions  -H "Content-Type: application/json" -d '{
    "model": "Qwen/Qwen3.8-2.4T-A95B-FP8",
    "messages": [
      {
        "role": "user",
        "content": "Hello! How are you?"
      }
    ],
    "max_tokens": 100
  }'

The expected response:

{
  "id": "a901115decc04acbae21feffdc13df7a",
  "object": "chat.completion",
  "created": 1786521015,
  "model": "Qwen/Qwen3.8-2.4T-A95B-FP8",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "\n\nHello! I’m doing well, thank you. How can I help you today?",
        "reasoning_content": "We need to respond to user: \"Hello! How are you?\" Need final answer. Simple greeting. Same language English. Concise.\n",
        "tool_calls": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "matched_stop": 248046
    }
  ],
  "usage": {
    "prompt_tokens": 58,
    "total_tokens": 107,
    "completion_tokens": 49,
    "prompt_tokens_details": null,
    "reasoning_tokens": 30
  },
  "metadata": {
    "weight_version": "default"
  }
}

Verify Health Endpoints

You can verify that the NIM container is running and ready to accept requests by checking its health endpoints. By default, these endpoints are served on port 8000. If you set NIM_HEALTH_PORT, use that port instead.

Live Endpoint

Perform a liveness check to see if the server is running:

kubectl -n ${YOUR_NAMESPACE} exec mn-qwen38-max-nim-llm-0 -- curl -v http://localhost:8000/v1/health/live

Example response:

> GET /v1/health/live HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.24.0 (Ubuntu)
< Date: Wed, 12 Aug 2026 07:52:41 GMT
< Content-Type: application/json
< Content-Length: 61
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
< Access-Control-Allow-Headers: Content-Type, Authorization, X-Request-Id, X-Session-Id, X-Correlation-Id
< Access-Control-Expose-Headers: X-Request-Id
< Access-Control-Max-Age: 3600

{"object":"health.response","message":"live","status":"live"}
{ [61 bytes data]
100    61  100    61    0     0   110k      0 --:--:-- --:--:-- --:--:-- 61000
* Connection #0 to host localhost left intact

Ready Endpoint

Perform a readiness check to see if the model is fully loaded and ready for inference:

kubectl -n ${YOUR_NAMESPACE} exec mn-qwen38-max-nim-llm-0 -- curl -v http://localhost:8000/v1/health/ready

Example response:

> GET /v1/health/ready HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.24.0 (Ubuntu)
< Date: Wed, 12 Aug 2026 07:52:41 GMT
< Content-Type: application/json
< Content-Length: 61
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
< Access-Control-Allow-Headers: Content-Type, Authorization, X-Request-Id, X-Session-Id, X-Correlation-Id
< Access-Control-Expose-Headers: X-Request-Id
< Access-Control-Max-Age: 3600

{"object":"health.response","message":"ready","status":"ready"}
{ [61 bytes data]
100    61  100    61    0     0   110k      0 --:--:-- --:--:-- --:--:-- 61000
* Connection #0 to host localhost left intact

Streaming

To receive responses incrementally as they are generated, you can enable streaming by adding "stream": true to your request payload. This is supported across the /v1/chat/completions, /v1/completions, and /v1/responses endpoints.

When streaming is enabled, the API returns a sequence of Server-Sent Events (SSE).

kubectl -n ${YOUR_NAMESPACE} exec mn-qwen38-max-nim-llm-0 -- curl -v http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
    "model":"Qwen/Qwen3.8-2.4T-A95B-FP8",
    "messages": [
      {
        "role": "user",
        "content": "Write a short poem about a robot."
      }
    ],
    "max_tokens": 100,
    "stream": true
  }'

The response will be streamed back in chunks, with each chunk containing a data JSON object. The stream terminates with a data: [DONE] message:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"Qwen/Qwen3.8-2.4T-A95B-FP8","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"prompt_token_ids":null}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"Qwen/Qwen3.8-2.4T-A95B-FP8","choices":[{"index":0,"delta":{"content":"In"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"Qwen/Qwen3.8-2.4T-A95B-FP8","choices":[{"index":0,"delta":{"content":" cir"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"Qwen/Qwen3.8-2.4T-A95B-FP8","choices":[{"index":0,"delta":{"content":"cuits"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

...

data: [DONE]

Enable Thinking

Qwen3.8-2.4T-A95B supports both thinking (default) and non-thinking modes. The thinking mode offers three reasoning_effort levels. The specific settings are as follows:

Reasoning Mode	Characteristics	Request Configuration
Non-thinking	Fast, intuitive responses; chat mode without reasoning	chat_template_kwargs: {"enable_thinking": false}
Thinking Low	efficient reasoning optimizing for speed and cost	"reasoning_effort": "low"
Thinking Medium	balancing accuracy and speed	"reasoning_effort": "medium"
Thinking xhigh	(Default) Maximum reasoning effort; complex tasks demanding thorough analysis	Default,"reasoning_effort": "xhigh"

preserve_thinking is enabled by default for all workloads for the best starting experience.