Is this page helpful?

Get Started with GLM-5.3#

GLM-5.3 is a Z.ai GLM-series large language model designed for advanced reasoning, coding, and agentic workflows. Use it for applications that need strong instruction following, long-context problem solving, and reliable text generation.

This procedure deploys GLM-5.3 with the SGLang Model-Free NIM. The model-free container downloads model assets from a remote source at startup, so you can serve GLM-5.3 without building a model-specific NIM image. For more information, refer to Model-Free NIM.

Known Issues#

This release includes the following known issues and limitations.

Environment variable NIM_DISABLE_MODEL_DOWNLOAD is not supported.

For air-gap deployment, mount the local model path and set NIM_MODEL_PATH:

-v <local-model-path>:/opt/nim/workspace \
-e NIM_MODEL_PATH=/opt/nim/workspace

Prerequisites#

Before deploying a NIM LLM container, ensure your environment meets the following requirements:

Hardware Requirements#

The following are the minimum required specifications for supported hardware components:

Requirement	Specification
CPU	AMD64, ARM64
GPU	Refer to the Support Matrix

Software Requirements#

Minimum required versions for supported software components.

Requirement	Specification
Operating System	Ubuntu 22.04 LTS or later recommended
Container Toolkit	1.14.0 or later
CUDA SDK	12.9 or later
GPU Driver	580 or later
Docker	24.0 or later

Operating System#

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK#

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers#

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker#

Docker is required to run the containerized NIM services.

Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.
Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:
```
sudo groupadd docker
sudo usermod -aG docker $USER
```
Log out and back in for the group change to take effect.

Container Toolkit#

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

Install the toolkit by following the NVIDIA Container Toolkit installation guide.
Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.
Restart the Docker daemon after configuration:
```
sudo systemctl restart docker
```

NIM Container Access#

To download and deploy NIM containers, you need one of the following:

A free NVIDIA Developer Program membership.
An NVIDIA AI Enterprise license. To request a free 90-day evaluation license, refer to Ways to Get Started With NVIDIA AI Enterprise and Activate Your NVIDIA AI Enterprise License.

Verify NVIDIA Runtime Access#

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuration#

Use environment variables to control authentication and model caching.

Create and Export a Token for the Remote Repository#

Create a Hugging Face access token with read permissions (or higher, only if your org or private repos require it).
Export the token in your shell (temporary), replacing <token-value> with your actual token:
```
export HF_TOKEN=<token-value>
```

Persist the variable (optional):

If using bash:

echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc

If using zsh:

echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc

Verify the variable is set:
```
echo "$HF_TOKEN"
```

Note

If you want to serve a pre-downloaded local model or a private cloud model instead of downloading one from Hugging Face, you do not need a Hugging Face access token. Refer to Model Downloads for your workflow.

Important

For enhanced security, consider storing your token in a file and retrieving it as needed with cat. Alternatively, consider using a password manager.

Model Cache and Source#

NIM downloads model weights and other assets to a cache directory inside the container. Map a persistent directory on your host machine to that path so files survive restarts and later deployments can reuse them.

Local Cache#

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions#

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Custom Model Source#

You can configure model-free NIM to download a model from Hugging Face or other supported sources. To do this, specify the directory where the model files should be stored and set the appropriate environment variable.

Specify a Hugging Face model directly using the hf:// prefix.

export NIM_MODEL_PATH="hf://zai-org/GLM-5.3"

The NIM will automatically download the model from Hugging Face and cache it in your configured LOCAL_NIM_CACHE. When running the container, you will map your host machine’s local cache directory to a path inside the container using the -v flag. This ensures that the downloaded model weights persist on your host machine across ephemeral container restarts.

For example, using -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" tells Docker: “Take the folder at $LOCAL_NIM_CACHE on my host machine, and make it available inside the container at the path /opt/nim/.cache.”

Pull the Container Image#

Use docker pull to fetch the NIM container image:

docker pull nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.2

How to Choose the Right Model Profile#

NIM LLM automatically selects the most optimal model profile based on the detected hardware (for example, number of GPUs or GPU architecture). If you need to manually override this selection, you can specify the NIM_MODEL_PROFILE environment variable. For more information, refer to Model Profiles.

Run NIM#

Run the container with the model-source credentials required by your selected model. The following example uses a Hugging Face token to authenticate and download the model.

export NIM_LLM_MODEL_FREE_IMAGE=nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.2

docker run --gpus=all \
  -e NIM_MODEL_PATH=$NIM_MODEL_PATH \
  -e NIM_SERVED_MODEL_NAME="zai-org/GLM-5.3" \
  -e HF_TOKEN=$HF_TOKEN \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}

Note

If you want to serve a pre-downloaded local model or a private cloud model instead of downloading one from Hugging Face, refer to Model Downloads for your workflow.

Reference Configurations for H200 and B200 GPUs#

The following SGLang configurations are recommended starting points for specific workloads on H200 and B200 GPUs. Benchmark and adjust these values for your deployment, workload, and latency or throughput requirements.

Important

Pass these settings as CLI arguments after nim-serve. The configuration snippets use underscore-separated key names for compact comparison; replace underscores with hyphens for CLI arguments. Pass a true Boolean setting as a standalone flag, for example, enable_prefill_cp: true, which becomes --enable-prefill-cp.

For more information and a complete container example, refer to Advanced Configuration.

Set SGLANG_ENABLE_SPEC_V2=1 on the container when using either EAGLE configuration. For example:

docker run --gpus=all \
  -e SGLANG_ENABLE_SPEC_V2=1 \
  ... \
  ${NIM_LLM_MODEL_FREE_IMAGE} \
  nim-serve \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  ...

Long-Context Workload#

For a long-context workload with an input sequence length (ISL) of 64K tokens and an output sequence length (OSL) of 400 tokens, use the following common configuration on H200 and B200 GPUs:

speculative_algorithm: EAGLE
speculative_num_steps: 3
speculative_eagle_topk: 1
speculative_num_draft_tokens: 4
enable_prefill_cp: true
attn_cp_size: 8
cp_strategy: interleave
max_running_requests: 32
cuda_graph_max_bs_decode: 32
watchdog_timeout: 3600
mem_fraction_static: 0.9
json_model_override_args: "{\"index_topk_pattern\": \"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS\"}"

H200 GPUs require no additional overrides. For B200 GPUs, add the following:

chunked_prefill_size: 16384

RAG-QA Workload#

For a retrieval-augmented generation question answering (RAG-QA) workload with an ISL of 8K tokens and an OSL of 1K tokens, use the following common configuration on H200 and B200 GPUs:

speculative_algorithm: EAGLE
speculative_num_steps: 3
speculative_eagle_topk: 1
speculative_num_draft_tokens: 4
schedule_policy: lpm
enable_dynamic_chunking: true
chunked_prefill_size: 4096
max_prefill_tokens: 65536
mem_fraction_static: 0.9
max_running_requests: 96
cuda_graph_max_bs_decode: 64
enable_fused_qk_norm_rope: true
watchdog_timeout: 3600
json_model_override_args: "{\"index_topk_pattern\": \"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS\"}"

Interact with the API#

There are three main inference endpoints:

Chat Completions: /v1/chat/completions
Text Completions: /v1/completions
Responses: /v1/responses

Tip

All three endpoints support streaming.

Find the Model Name#

Replace the "model" value in the examples below with the model name served by your NIM container. To find it, query the models endpoint:

curl -s http://localhost:8000/v1/models

The id field in the response is the model name to use in your requests. For model-free NIMs, the name is set by the NIM_SERVED_MODEL_NAME environment variable (for example, zai-org/GLM-5.3).

Note

You can set the environment variable NIM_SERVED_MODEL_NAME to a desired string. This name is used as the model name in API requests. For more information, refer to Environment Variables.

Send a Chat Completion Request#

Once the server is running, you can send a request to the chat completion endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [
      {
        "role": "user",
        "content": "Hello! How are you?"
      }
    ],
    "max_tokens": 1024
  }'

The expected response:

{
  "id": "6cfe4d62d3bb4fdb98558adc9879aeb2",
  "object": "chat.completion",
  "created": 1787901368,
  "model": "zai-org/GLM-5.3",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I am doing well and ready to help. What would you like to work on today?",
        "reasoning_content": "The user is greeting me and asking how I am. A concise, friendly reply is appropriate, followed by an invitation to continue.",
        "tool_calls": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "matched_stop": null
    }
  ],
  "usage": {
    "prompt_tokens": 18,
    "total_tokens": 58,
    "completion_tokens": 40,
    "prompt_tokens_details": null,
    "reasoning_tokens": 22
  },
  "metadata": {
    "weight_version": "default"
  }
}

Verify Health Endpoints#

You can verify that the NIM container is running and ready to accept requests by checking its health endpoints. By default, these endpoints are served on port 8000. If you set NIM_HEALTH_PORT, use that port instead.

Live Endpoint#

Perform a liveness check to see if the server is running:

curl -v http://localhost:8000/v1/health/live

Example response:

GET /v1/health/live HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "live",
  "status": "live"
}

Ready Endpoint#

Perform a readiness check to see if the model is fully loaded and ready for inference:

curl -v http://localhost:8000/v1/health/ready

Example response:

GET /v1/health/ready HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 63
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "ready",
  "status": "ready"
}

Streaming#

To receive responses incrementally as they are generated, you can enable streaming by adding "stream": true to your request payload. This is supported across the /v1/chat/completions, /v1/completions, and /v1/responses endpoints.

When streaming is enabled, the API returns a sequence of Server-Sent Events (SSE).

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model":"zai-org/GLM-5.3",
    "messages": [
      {
        "role": "user",
        "content": "Write a short poem about a robot."
      }
    ],
    "max_tokens": 100,
    "stream": true
  }'

The response will be streamed back in chunks, with each chunk containing a data JSON object. The stream terminates with a data: [DONE] message:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"zai-org/GLM-5.3","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"prompt_token_ids":null}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"zai-org/GLM-5.3","choices":[{"index":0,"delta":{"content":"In"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"zai-org/GLM-5.3","choices":[{"index":0,"delta":{"content":" cir"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"zai-org/GLM-5.3","choices":[{"index":0,"delta":{"content":"cuits"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

...

data: [DONE]