Is this page helpful?

Get Started with GLM-5.2#

Deploy GLM-5.2 with NVIDIA NIM, using SGLang as the backend inference engine. GLM-5.2 is a text model for reasoning, coding, and tool calling. For information about the model, refer to the GLM-5.2 model card.

NIM packages the model artifacts, deployment profiles, and runtime settings in a model-specific container. It selects a compatible profile for the detected hardware, including tensor parallelism, precision, and parser settings. You can also select a profile explicitly, as shown in this guide.

For the general onboarding path, refer to Quickstart. The NGC Catalog page provides container release information.

Notes#

Use the following information when configuring your deployment:

  • Performance-tuned GPUs: The container provides throughput profiles for NVIDIA H200 and NVIDIA B200. The H200 profile uses FP8 with tensor-parallel size 8, and the B200 profile uses NVFP4 with tensor-parallel size 4. Generic profiles cover additional GPU and LoRA configurations. Refer to GLM-5.2 in the support matrix for the full profile list.

  • Reasoning and tool calling: Profiles configure the glm45 reasoning parser and the glm47 tool-call parser. The standard launch does not need additional parser flags. Parsed reasoning is returned in choices[].message.reasoning_content, and tool calls are returned in choices[].message.tool_calls.

  • Thinking mode: Use chat_template_kwargs.enable_thinking to control thinking for a request. The introductory examples disable thinking to keep responses brief. Reasoning examples need a token budget for both reasoning and the final answer.

  • Speculative decoding: The performance-tuned profiles configure EAGLE speculative decoding for the model’s multi-token prediction (MTP) capability, with 5 speculative steps, top-k 1, and 6 draft tokens. These settings are applied by the selected profile.

  • Context length: The H200 throughput profile configures a maximum of 250,000 tokens, and the B200 throughput profile configures 500,000 tokens. Actual capacity depends on available KV cache and request concurrency. Check the startup logs and validate your workload before sending long requests.

Prerequisites#

Set up the host by following Prerequisites, including the NVIDIA GPU driver, Docker, and NVIDIA Container Toolkit installation. Verify that a container can access the GPUs:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Deploy on a single node with one of the following GPU configurations:

Performance-Tuned GLM-5.2 Profiles#

GPU

GPU Count

Precision

TP

PP

Minimum VRAM per GPU (GB)

Configured Maximum Context (Tokens)

NVIDIA H200

8

FP8

8

1

135

250,000

NVIDIA B200

4

NVFP4

4

1

175

500,000

TP is the tensor-parallel size, and PP is the pipeline-parallel size. Both throughput profiles have LoRA disabled (feat_lora=false). The context values are configured limits; validate your workload’s context length and concurrency on the selected configuration.

Reserve disk space for both the container image and downloaded model artifacts. Different precision profiles can download different artifacts, so selecting another profile can increase cache usage.

Configuration#

Access Credentials#

Public images and model artifacts that support keyless access do not need an NGC API key. For private or gated artifacts, follow Generate Access Credentials and export your key in the shell:

read -rsp "NGC API key: " NGC_API_KEY
export NGC_API_KEY

The Docker commands in this guide use -e NGC_API_KEY to pass the exported key when it is set.

Model Cache#

Create a persistent cache and make it writable by the container’s group, GID 0:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Mount this directory at /opt/nim/.cache in every download and serving container. Subsequent launches reuse the cached artifacts.

Installation#

Pull the Container Image#

If the image requires authentication, log in with the exported API key:

printf '%s' "$NGC_API_KEY" | docker login nvcr.io \
  --username '$oauthtoken' --password-stdin

Pull the model-specific image:

docker pull nvcr.io/nim/zai-org/glm-5.2:2.1.3

Select and Download a Model Profile#

List the packaged profiles on the deployment host:

docker run --rm --gpus all \
  -e NGC_API_KEY \
  nvcr.io/nim/zai-org/glm-5.2:2.1.3 \
  list-model-profiles

Select the SGLang profile with profile=throughput and feat_lora=false, matching the GPU, precision, and TP in the table: H200 FP8 TP8 or B200 NVFP4 TP4. Copy its profile ID into NIM_MODEL_PROFILE. Use that same ID when downloading and running the model:

export NIM_MODEL_PROFILE="<compatible-profile-id>"

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/zai-org/glm-5.2:2.1.3 \
  download-to-cache -p "$NIM_MODEL_PROFILE"

The first download can take time. For profile selection and cache management details, refer to Model Profiles and Selection and Model Download.

Run NIM#

Start the container with the profile selected in the previous step. The command exposes all host GPUs, and the selected profile determines the GPU count used:

docker run --rm --name glm-5-2-nim --gpus all \
  --shm-size=32g \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -e NIM_SERVED_MODEL_NAME=zai-org/GLM-5.2 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/zai-org/glm-5.2:2.1.3

Keep this terminal open and send requests from another terminal.

Verify Readiness and Model Identity#

Check the health endpoints. Readiness returns HTTP 200 after the model is loaded and the backend is ready for inference:

curl --fail http://localhost:8000/v1/health/live
curl --fail http://localhost:8000/v1/health/ready

If you set NIM_HEALTH_PORT, also publish that port in docker run and use it for health checks. For example, -e NIM_HEALTH_PORT=8001 requires -p 8001:8001. Inspect the served model identifier and deployment metadata:

curl --fail http://localhost:8000/v1/models
curl --fail http://localhost:8000/v1/metadata

The launch command sets NIM_SERVED_MODEL_NAME to zai-org/GLM-5.2. Use this identifier in each request’s model field. Changing the served name does not change the weights selected by the packaged manifest and profile.

Runtime Configuration#

Start with the packaged profile settings. To limit the served context length, add -e NIM_MAX_MODEL_LEN=32768 to the docker run command before the image name. Choose a value that fits your workload and hardware.

NIM_KVCACHE_PERCENT maps to SGLang’s --mem-fraction-static. It controls the static memory fraction used for model weights and KV cache. The H200 throughput profile sets this fraction to 0.90, and the B200 throughput profile sets it to 0.85. Lowering it leaves more memory for other runtime allocations but reduces available cache capacity.

The H200 throughput profile uses an FP8 E4M3 KV cache (fp8_e4m3). The B200 throughput profile sets max_running_requests to 128; the number of requests that can run concurrently also depends on their lengths and available memory.

For additional SGLang options, append flags after the image name in the docker run command. This preserves environment settings packaged in the image. For configuration precedence and the NIM_PASSTHROUGH_ARGS alternative, refer to Advanced Configuration. Verify the effective settings in the startup logs after applying overrides.

Interact with the API#

These examples use the OpenAI-compatible Chat Completions endpoint at http://localhost:8000/v1/chat/completions.

Send a Chat Completion Request#

Disable thinking for a short introductory response:

curl --fail http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.2",
    "messages": [
      {"role": "user", "content": "Explain GPU computing in one sentence."}
    ],
    "max_tokens": 1024,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

The answer is returned in choices[0].message.content. To continue a conversation, include the previous messages and assistant answers in the next request’s messages array.

Enable Thinking#

Enable thinking and allow a larger token budget:

curl --fail http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.2",
    "messages": [
      {"role": "user", "content": "Compare binary search and linear search for a sorted list. Explain when each is appropriate."}
    ],
    "max_tokens": 4096,
    "chat_template_kwargs": {"enable_thinking": true}
  }'

Inspect choices[0].message.reasoning_content for parsed reasoning and choices[0].message.content for the final answer. If the response ends with finish_reason: "length" before producing an answer, increase max_tokens or disable thinking for the request.

Use the OpenAI Python SDK#

Install the openai Python package in your client environment, then run:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
response = client.chat.completions.create(
    model="zai-org/GLM-5.2",
    messages=[{"role": "user", "content": "Summarize GPU computing in one sentence."}],
    max_tokens=1024,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Stream a Response#

Set stream to true and use curl -N to display events as they arrive:

curl --fail -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.2",
    "messages": [{"role": "user", "content": "Write a short poem about a robot."}],
    "max_tokens": 1024,
    "chat_template_kwargs": {"enable_thinking": false},
    "stream": true
  }'

The endpoint returns Server-Sent Events (SSE). Text chunks appear in choices[].delta.content, and the stream ends with data: [DONE].

Call a Tool#

The packaged glm47 parser handles tool calls. Provide tool definitions and allow the model to select a tool:

curl --fail http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.2",
    "messages": [{"role": "user", "content": "What is the weather in Santa Clara, CA?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 1024,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

When the model selects a tool, the response includes choices[0].message.tool_calls. Your application executes the function and returns the result in a follow-up request. For the tool execution cycle and MCP integration, refer to Tool Calling and MCP Integration.