Get Started with Qwen3.8-27B

This guide provides the model-specific advanced deployment procedure for Qwen3.8-27B. It builds on the standard NIM VLM setup and documents model-specific runtime behavior.

Run NIM

For information on running this NIM, refer to the Get Started with NIM VLM guide before proceeding with the steps on this page.

Enable Thinking

Thinking is enabled by default for Qwen3.8-27B. Use reasoning_effort to control the amount of reasoning, or set chat_template_kwargs to {"enable_thinking": false} to use non-thinking mode. The default reasoning effort is xhigh.

Reasoning modes

Reasoning Mode	Characteristics	Request Configuration
Non-Thinking	Produces a direct response without a thinking trace.	"chat_template_kwargs": {"enable_thinking": false}
Low	Uses a smaller reasoning budget for lower latency.	"reasoning_effort": "low"
Medium	Balances reasoning depth and latency.	"reasoning_effort": "medium"
XHigh	Uses the largest reasoning budget and is the default.	"reasoning_effort": "xhigh"

The preserve_thinking option is enabled by default for all workloads. It preserves reasoning content across turns in multi-turn interactions. No request field is required because this behavior is enabled by default.

For example, request low reasoning effort as follows:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.8-27b",
    "messages": [
      {
        "role": "user",
        "content": "Briefly explain why the sky appears blue."
      }
    ],
    "reasoning_effort": "low"
  }'

Enable Speculative Decoding

Speculative decoding is a runtime toggle, not a separate model profile. The same profile serves requests with speculative decoding enabled or disabled. The selected profile determines the default behavior, and NIM_SPECDEC_ENABLE provides a global override.

Speculative decoding runtime configuration

Variable	Value	Behavior
NIM_SPECDEC_ENABLE	Unset	Follows the selected profile’s built-in configuration.
NIM_SPECDEC_ENABLE	1	Forces speculative decoding on wherever a built-in configuration or NIM_SPECDEC_ARGS configuration exists. If neither configuration exists, NIM logs a warning and starts without speculative decoding.
NIM_SPECDEC_ENABLE	0	Forces speculative decoding off.
NIM_SPECDEC_ARGS	JSON	Overrides the built-in configuration. It is consumed only while speculative decoding is active and does not activate speculative decoding on its own.

To force speculative decoding off, start the same image with NIM_SPECDEC_ENABLE=0:

export IMG=nvcr.io/nim/qwen/qwen3.8-27b:2.1.1-variant

docker run --rm \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY="$NGC_API_KEY" \
  -e NIM_SPECDEC_ENABLE=0 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  "$IMG"

Set NIM_SPECDEC_ENABLE=1 to force speculative decoding on when the selected profile has a built-in configuration or you supply NIM_SPECDEC_ARGS. EAGLE3 draft weights shipped by a profile are staged and cached with the model checkpoint. The runtime toggle gates whether they are used. A draft model supplied only through NIM_DRAFT_MODEL_PATH is downloaded at launch only when speculative decoding is active.

After a download to LOCAL_NIM_CACHE, built-in EAGLE3 draft weights are ready for air-gapped use. An external draft model must already be cached or available from a local path in an air-gapped environment. Setting NIM_SPECDEC_ENABLE=0 disables speculative decoding in either case.

Override the Built-in Configuration

Use NIM_SPECDEC_ARGS to pass a vLLM speculative-decoding runtime configuration that overrides the profile’s built-in configuration. Enable speculative decoding explicitly because NIM_SPECDEC_ARGS does not activate it by itself. The following example selects n-gram prompt lookup:

export IMG=nvcr.io/nim/qwen/qwen3.8-27b:2.1.1-variant
export NIM_SPECDEC_ARGS='{"speculative_config": "'\
'{\"method\": \"ngram\", '\
'\"num_speculative_tokens\": 3, '\
'\"prompt_lookup_max\": 4, '\
'\"prompt_lookup_min\": 1}"}'

docker run --rm \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY="$NGC_API_KEY" \
  -e NIM_SPECDEC_ENABLE=1 \
  -e NIM_SPECDEC_ARGS \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  "$IMG"