Is this page helpful?

Model-Specific Deployment Considerations#

Some NIM LLM and VLM containers, including the Model-Free NIMs, need extra profile selection or startup settings beyond the standard Get Started path. Use this page for those models.

Important

Check the version of the NIM you are deploying. This page only covers NIMs for the current release version. Refer to the previous versions of the documentation for NIMs that have already been released.

For generic profile selection, refer to Model Profiles and Selection. For validated GPU and precision combinations, refer to the Support Matrix for NIMs. Check the Release Notes for the requirements that apply to your image version.

Kimi K3.1 Pro#

Kimi K3.1 Pro from Moonshot AI accepts text and image input and uses the vLLM backend. Deployment requires the container, checkpoint, and matching serving recipes for NIM 2.1.6-variant-kimi-k3.1-pro. Refer to Kimi K3.1 Pro for deployment configurations and recipe filenames.

Artifacts and Prerequisites#

The vLLM Model-Free NIM (Experimental) collection contains the container and the serving-recipe resource for all three layouts. All three layouts use the same public MXFP4 checkpoint from Hugging Face. B300 uses a separate single-node recipe. GB300 is not included in this release.

Follow Prerequisites and Configuration for the host, container runtime, and authentication requirements. Authenticate the NGC CLI and Docker before downloading. Choose the GPU and node layout in Kimi K3.1 Pro.

H20-3e requires a working RDMA fabric.

For H20, each node loads its own YAML file through NIM_CONFIG_FILE. Use the head recipe on the primary node and the corresponding worker recipe on each worker. Use the checkpoint and recipes for the exact GPU and node layout. Kimi-K3 checkpoint paths and profile IDs apply to a different model release.

For this model and release, NIM_DISABLE_MODEL_DOWNLOAD is unsupported and deprecated. Do not rely on it to prevent writes to checkpoint files. Mount the downloaded checkpoint read-only, as shown in the commands below.

Prepare Every H20 Node#

Download the recipe bundle and a complete copy of the matching checkpoint on every participating node. The following commands use /raid for storage; change ARTIFACT_ROOT to a filesystem with enough space for the checkpoint. Set PRIMARY_NODE to the head node’s address, reachable from all workers.

export ARTIFACT_ROOT=/raid
export PRIMARY_NODE="<head-node-IP>"
export IMAGE="nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.6-variant-kimi-k3.1-pro"

docker pull "$IMAGE"
ngc registry resource download-version "nim/nvidia/kimi-k3.1-pro-recipe:2.1.6-variant-kimi-k3.1-pro" \
  --dest "$ARTIFACT_ROOT/recipes"
export RECIPE_DIR="$ARTIFACT_ROOT/recipes/kimi-k3.1-pro-recipe_v2.1.6-variant-kimi-k3.1-pro"

Replace <head-node-IP> before running the commands. Keep the same IMAGE and PRIMARY_NODE on every node.

H20-3e: Two or Four Nodes#

Run the following on every H20 node to download the checkpoint (about 1.6 TB):

hf download "moonshotai/Kimi-K3.1-Pro" --revision "6d0a0202956796543364d666ab19c6ad1a494ee1" \
  --local-dir "$ARTIFACT_ROOT/models/Kimi-K3.1-Pro"
export MODEL_DIR="$ARTIFACT_ROOT/models/Kimi-K3.1-Pro"

Set ROLE=head on the primary node. For two nodes, set ROLE=worker1 on the other node. For four nodes, assign worker1, worker2, and worker3 to the three workers.

For two nodes, select the TP8/PP2 recipe on each node:

export ROLE=head  # Change to this node's role.
export RECIPE="kimi-k3.1-pro-h20-3e-16gpu-nn2-tp8-pp2-$ROLE.yaml"

For four nodes, select the TP8/PP4 recipe instead:

export ROLE=head  # Change to this node's role.
export RECIPE="kimi-k3.1-pro-h20-3e-32gpu-nn4-tp8-pp4-$ROLE.yaml"

Run the head command first, then start all workers using their own ROLE and RECIPE. Do not wait for head readiness before starting workers.

docker run -d --name nim-kimi-k3-1-pro --runtime runc --user 0:0 \
  --gpus all --device /dev/infiniband:/dev/infiniband:rw \
  --network host --ipc private --shm-size 32g --ulimit memlock=-1:-1 \
  -v "$MODEL_DIR:/model:ro" -v "$RECIPE_DIR:/recipes:ro" \
  -e NIM_MODEL_PATH=/model -e NIM_WORKSPACE_PATH=/model \
  -e "NIM_CONFIG_FILE=/recipes/$RECIPE" -e NIM_TRUST_CUSTOM_CODE=1 \
  -e "NIM_PRIMARY_NODE=$PRIMARY_NODE" -e NIM_NODE_MANAGER_PORT=29541 \
  -e NIM_BACKEND_HOST=127.0.0.1 "$IMAGE" nim-serve \
  --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
  --enable-auto-tool-choice

The serving recipes do not configure the Kimi K3 parsers, so pass the three arguments after nim-serve on every node. Without them, the server starts and returns HTTP 200, but chat and image responses contain raw Kimi control tokens such as <|close|>, <|sep|>, and <|open|> instead of parsed content, and automatic tool choice is unavailable.

The H20 commands mount the checkpoint and recipes read-only and use the default API/backend ports 8000/8001. Keep these ports and manager ports 29541–29544 available. Use the same primary node address and manager base port on every node in a deployment.

B300: One Node#

B300 uses eight GPUs on one node with TP8 and PP1. Set PRIMARY_NODE to the B300 host address for the API checks below. Download the same checkpoint as H20:

export ARTIFACT_ROOT=/raid
export PRIMARY_NODE="<B300-host-IP>"
hf download "moonshotai/Kimi-K3.1-Pro" --revision "6d0a0202956796543364d666ab19c6ad1a494ee1" \
  --local-dir "$ARTIFACT_ROOT/models/Kimi-K3.1-Pro"
export MODEL_DIR="$ARTIFACT_ROOT/models/Kimi-K3.1-Pro"

Pull the image and download the recipe bundle on the B300 host, then start the container with the single-node kimi-k3.1-pro-mxfp4-b300-8gpu-nn1-tp8-pp1.yaml config. The H20 head/worker recipes do not apply. The command mounts the checkpoint and the config read-only and loads the config through NIM_CONFIG_FILE.

export IMAGE="nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.6-variant-kimi-k3.1-pro"
docker pull "$IMAGE"
ngc registry resource download-version "nim/nvidia/kimi-k3.1-pro-recipe:2.1.6-variant-kimi-k3.1-pro" \
  --dest "$ARTIFACT_ROOT/recipes"
export RECIPE_DIR="$ARTIFACT_ROOT/recipes/kimi-k3.1-pro-recipe_v2.1.6-variant-kimi-k3.1-pro"
export RECIPE="kimi-k3.1-pro-mxfp4-b300-8gpu-nn1-tp8-pp1.yaml"

docker run -d --name kimi-k31-b300 --user 0:0 --gpus all \
  --network host --ipc private --shm-size 32g --ulimit memlock=-1:-1 \
  -v "$MODEL_DIR:/model:ro" -v "$RECIPE_DIR/$RECIPE:/config/b300.yaml:ro" \
  -v kimi-k31-b300-cache:/nim-cache \
  -e NIM_CONFIG_FILE=/config/b300.yaml \
  -e NIM_MODEL_PATH=/model -e NIM_WORKSPACE_PATH=/model \
  -e NIM_CACHE_PATH=/nim-cache -e NIM_RUNTIME_CACHE_PATH=/nim-cache/runtime-cache \
  -e NIM_TRUST_CUSTOM_CODE=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=7200 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 \
  "$IMAGE" nim-serve \
  --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
  --enable-auto-tool-choice

B300 needs the same Kimi K3 parser arguments as H20, because the recipe does not configure them.

Check Readiness and Send a Request#

After starting the deployment, wait for the API host’s readiness endpoint to return HTTP 200. Send the readiness check as a GET request without a request body:

curl --fail "http://$PRIMARY_NODE:8000/v1/health/ready"

Check the startup logs on all nodes while the checkpoint loads. Read the served model name from /v1/models instead of assuming a Hugging Face repository name:

MODEL=$(curl --fail -s "http://$PRIMARY_NODE:8000/v1/models" | jq -r '.data[0].id')

curl --fail -s "http://$PRIMARY_NODE:8000/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d "{
    \"model\": \"$MODEL\",
    \"messages\": [{\"role\": \"user\", \"content\": \"Explain how a rainbow forms.\"}],
    \"max_tokens\": 8192
  }"

Reasoning can consume output tokens before an answer is returned. Allow enough output tokens for both reasoning and the answer. For image request formats, refer to Image, Audio, and Video Input. Use the model name returned above in the model field. For endpoint examples, refer to API Reference. The release also supports /v1/responses, /v1/messages, tool calling, and guided decoding. Tool calling and parsed reasoning output require the --reasoning-parser, --tool-call-parser, and --enable-auto-tool-choice arguments on every node. LoRA is not enabled. Audio and video input are not supported for this release.

For API limitations, refer to the known issues in the release notes.