跳到导航

Inkling 2 Mini

Serve Inkling 2 Mini with Dynamo and SGLang on B200 or VR200.

Each target below is a Dynamo + SGLang deployment of Thinking Machines Lab’s Inkling 2 Mini, a multimodal (text and image in, text out) thinking MoE model with NVFP4 routed experts. Pick your GPU and topology to see the deployment, then follow the steps below.

Choose your deployment target

GPU
Topology
Checkpoint thinkingmachines/Inkling-2-MiniPrecision NVFP4 routed expertsGPUs 2x B200 (1 replica)Parallelism TP2, EP2Features Reasoning, tool calling, structured output, image input
Checkpoint thinkingmachines/Inkling-2-MiniPrecision NVFP4 routed expertsGPUs 4x B200 (1P1D)Parallelism TP2, EP2 per roleKV transfer NIXL over UCX, KV-aware routing
Checkpoint thinkingmachines/Inkling-2-MiniPrecision NVFP4 routed expertsGPUs 2x VR200 (1 replica)Parallelism TP2, EP2Features Reasoning, tool calling, structured output, image input
Checkpoint thinkingmachines/Inkling-2-MiniPrecision NVFP4 routed expertsGPUs 4x VR200 (1P1D)Parallelism TP2, EP2 per roleKV transfer Mooncake over NVLink, KV-aware routing

Prerequisites

  • A Kubernetes cluster with the Dynamo platform installed and 2 B200 GPUs available.
  • A Kubernetes cluster with the Dynamo platform installed and 4 B200 GPUs available.
  • InfiniBand access for the prefill and decode workers. The manifest tunes UCX for it, but the rdma/ib resource and any node placement are cluster-owned, so add them for your cluster. Without RDMA, KV moves over TCP and time to first token grows with prompt length.
  • A Kubernetes cluster with the Dynamo platform installed and 2 VR200 GPUs available on arm64 nodes.
  • A Kubernetes cluster with the Dynamo platform installed and 4 VR200 GPUs available on arm64 nodes.
  • Both workers in one NVLink domain, on one NVL72 node or with a ComputeDomain channel claim per worker. KV moves between the workers with the Mooncake backend forced onto NVLink, and the manifest does not enforce the placement, so a cross-domain placement can fail the transfer.
  • A ReadWriteMany model-cache volume of at least 100 GiB.
  • A Hugging Face token only if the checkpoint repository is private or gated for your account.

The runtime image is nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.8.0-inkling-2-mini-dev.1. It carries the model owners’ SGLang runtime for this model, so no source patching happens at startup.

Run commands from the repository root:

export CONTEXT="<your-kubernetes-context>"
export NAMESPACE="<your-namespace>"
kubectl --context "$CONTEXT" create namespace "$NAMESPACE"

Cluster-specific settings

The manifests contain no credentials, image-pull Secrets, or priority classes. The VR200 manifests select workers with kubernetes.io/arch: arm64 and nvidia.com/gpu.machine: VR-NVL72; replace those if your GPU nodes carry different labels. The B200 manifests set no node selector. Add what your cluster needs in a private Kustomize overlay; the usual cases are an image-pull Secret for a private registry mirror, a node selector for the GPU nodes, and a priority class on clusters with a preemptible default.

Populate the model cache

Skip this step if shared-model-cache already holds thinkingmachines/Inkling-2-Mini at /shared-model-cache/Inkling-2-Mini. PVC storage class and access mode fields are immutable, so keep an existing cache.

For a new cache, replace the placeholder storageClassName in model-cache.yaml with a ReadWriteMany-capable storage class. If the repository is private or gated, create the token Secret first:

kubectl --context "$CONTEXT" -n "$NAMESPACE" create secret generic hf-token \
--from-literal=HF_TOKEN="<your-hugging-face-token>"

Then download the checkpoint:

kubectl --context "$CONTEXT" -n "$NAMESPACE" apply \
-f recipes/inkling-2-mini/model-cache/model-cache.yaml
kubectl --context "$CONTEXT" -n "$NAMESPACE" delete job \
inkling-2-mini-model-download --ignore-not-found
kubectl --context "$CONTEXT" -n "$NAMESPACE" apply \
-f recipes/inkling-2-mini/model-cache/model-download.yaml
kubectl --context "$CONTEXT" -n "$NAMESPACE" wait \
--for=condition=Complete job/inkling-2-mini-model-download --timeout=7200s

Deploy

export DGD="inkling-2-mini-sglang-agg"
kubectl --context "$CONTEXT" -n "$NAMESPACE" apply \
-f recipes/inkling-2-mini/sglang/agg-b200/deploy.yaml
export DGD="inkling-2-mini-sglang-disagg"
kubectl --context "$CONTEXT" -n "$NAMESPACE" apply \
-f recipes/inkling-2-mini/sglang/disagg-b200/deploy.yaml
export DGD="inkling-2-mini-sglang-agg"
kubectl --context "$CONTEXT" -n "$NAMESPACE" apply \
-f recipes/inkling-2-mini/sglang/agg-vr/deploy.yaml
export DGD="inkling-2-mini-sglang-disagg"
kubectl --context "$CONTEXT" -n "$NAMESPACE" apply \
-f recipes/inkling-2-mini/sglang/disagg-vr/deploy.yaml

Wait for the deployment to become ready. Loading the weights and capturing CUDA graphs takes several minutes:

kubectl --context "$CONTEXT" -n "$NAMESPACE" wait \
--for=condition=Ready "dynamographdeployment/${DGD}" --timeout=3600s

Verify

Forward the frontend Service. The command keeps running:

kubectl --context "$CONTEXT" -n "$NAMESPACE" port-forward \
"service/${DGD}-frontend" 8000:8000

In a second terminal, check that it serves the model:

curl -fsS http://localhost:8000/v1/models

The response lists inkling-2-mini.

Smoke test

Send a chat request. The model thinks before it answers, and the reasoning parser returns the thinking in reasoning_content and the answer in content:

curl -fsS http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "inkling-2-mini",
"messages": [{"role": "user", "content": "What is the capital of France? Answer in one word."}],
"max_tokens": 1024
}'

Check structured output. The response content is only the JSON object; the thinking is in reasoning_content:

curl -fsS http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "inkling-2-mini",
"messages": [{"role": "user", "content": "Return a JSON object with keys name and age for a fictional person."}],
"response_format": {"type": "json_object"},
"max_tokens": 3000
}'

Check image input with an image URL or a base64 data URL:

curl -fsS http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "inkling-2-mini",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "What color is this image? Answer in one word."},
{"type": "image_url", "image_url": {"url": "<image-url-or-data-url>"}}
]}],
"max_tokens": 1024
}'

Notes

  • Supported features. Streaming and non-streaming chat and completions, a reasoning parser, tool calling (single, parallel, required, named, and streamed), structured output (json_object and json_schema), logprobs, stop sequences, and image input. The recipes pass --grammar-backend xgrammar and --enable-multimodal. All recipes use the Dynamo frontend’s default parser (--dyn-chat-processor dynamo), with --dyn-reasoning-parser inkling and --dyn-tool-call-parser inkling on the workers.
  • Parallelism. The NVFP4 expert kernel needs expert parallelism, so every worker runs --tp 2 --ep-size 2. A single-GPU worker is rejected.
  • Frontend limits. The Dynamo frontend accepts only n=1. A fixed seed does not give reproducible output. Audio and video input are not supported.
  • Vera Rubin first request. The first request that sets top_p, top_k, or min_p on VR200 compiles a sampling kernel, which takes about a minute. Later requests are not affected.
  • Disaggregated serving. The frontend uses KV-aware routing with session affinity, and the prefill worker publishes KV events. Both workers need --host 0.0.0.0 for the handoff, which the manifests set.
  • Status. This recipe is experimental.

Source