Inkling 2 Mini
Each target below is a Dynamo + SGLang deployment of Thinking Machines Lab’s Inkling 2 Mini, a multimodal (text and image in, text out) thinking MoE model with NVFP4 routed experts. Pick your GPU and topology to see the deployment, then follow the steps below.
Choose your deployment target
Prerequisites
- A Kubernetes cluster with the Dynamo platform installed and 2 B200 GPUs available.
- A Kubernetes cluster with the Dynamo platform installed and 4 B200 GPUs available.
- InfiniBand access for the prefill and decode workers. The manifest tunes UCX for it, but the
rdma/ibresource and any node placement are cluster-owned, so add them for your cluster. Without RDMA, KV moves over TCP and time to first token grows with prompt length.
- A Kubernetes cluster with the Dynamo platform installed and 2 VR200 GPUs available on arm64 nodes.
- A Kubernetes cluster with the Dynamo platform installed and 4 VR200 GPUs available on arm64 nodes.
- Both workers in one NVLink domain, on one NVL72 node or with a ComputeDomain channel claim per worker. KV moves between the workers with the Mooncake backend forced onto NVLink, and the manifest does not enforce the placement, so a cross-domain placement can fail the transfer.
- A ReadWriteMany model-cache volume of at least 100 GiB.
- A Hugging Face token only if the checkpoint repository is private or gated for your account.
The runtime image is nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.8.0-inkling-2-mini-dev.1. It carries the model owners’ SGLang runtime for this model, so no source patching happens at startup.
Run commands from the repository root:
Cluster-specific settings
The manifests contain no credentials, image-pull Secrets, or priority classes. The VR200 manifests select workers with kubernetes.io/arch: arm64 and nvidia.com/gpu.machine: VR-NVL72; replace those if your GPU nodes carry different labels. The B200 manifests set no node selector. Add what your cluster needs in a private Kustomize overlay; the usual cases are an image-pull Secret for a private registry mirror, a node selector for the GPU nodes, and a priority class on clusters with a preemptible default.
Populate the model cache
Skip this step if shared-model-cache already holds thinkingmachines/Inkling-2-Mini at /shared-model-cache/Inkling-2-Mini. PVC storage class and access mode fields are immutable, so keep an existing cache.
For a new cache, replace the placeholder storageClassName in model-cache.yaml with a ReadWriteMany-capable storage class. If the repository is private or gated, create the token Secret first:
Then download the checkpoint:
Deploy
Wait for the deployment to become ready. Loading the weights and capturing CUDA graphs takes several minutes:
Verify
Forward the frontend Service. The command keeps running:
In a second terminal, check that it serves the model:
The response lists inkling-2-mini.
Smoke test
Send a chat request. The model thinks before it answers, and the reasoning parser returns the thinking in reasoning_content and the answer in content:
Check structured output. The response content is only the JSON object; the thinking is in reasoning_content:
Check image input with an image URL or a base64 data URL:
Notes
- Supported features. Streaming and non-streaming chat and completions, a reasoning parser, tool calling (single, parallel, required, named, and streamed), structured output (
json_objectandjson_schema), logprobs, stop sequences, and image input. The recipes pass--grammar-backend xgrammarand--enable-multimodal. All recipes use the Dynamo frontend’s default parser (--dyn-chat-processor dynamo), with--dyn-reasoning-parser inklingand--dyn-tool-call-parser inklingon the workers. - Parallelism. The NVFP4 expert kernel needs expert parallelism, so every worker runs
--tp 2 --ep-size 2. A single-GPU worker is rejected. - Frontend limits. The Dynamo frontend accepts only
n=1. A fixedseeddoes not give reproducible output. Audio and video input are not supported. - Vera Rubin first request. The first request that sets
top_p,top_k, ormin_pon VR200 compiles a sampling kernel, which takes about a minute. Later requests are not affected. - Disaggregated serving. The frontend uses KV-aware routing with session affinity, and the prefill worker publishes KV events. Both workers need
--host 0.0.0.0for the handoff, which the manifests set. - Status. This recipe is experimental.