Is this page helpful?

Get Started with Inkling#

Deploy Inkling with NVIDIA NIM, using SGLang as the backend inference engine. Inkling is a vision language model from Thinking Machines. This guide covers text chat, image input, reasoning, and tool calling through the OpenAI-compatible Chat Completions API. For information about the model, refer to the Inkling model card.

NIM packages the model artifacts, deployment profiles, and runtime settings in a model-specific container. It selects a compatible profile for the detected hardware, including tensor parallelism and quantization settings. You can also select a profile explicitly, as shown in this guide.

For the general onboarding path, refer to Quickstart. The NGC Catalog page provides container release information.

Notes#

Use the following information when configuring your deployment:

  • Supported GPUs: The container provides six profiles covering NVIDIA H200, B200, B300, and GB300. All use NVFP4 model artifacts. H200 and B200 have performance-tuned throughput profiles with tensor-parallel size 8; generic profiles provide the additional configurations listed below.

  • H200 deployment: Pass NIM_ALLOW_NVFP4_EMULATION=1 at runtime to enable NVFP4 emulation. The H200 profile uses Marlin kernels. The commands below include this setting, which is also used by the release validation configuration.

  • Multimodal input: The container enables SGLang’s multimodal support. Image requests use image_url content parts in Chat Completions messages.

  • Reasoning and tool calling: The container configures both --reasoning-parser inkling and --tool-call-parser inkling. The standard launch does not need additional parser flags. Parsed reasoning is returned in choices[].message.reasoning_content, and tool calls are returned in choices[].message.tool_calls.

  • Reasoning effort: Use reasoning_effort to control reasoning for a request. The introductory examples use low. Allocate enough tokens for both reasoning and the final answer.

  • Speculative decoding: The performance-tuned profiles configure multi-layer EAGLE with eight speculative steps, nine draft tokens, top-k 1, and rejection sampling. Use the selected profile’s settings when starting the container.

  • LoRA: The two throughput profiles have LoRA disabled. Separate generic profiles include LoRA variants. The launch and request examples in this guide use a profile with feat_lora=false.

Prerequisites#

Set up the host by following Prerequisites, including the NVIDIA GPU driver, Docker, and NVIDIA Container Toolkit installation. Verify that a container can access the GPUs:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Deploy on a single node with one of the following profile configurations:

Inkling Profiles#

Profile Type

GPUs

Precision

TP

PP

LoRA

H200 throughput

H200

NVFP4

8

1

No

B200 throughput

B200

NVFP4

8

1

No

Generic

B300, GB300

NVFP4

4

1

No

Generic

B200, B300

NVFP4

8

1

No

Generic

B300, GB300

NVFP4

4

1

Yes

Generic

B200, B300

NVFP4

8

1

Yes

TP is the tensor-parallel size, and PP is the pipeline-parallel size. Each configuration uses TP GPUs. B300 refers to NVIDIA B300-SXM6-AC. The generic TP4 profiles require at least 279 GB of VRAM per GPU, and the generic TP8 profiles require at least 73 GB per GPU. These thresholds are profile-selection requirements; use the GPU combinations in the table. Image processing, context length, and concurrent requests require memory in addition to the model weights.

Reserve disk space for both the container image and downloaded model artifacts. Use the profiles packaged with the model-specific SGLang container when planning hardware capacity.

Configuration#

Access Credentials#

Public images and model artifacts that support keyless access do not need an NGC API key. For private or gated artifacts, follow Generate Access Credentials and export your key in the shell:

read -rsp "NGC API key: " NGC_API_KEY
export NGC_API_KEY

The Docker commands in this guide use -e NGC_API_KEY to pass the exported key when it is set.

Model Cache#

Create a persistent cache and make it writable by the container’s group, GID 0:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Mount this directory at /opt/nim/.cache in every download and serving container. Subsequent launches reuse the cached artifacts.

Installation#

Pull the Container Image#

If the image requires authentication, log in with the exported API key:

printf '%s' "$NGC_API_KEY" | docker login nvcr.io \
  --username '$oauthtoken' --password-stdin

Pull the model-specific image:

docker pull nvcr.io/nim/thinkingmachines/inkling:2.1.3

Select and Download a Model Profile#

List the packaged profiles on the deployment host:

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -e NIM_ALLOW_NVFP4_EMULATION=1 \
  nvcr.io/nim/thinkingmachines/inkling:2.1.3 \
  list-model-profiles

Select a compatible SGLang profile for the GPU and TP combination in the table. For the examples in this guide, use feat_lora=false:

  • On H200 or B200, select profile=throughput, tp=8, and the matching GPU tag. The H200 profile uses NVFP4 weights but does not declare a precision tag, so do not require a precision=nvfp4 match when selecting it.

  • On B300, select a generic NVFP4 profile with tp=4 or tp=8.

  • On GB300, select a generic NVFP4 profile with tp=4.

Generic profiles do not declare a GPU or profile=throughput tag. Copy the compatible profile ID into NIM_MODEL_PROFILE. Use that same ID when downloading and running the model:

export NIM_MODEL_PROFILE="<compatible-profile-id>"

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -e NIM_ALLOW_NVFP4_EMULATION=1 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/thinkingmachines/inkling:2.1.3 \
  download-to-cache -p "$NIM_MODEL_PROFILE"

The first download can take time. For profile selection and cache management details, refer to Model Profiles and Selection and Model Download.

Run NIM#

Start the container with the profile selected in the previous step. The command exposes all host GPUs, and the selected profile determines the GPU count used:

docker run --rm --name inkling-nim --gpus all \
  --shm-size=32g \
  -e NGC_API_KEY \
  -e NIM_ALLOW_NVFP4_EMULATION=1 \
  -e NIM_MODEL_PROFILE \
  -e NIM_SERVED_MODEL_NAME=thinkingmachines/Inkling-NVFP4 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/thinkingmachines/inkling:2.1.3

Keep this terminal open and send requests from another terminal.

The launch enables NVFP4 emulation for H200. The selected profile configures SGLang’s memory fraction and kernel backends. The H200 and B200 throughput profile settings are listed below.

Verify Readiness and Model Identity#

Check the health endpoints. Readiness returns HTTP 200 after the model is loaded and the backend is ready for inference:

curl --fail http://localhost:8000/v1/health/live
curl --fail http://localhost:8000/v1/health/ready

If you set NIM_HEALTH_PORT, also publish that port in docker run and use it for health checks. For example, -e NIM_HEALTH_PORT=8001 requires -p 8001:8001. Inspect the served model identifier and deployment metadata:

curl --fail http://localhost:8000/v1/models
curl --fail http://localhost:8000/v1/metadata

The launch command sets NIM_SERVED_MODEL_NAME to thinkingmachines/Inkling-NVFP4. Use this identifier in each request’s model field. Changing the served name does not change the weights selected by the packaged manifest and profile.

Runtime Configuration#

The throughput profiles configure the following SGLang settings:

Packaged Inkling Runtime Settings#

Setting

H200

B200

Attention backend

fa4

fa4

FP4 GEMM backend

marlin

flashinfer_trtllm

MoE runner backend

marlin

flashinfer_trtllm_routed

Static memory fraction (mem_fraction_static)

0.78

0.65

These are profile settings. Inspect the final SGLang launch arguments in the startup logs to confirm the effective configuration after applying runtime overrides. When changing the static memory fraction, use a direct --mem-fraction-static argument after the image name so the override applies to the same SGLang setting as the profile.

The profiles do not set an explicit context-length cap. To limit the served context length, add -e NIM_MAX_MODEL_LEN=32768 to the docker run command before the image name. Choose a value that fits your workload and hardware. Validate image requests as well as text requests after changing memory or context settings.

For additional SGLang options, append flags after the image name in the docker run command. This preserves environment settings packaged in the image.

The image’s NIM_PASSTHROUGH_ARGS includes multimodal support and parser settings. Setting this environment variable in docker run replaces the image’s value. When adding SGLang flags, preserve the required settings:

export NIM_PASSTHROUGH_ARGS="--trust-remote-code --enable-multimodal --reasoning-parser inkling --tool-call-parser inkling"

If you use this alternative, append your flags and pass the variable with -e NIM_PASSTHROUGH_ARGS. For configuration precedence, refer to Advanced Configuration. Verify the effective settings in the startup logs after applying overrides.

Interact with the API#

These examples use the OpenAI-compatible Chat Completions endpoint at http://localhost:8000/v1/chat/completions.

Known Issues#

Requests to the Responses API (/v1/responses) can return HTTP 400 with the following error:

texts cannot be empty and tokenizer must be initialized

Streaming requests can end with a response.failed event. Use /v1/chat/completions for the workflows in this guide.

Send a Chat Completion Request#

Use a text prompt to verify inference:

curl --fail http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [
      {"role": "user", "content": "Explain GPU computing in one sentence."}
    ],
    "max_tokens": 4096,
    "reasoning_effort": "low"
  }'

The final answer is returned in choices[0].message.content, and parsed reasoning is returned in choices[0].message.reasoning_content. If the response ends with finish_reason: "length" before producing an answer, increase max_tokens or use reasoning_effort: "none".

To continue a conversation, include the previous messages and assistant answers in the next request’s messages array.

Send an Image with a Text Prompt#

Include both text and image_url content parts. The image URL must be reachable from the container:

curl --fail http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe the image."},
        {"type": "image_url", "image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"}}
      ]
    }],
    "max_tokens": 4096,
    "reasoning_effort": "low"
  }'

Use the OpenAI Python SDK#

Install the openai Python package in your client environment, then run:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
response = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4",
    messages=[{"role": "user", "content": "Summarize GPU computing in one sentence."}],
    max_tokens=4096,
    extra_body={"reasoning_effort": "low"},
)
print(response.choices[0].message.content)

Stream a Response#

Set stream to true and use curl -N to display events as they arrive:

curl --fail -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [{"role": "user", "content": "Write a short poem about a robot."}],
    "max_tokens": 4096,
    "reasoning_effort": "low",
    "stream": true
  }'

The endpoint returns Server-Sent Events (SSE). Final-answer tokens appear in choices[].delta.content, and reasoning tokens appear in choices[].delta.reasoning_content. The stream ends with data: [DONE].

Call a Tool#

The packaged inkling parser handles tool calls. Provide tool definitions and allow the model to select a tool:

curl --fail http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [{"role": "user", "content": "What is the weather in Santa Clara, CA?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 4096,
    "reasoning_effort": "low"
  }'

When the model selects a tool, the response includes choices[0].message.tool_calls. Your application executes the function and returns the result in a follow-up request. For the tool execution cycle and MCP integration, refer to Tool Calling and MCP Integration.