Is this page helpful?

Get Started with Nemotron 3.5 Lightning#

Nemotron 3.5 Lightning is a compact, text-only, reasoning-capable model in the Nemotron 3.5 Lightning family. It uses a hybrid Latent Mixture-of-Experts (LatentMoE) architecture that interleaves Mamba-2 and MoE layers with select attention layers, and it activates only 3B of its 30B total parameters per token, so it delivers strong reasoning and coding quality with the memory footprint and latency of a much smaller, dense model. The model also includes Multi-Token Prediction (MTP) layers, which provide native speculative decoding. The NIM exposes OpenAI-compatible and Anthropic-compatible APIs, so existing clients can use the same container for Chat Completions, Responses, Anthropic Messages, tool calling, and agentic workflows.

What NIM adds on top of direct vLLM. The runtime, quantization choices, context length, GPU-memory allocation, parallelism, reasoning parser, and tool-call parser are the same building blocks you would configure yourself when running vLLM directly. The NIM’s contribution is the setup and validation burden it removes: one model-specific container image, a tested model/runtime combination, an automatically selected packaged deployment profile matched to the detected GPU, and a per-precision support matrix that tells you which profiles fit which GPUs before you start. You can still override the packaged profile, tune memory, context length, and parallelism, and pass raw vLLM flags through NIM_PASSTHROUGH_ARGS.

This page is the model-specific guide for Nemotron 3.5 Lightning. For the generic NIM LLM onboarding path, refer to Quickstart. For full endpoint examples, refer to API Reference. For tool calling and MCP integration, refer to Tool Calling and MCP Integration.

Known Issues#

This release includes the following known issues and limitations.

The following items are inherited from the vLLM backend or the model checkpoint configuration. You can use these items to configure and validate deployments.

  • Reasoning parser: Start the container with --reasoning-parser nemotron_v3 when you want vLLM to parse Nemotron 3 reasoning output. Without a reasoning parser, non-JSON responses can include <think> content in the assistant content field, the reasoning field can be null, and reasoning-token accounting can be reported as regular output tokens.

  • Thinking budget: This model supports thinking_token_budget when thinking is enabled through chat_template_kwargs: {"enable_thinking": true}. thinking_token_budget limits the thinking portion of generation, while max_tokens still caps total generated tokens for the request. If a response ends before producing a final answer, increase max_tokens, increase thinking_token_budget, or simplify the prompt.

  • Responses token caps: For /v1/responses, max_output_tokens covers the total generated output budget. If the request spends the budget before producing final visible text, the response can return status: "incomplete". Increase max_output_tokens for prompts that need more reasoning or longer final answers.

  • JSON mode: response_format: {"type": "json_object"} can suppress visible reasoning output and is recommended for structured responses that you validate with a schema.

  • Tool calling: Requests that use "tool_choice": "auto" require the container to start with --enable-auto-tool-choice and --tool-call-parser qwen3_coder. Pass these using NIM_PASSTHROUGH_ARGS in Docker, Kubernetes, or other orchestrated environments.

  • BF16 multi-GPU startup on B200 and B300: On NVIDIA-B200 and NVIDIA-B300-SXM6-AC, a BF16 profile with a tensor-parallel size of 2 or more can fail during startup while the FlashInfer MoE kernels are being auto-tuned, with a CUDA illegal memory access raised from flashinfer::trtllm_bf16_moe. This is an upstream FlashInfer defect in the auto-tuning sweep and is not an out-of-memory condition. It does not affect every deployment: it has been reproduced on driver 595.58.03 (R595), and the same container and profile start successfully on driver 580.126.20 (R580). If you encounter it, add --no-enable-flashinfer-autotune to NIM_PASSTHROUGH_ARGS, or select a different MoE backend with --moe-backend triton. On Blackwell GPUs, an NVFP4 profile avoids this code path entirely. Refer to the Blackwell BF16 Multi-GPU Startup section.

  • KV cache dtype: This model is calibrated for an FP8 KV cache, which is the NIM default. Do not set --kv-cache-dtype nvfp4; NVFP4 KV cache is not recommended and can degrade multi-turn and reasoning quality. NVFP4 checkpoints automatically use an FP8 KV cache when --kv-cache-dtype is left unset.

  • Generation defaults: The model checkpoint can provide generation_config.json defaults. Override sampling parameters per request when you need a different value. To ignore model-provided generation defaults and use vLLM defaults, add --generation-config vllm to NIM_PASSTHROUGH_ARGS.

  • Profile verification: Use the support matrix as the source of truth for supported GPU, precision, and TP combinations. After startup, verify the selected profile and served model with /v1/metadata, /v1/models, and the startup logs.

  • Backend validation errors: Some malformed requests, such as empty or missing messages, are validated by the backend and can return backend error wording. Send a non-empty messages array for Chat Completions and Anthropic Messages requests.

  • Scope: Nemotron 3.5 Lightning is a text-only model. Use it for text reasoning, coding, tool calling, and agentic workflows. It is not a multimodal image, audio, or video model.

Prerequisites#

Before deploying a NIM LLM container, ensure your environment meets the following requirements:

Hardware Requirements#

The following are the minimum required specifications for supported hardware components:

Requirement

Specification

CPU

AMD64, ARM64

GPU

Refer to the Support Matrix

Software Requirements#

Minimum required versions for supported software components.

Requirement

Specification

Operating System

Ubuntu 22.04 LTS or later recommended

Container Toolkit

1.14.0 or later

CUDA SDK

12.9 or later

GPU Driver

580 or later

Docker

24.0 or later

Operating System#

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK#

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers#

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker#

Docker is required to run the containerized NIM services.

  1. Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.

  2. Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:

    sudo groupadd docker
sudo usermod -aG docker $USER

  3. Log out and back in for the group change to take effect.

Container Toolkit#

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

  1. Install the toolkit by following the NVIDIA Container Toolkit installation guide.

  2. Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.

  3. Restart the Docker daemon after configuration:

    sudo systemctl restart docker

NIM Container Access#

To download and deploy NIM containers, you need one of the following:

Generate Access Credentials#

An NGC Personal API key is not required to access most NVIDIA NIM containers and models hosted on NGC. An NGC API key is only required to download Production Branch (PB) models. This Get Started documentation uses a feature branch model as an example, so creating an NGC Personal API key is optional.

If you want to create an NGC Personal API key, do the following:

  1. Generate the Personal API Key on the Setup API Keys page.

  2. When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM. Always use a Personal API Key.

Verify NVIDIA Runtime Access#

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuration#

Use environment variables to control authentication and model caching.

Export NGC API Key#

Note

An NGC API key is only required to download Production Branch (PB) models. You can skip this section if you did not create an NGC API key.

  1. Export the variable in your shell (temporary), replacing <VALUE> with your actual API key:

    export NGC_API_KEY=<VALUE>

  2. Persist the variable (optional):

    If using bash:

    echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc

    If using zsh:

    echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc

  3. Verify the variable is set:

    echo "$NGC_API_KEY"

Model Cache#

NIM downloads model weights to a cache on the host that you mount into the container. Artifacts persist across restarts, so you do not pull the full model on every run.

Local Cache#

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions#

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

To run as a custom user (e.g., your host user), pass -u <uid>:0:

docker run --gpus all -u $(id -u):0 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Important

When using -u <uid>, you must include :0 to set GID 0 (e.g., -u $(id -u):0). The container’s writable directories are group-owned by GID 0. Without it, the container will fail with PermissionError when writing to cache, config, or log paths.

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Installation#

Before running a NIM LLM container, you must authenticate with your deployment source, accept the governing terms, and pull the container image.

Docker Login#

If you created an NGC API key, authenticate with the NVIDIA Container Registry using the following command:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Use $oauthtoken as the username and NGC_API_KEY as the password. The $oauthtoken username is a special name that indicates that you will authenticate with an API key and not a user name and password.

Accept the Governing Terms#

Before you download a given NIM for the first time, you must accept the governing terms in the browser. Go to Nemotron 3.5 Lightning on NGC and click Accept Terms.

Pull the Container Image#

After you generate your API key and authenticate with your deployment source, download the NIM container image to your host machine.

Use the docker pull command to fetch the NIM container image.

docker pull nvcr.io/nim/nvidia/nemotron-3.5-lightning-30b-a3b:2.0.9-variant

Storage and Startup Notes#

Ensure the host has enough free disk space for the container image and the model cache. As a rough guide, the container image is on the order of tens of GB, and the model cache ranges from approximately 19 GB for an NVFP4 profile to approximately 63 GB for a BF16 profile, depending on precision and GPU configuration. Reserve additional space if you download multiple profiles or keep older container images on the same host.

The first launch downloads the model artifacts into the mounted cache directory, which can take some time depending on your hardware and network. Subsequent launches reuse the mounted cache and start faster.

Tip

By default, the model download produces little log output and can appear idle. To follow the download progress, add -e NIM_LOG_LEVEL=INFO to the docker run command.

To pre-populate the cache before serving traffic, first discover the profile hash that matches your GPU and precision, then pass it to download-to-cache with the --profile selector. Passing --profile avoids downloading the wrong profile on hosts where profile auto-detection depends on runtime state.

  1. List the available profiles for your hardware:

    docker run --rm --gpus=all \
  -e NGC_API_KEY=$NGC_API_KEY \
  nvcr.io/nim/nvidia/nemotron-3.5-lightning-30b-a3b:2.0.9-variant \
  list-model-profiles

    Copy the profile hash from the row that matches your GPU, precision, and tensor-parallel size (for example, an NVFP4 or BF16 profile).

  2. Export the profile hash and download it into the cache:

    export NIM_MODEL_PROFILE="<profile-hash-from-step-1>"

docker run --rm --gpus=all \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/nvidia/nemotron-3.5-lightning-30b-a3b:2.0.9-variant \
  download-to-cache -p "$NIM_MODEL_PROFILE"

For details on discovering, selecting, and managing profiles, refer to Model Download.

Run NIM#

Run the container using your NGC API key to authenticate and download the model. Nemotron 3.5 Lightning is a reasoning model, so launch it with the nemotron_v3 reasoning parser enabled. Without the parser, the reasoning trace and the final answer arrive together in choices[].message.content and include a literal </think> marker, which most callers do not expect.

docker run --gpus=all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_PASSTHROUGH_ARGS="--reasoning-parser nemotron_v3" \
  -e NIM_MODEL_NAME=nvidia/nemotron-3.5-lightning-30b-a3b \
  -e NIM_SERVED_MODEL_NAME=nvidia/nemotron-3.5-lightning-30b-a3b \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/nvidia/nemotron-3.5-lightning-30b-a3b:2.0.9-variant

With the parser enabled, choices[].message.content contains only the final answer, and any reasoning tokens are returned separately in choices[].message.reasoning. The examples in the rest of this guide assume this launch. If you launch without the parser, expect <think>...</think> in content and adjust your client code accordingly.

Nemotron 3.5 Lightning runs on a single node. Most profiles start successfully with default settings and do not require additional NIM_PASSTHROUGH_ARGS beyond --reasoning-parser nemotron_v3. For information about how NIM_PASSTHROUGH_ARGS is processed, refer to Advanced Configuration.

Model Identifier#

Every request must send the served model identifier in the model field. The launch command above sets it explicitly with NIM_SERVED_MODEL_NAME, so the identifier matches the container name:

nvidia/nemotron-3.5-lightning-30b-a3b

Variable

Purpose

NIM_SERVED_MODEL_NAME

The identifier the server registers and reports through /v1/models. This is what clients send in the model field.

NIM_MODEL_NAME

The model identity the NIM reports through its metadata endpoints. Keep it identical to NIM_SERVED_MODEL_NAME.

You can choose any identifier that suits your deployment — set both variables to the same value and use it in every request. To confirm what a running server serves:

curl -s http://localhost:8000/v1/models | jq -r '.data[].id'

That value is the only one the model field accepts.

Note

These variables control only the identifier the service advertises. They do not change which weights are downloaded — those are resolved from the model manifest inside the image.

Setting them explicitly also keeps your deployment stable across releases: your clients continue to use the value you chose.

Interact with the API#

The NIM exposes OpenAI-compatible and Anthropic-compatible inference endpoints. The OpenAI-compatible endpoints are the following:

  • Chat Completions: /v1/chat/completions

  • Text Completions: /v1/completions

  • Responses: /v1/responses

The Anthropic-compatible endpoints are the following:

  • Messages: /v1/messages

  • Count Tokens: /v1/messages/count_tokens

Tip

Chat Completions, Text Completions, Responses, and Anthropic Messages support streaming.

Note

To carry context across turns, send the prior turns inline in the input array. To enable server-side storage so that previous_response_id and response retrieval work, start the container with VLLM_ENABLE_RESPONSES_API_STORE=1. This increases memory usage.

Important

Reasoning and visible output share one max_tokens budget. Nemotron 3.5 Lightning is a reasoning model. When thinking is enabled (the default for the chat template), the tokens consumed by the reasoning trace and the tokens consumed by the visible answer both count against max_tokens. If the reasoning trace fills the budget first, the response returns with finish_reason: "length" and no final answer.

Follow these rules in every example:

  • For concise or structured examples where you do not need reasoning, disable thinking by passing chat_template_kwargs={"enable_thinking": false} in the request. This guarantees the entire max_tokens budget is available for the visible answer.

  • For examples that explicitly demonstrate reasoning, size max_tokens large enough for both the reasoning trace and the final answer, and either cap reasoning with thinking_token_budget on /v1/chat/completions or plan the budget explicitly.

  • Verify that every request in your CI returns finish_reason: "stop" and a non-empty final answer, not finish_reason: "length".

Send a Chat Completion Request#

After the server is running, you can send a request to the chat completion endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "messages": [
      {
        "role": "user",
        "content": "Hello! How are you?"
      }
    ],
    "max_tokens": 128,
    "temperature": 0.0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

The chat_template_kwargs.enable_thinking: false flag disables reasoning for this concise example so the full max_tokens budget is available for the visible answer. Omit this flag to keep reasoning enabled, but size max_tokens accordingly. Refer to the reasoning-budget note above.

The response has the following general format. The exact generated text can vary by sampling settings and runtime configuration. Because the primary launch enables --reasoning-parser nemotron_v3, the reasoning field contains the parsed reasoning trace (or is null when the request disables thinking, as this concise example does). If you launch without the parser, the reasoning text and the </think> marker appear in content instead.

{
  "id": "chatcmpl-87d0c4524fb6f1a4",
  "object": "chat.completion",
  "created": 1769635152,
  "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I'm ready to help.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": "..."
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 41,
    "total_tokens": 46,
    "completion_tokens": 5,
    "prompt_tokens_details": null,
    "completion_tokens_details": {
      "reasoning_tokens": 2
    }
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

Hold a Multi-Turn Conversation#

The Chat Completions endpoint is stateless. To carry context across turns, resend the full conversation in the messages array, alternating user and assistant turns after an optional system message:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "My favorite color is blue."},
      {"role": "assistant", "content": "Noted — your favorite color is blue."},
      {"role": "user", "content": "What did I just tell you?"}
    ],
    "max_tokens": 128,
    "temperature": 0.0
  }'

Note

Append only the assistant content from prior turns to the history. When the reasoning parser (--reasoning-parser nemotron_v3) is enabled, do not feed the reasoning field back into messages.

Use the OpenAI Python SDK#

You can direct the OpenAI Python SDK at the NIM endpoint by setting base_url to the local /v1 API path and providing any non-empty API key:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-used",
)

response = client.chat.completions.create(
    model="nvidia/nemotron-3.5-lightning-30b-a3b",
    messages=[{"role": "user", "content": "Summarize GPU computing in one sentence."}],
    max_tokens=128,
    temperature=0.0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

print(response.choices[0].message.content)

The extra_body={"chat_template_kwargs": {"enable_thinking": False}} argument disables reasoning so the full max_tokens budget is available for the one-sentence summary. Omit it when you want the model to reason before answering, but raise max_tokens to leave room for both the reasoning trace and the final answer.

For Anthropic Python SDK examples, refer to Messages (Anthropic-compatible).

Control Thinking Budget#

The primary launch above already enables the nemotron_v3 reasoning parser, so parsed reasoning is available in choices[].message.reasoning. To use thinking controls, enable thinking and set a thinking-token budget in the request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "messages": [
      {
        "role": "user",
        "content": "I have a 3x3 grid of integers. Rows sum to 15, 18, 21. Columns sum to 12, 20, 22. Center is 7, top-left is 2. Find one valid grid."
      }
    ],
    "max_tokens": 4096,
    "temperature": 0,
    "seed": 42,
    "chat_template_kwargs": {"enable_thinking": true},
    "thinking_token_budget": 2048
  }'

The thinking_token_budget value caps the reasoning portion of generation, and max_tokens caps the combined total (reasoning + visible answer). Size the two values so that the visible answer has room after reasoning: max_tokens >= thinking_token_budget + expected_answer_tokens. In the example above, max_tokens=4096 and thinking_token_budget=2048 leave up to 2048 tokens for the final grid answer even if reasoning uses its full budget.

When the reasoning parser is enabled, the response reports how many of the generated tokens were reasoning tokens in usage.completion_tokens_details.reasoning_tokens (completion_tokens remains the combined total of reasoning tokens and content tokens). If a response returns with finish_reason: "length" and an empty final answer, raise max_tokens or lower thinking_token_budget and retry.

Note

thinking_token_budget is supported on the Chat Completions endpoint (/v1/chat/completions). The Responses endpoint (/v1/responses) does not enforce a thinking-token budget.

Enable Tool Calling and MCP Workflows#

The primary launch already sets --reasoning-parser nemotron_v3. To also enable OpenAI-compatible tool calling, add the auto-tool-choice and tool-call-parser flags to NIM_PASSTHROUGH_ARGS alongside the reasoning parser:

export NIM_PASSTHROUGH_ARGS="--reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder"

If you already use NIM_PASSTHROUGH_ARGS for profile-specific settings, append these arguments to the same string. Both --enable-auto-tool-choice and --tool-call-parser qwen3_coder are required when using "tool_choice": "auto".

The following request provides multiple tool choices and lets the model choose which one to call:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "messages": [
      {
        "role": "user",
        "content": "What is the weather in Santa Clara, CA?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather for a city.",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {"type": "string"}
            },
            "required": ["location"]
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "search_docs",
          "description": "Search internal documentation.",
          "parameters": {
            "type": "object",
            "properties": {
              "query": {"type": "string"}
            },
            "required": ["query"]
          }
        }
      }
    ],
    "tool_choice": "auto",
    "max_tokens": 256
  }'

A successful tool-calling response includes a tool_calls array under choices[0].message. Your application executes the selected tool and sends the tool result back to the model in a follow-up Chat Completions request.

For MCP, connect to MCP servers in your client application, convert the MCP tool schemas to the OpenAI tools format, and pass them to /v1/chat/completions. The NIM container does not connect to MCP servers directly. For details and LangChain/LangGraph examples, refer to Tool Calling and MCP Integration.

Structured JSON Output#

For structured-output use cases, request JSON mode through the OpenAI-compatible response_format parameter and validate the response with your preferred schema library, such as Pydantic:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "messages": [
      {
        "role": "user",
        "content": "Return JSON with keys name and purpose for NVIDIA NIM."
      }
    ],
    "response_format": {"type": "json_object"},
    "max_tokens": 128,
    "temperature": 0.0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

For structured-output use cases, disable reasoning with chat_template_kwargs.enable_thinking: false so the full max_tokens budget produces the JSON payload rather than a reasoning trace followed by a truncated JSON. If you leave reasoning enabled, raise max_tokens to accommodate both the reasoning trace and the JSON output.

Frameworks that support OpenAI-compatible chat completions, such as LangChain, LangGraph, LlamaIndex, Pipecat, OpenCode, and similar agent frameworks, can use the local NIM endpoint by setting their base URL to http://localhost:8000/v1 and using the served model name.

Verify Health Endpoints#

Verify that the NIM container is running by checking its health endpoints. By default, these endpoints are served on port 8000. If you set NIM_HEALTH_PORT, use that port instead.

Live Endpoint#

Check whether the server is running:

curl -v http://localhost:8000/v1/health/live

Example response:

GET /v1/health/live HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "live",
  "status": "live"
}

Ready Endpoint#

Check whether the model is fully loaded and ready for inference:

curl -v http://localhost:8000/v1/health/ready

Example response:

GET /v1/health/ready HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 63
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "ready",
  "status": "ready"
}

Streaming#

To receive responses incrementally as they are generated, enable streaming by adding "stream": true to your request payload. Streaming is supported across the /v1/chat/completions, /v1/completions, /v1/responses, and /v1/messages endpoints.

When streaming is enabled, the API returns a sequence of Server-Sent Events (SSE).

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
    "messages": [
      {
        "role": "user",
        "content": "Write a short poem about a robot."
      }
    ],
    "max_tokens": 100,
    "stream": true,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

The chat_template_kwargs.enable_thinking: false flag disables reasoning for this streaming example so the full max_tokens: 100 budget is spent on the poem tokens. Without it, the model would consume the budget on the reasoning trace first and the stream would end with finish_reason: "length" before producing visible poem text. Omit the flag to keep reasoning enabled, but raise max_tokens to leave room for both the reasoning trace and the final poem. Refer to the reasoning-budget note earlier in this guide.

For Chat Completions and Text Completions, the response streams back in chunks. Each chunk contains a data JSON object. These streams terminate with a data: [DONE] message:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"nvidia/nemotron-3.5-lightning-30b-a3b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"prompt_token_ids":null}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"nvidia/nemotron-3.5-lightning-30b-a3b","choices":[{"index":0,"delta":{"content":"In"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"nvidia/nemotron-3.5-lightning-30b-a3b","choices":[{"index":0,"delta":{"content":" cir"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"nvidia/nemotron-3.5-lightning-30b-a3b","choices":[{"index":0,"delta":{"content":"cuits"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

...

data: [DONE]

For the Responses API, stream events use typed SSE events such as response.output_text.delta and terminate with response.completed. For Anthropic-compatible Messages, stream events use Anthropic event names such as content_block_delta and terminate with message_stop.