Is this page helpful?

Get Started with DeepSeek-V4-Pro-0813#

DeepSeek-V4-Pro-0813 is the official DeepSeek-V4-Pro release. It is a text-only mixture-of-experts (MoE) model with 1.6 trillion total parameters and 49 billion activated parameters per token. The architecture uses hybrid attention that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) and Manifold-Constrained Hyper-Connections (mHC). The checkpoint ships with a DSpark speculative-decoding module in the same weights, so you do not need a separate draft model. The NIM is an SGLang 2.1.2-variant container that exposes OpenAI-compatible APIs for Chat Completions, Text Completions, Responses, tool calling, and agentic workflows.

What NIM adds on top of direct SGLang. Direct SGLang deployments require you to configure tensor-parallel size, GPU memory allocation, the reasoning parser, the tool-call parser, and DSpark speculative decoding. NIM removes that setup and validation burden with one model-specific container image and a tested model and runtime combination. It also provides an automatically selected packaged deployment profile matched to the detected GPU, and a per-precision support matrix that tells you which profiles fit which GPUs before you start. You can still override the packaged profile, tune memory and context length, and pass raw SGLang flags through NIM_PASSTHROUGH_ARGS.

This page is the model-specific guide for DeepSeek-V4-Pro-0813. For the generic NIM LLM and VLM onboarding path, refer to Quickstart. For full endpoint examples, refer to API Reference. For tool calling and MCP integration, refer to Tool Calling and MCP Integration.

Notes#

Use the following items to configure and validate deployments. They describe default behavior and supported scope, not defects in this release.

Reasoning parser and tool-call parser: The NIM defaults both --reasoning-parser and --tool-call-parser to auto. For this model, auto resolves to the DeepSeek V4 parsers (deepseek-v4 and deepseekv4) and both parsers are enabled by default. You do not pass these flags in the primary launch. Parsed reasoning is returned in choices[].message.reasoning_content (not mixed into content), and tool calls are returned in tool_calls. Do not set either parser to a different value unless you intend to disable that behavior.
Eight-GPU requirement: Every packaged profile uses tensor-parallel size 8. Deploy on a single node with eight compatible GPUs. The fallback profile requires more than 1,024 GB of aggregate GPU memory.
DSpark speculative decoding: Packaged H200, B200, and H20-3e throughput profiles enable DSpark speculative decoding by default through the container tuning configuration. You do not pass a separate draft-model path. Refer to Release Notes.
Reasoning effort: The model supports reasoning_effort values of low, high, and max. Reasoning tokens and visible answer tokens share one max_tokens budget. If a response ends before producing a final answer, increase max_tokens, lower reasoning_effort, or simplify the prompt.
JSON mode: response_format: {"type": "json_object"} can suppress visible reasoning output and is recommended for structured responses that you validate with a schema.
Tool calling: "tool_choice": "none" does not advertise tool schemas in the encoded prompt. "tool_choice": "auto" uses the default DeepSeek V4 tool-call parser.
Generation defaults: The checkpoint generation_config.json defaults are temperature=1.0 and top_p=1.0. For agentic workloads, DeepSeek recommends top_p=0.95. Override sampling parameters per request when you need a different value.
Profile verification: Use the Support Matrix as the source of truth for supported GPU, precision, and TP combinations. After startup, verify the selected profile and served model with /v1/metadata, /v1/models, and the startup logs.
Scope: DeepSeek-V4-Pro-0813 is a text-only model. Use it for text reasoning, coding, tool calling, and agentic workflows. It is not a multimodal image, audio, or video model.

Known Issues#

This release includes the following known issue.

FP4 MoE runner: Routed experts in this checkpoint use FP4. Packaged H200, B200, H20-3e, and fallback tuning configs set --moe-runner-backend flashinfer_mxfp4. If you override that backend to the Triton fused-MoE path, startup can fail with AssertionError: Hidden size mismatch. Do not override the packaged MoE runner unless you have a validated alternative.

Prerequisites#

Before deploying a NIM LLM and VLM container, ensure your environment meets the following requirements:

Hardware Requirements#

The following are the minimum required specifications for supported hardware components:

Requirement	Specification
CPU	AMD64, ARM64
GPU	Refer to the Support Matrix

Software Requirements#

Minimum required versions for supported software components.

Requirement	Specification
Operating System	Ubuntu 22.04 LTS or later recommended
Container Toolkit	1.14.0 or later
CUDA SDK	12.9 or later
GPU Driver	580 or later
Docker	24.0 or later

Operating System#

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK#

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers#

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker#

Docker is required to run the containerized NIM services.

Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.
Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:
```
sudo groupadd docker
sudo usermod -aG docker $USER
```
Log out and back in for the group change to take effect.

Container Toolkit#

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

Install the toolkit by following the NVIDIA Container Toolkit installation guide.
Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.
Restart the Docker daemon after configuration:
```
sudo systemctl restart docker
```

NIM Container Access#

To download and deploy NIM containers, you need one of the following:

A free NVIDIA Developer Program membership.
An NVIDIA AI Enterprise license. To request a free 90-day evaluation license, refer to Ways to Get Started With NVIDIA AI Enterprise and Activate Your NVIDIA AI Enterprise License.

Generate Access Credentials#

An NGC Personal API key is not required to access most NVIDIA NIM containers and models hosted on NGC. An NGC API key is only required to download Production Branch (PB) models. This Get Started documentation uses a feature branch model as an example, so creating an NGC Personal API key is optional.

If you want to create an NGC Personal API key, do the following:

Generate the Personal API Key on the Setup API Keys page.
When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM. Always use a Personal API Key.

Verify NVIDIA Runtime Access#

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuration#

Use environment variables to control authentication and model caching.

Export NGC API Key#

Note

An NGC API key is only required to download Production Branch (PB) models. You can skip this section if you did not create an NGC API key.

Export the variable in your shell (temporary), replacing <VALUE> with your actual API key:
```
export NGC_API_KEY=<VALUE>
```

Persist the variable (optional):

If using bash:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc

If using zsh:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc

Verify the variable is set:
```
echo "$NGC_API_KEY"
```

Model Cache#

NIM downloads model weights to a cache on the host that you mount into the container. Artifacts persist across restarts, so you do not pull the full model on every run.

Local Cache#

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions#

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Installation#

Before running a NIM LLM and VLM container, configure your environment and pull the container image. Most current public NIM images support keyless access from NGC, so Docker login is optional unless your image, tag, or model artifact requires credentials.

Docker Login#

If you created an NGC API key, authenticate with the NVIDIA Container Registry using the following command:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Use $oauthtoken as the username and NGC_API_KEY as the password. The $oauthtoken username is a special name that indicates that you will authenticate with an API key and not a user name and password.

Pull the Container Image#

Use the docker pull command to fetch the NIM container image. For eligible public-catalog NIMs, you can pull the image without Docker login. Authenticate first only when you use a Production Branch (PB) NIM, a NIM released prior to NIM LLM version 2.0.10, or a private or gated NIM image or model artifact.

docker pull nvcr.io/nim/deepseek-ai/deepseek-v4-pro-0813:2.1.2-variant

Storage and Startup Notes#

Ensure the host has enough free disk space for the container image and the model cache. Combined container and model disk space is approximately 831 GB. Reserve additional space if you keep older container images on the same host.

The first launch downloads the model artifacts into the mounted cache directory. Download time depends on your hardware and network. Subsequent launches reuse the mounted cache and start faster.

Tip

By default, the model download produces little log output and can appear idle. To follow the download progress, add -e NIM_LOG_LEVEL=INFO to the docker run command.

To pre-populate the cache before serving traffic, first discover the profile hash that matches your GPU and precision, then pass it to download-to-cache with the --profile selector. Passing --profile avoids downloading the wrong profile on hosts where profile auto-detection depends on runtime state.

You can omit the -e NGC_API_KEY=$NGC_API_KEY line from docker run commands when the selected image and model artifacts support keyless access.

List the available profiles for your hardware:
```
docker run --rm --gpus=all \
  -e NGC_API_KEY=$NGC_API_KEY \
  nvcr.io/nim/deepseek-ai/deepseek-v4-pro-0813:2.1.2-variant \
  list-model-profiles
```
Copy the profile hash from the row that matches your GPU, precision, and tensor-parallel size (for example, an H200, B200, or H20-3e FP8 TP=8 profile).

Export the profile hash and download it into the cache:

export NIM_MODEL_PROFILE="<profile-hash-from-step-1>"

docker run --rm --gpus=all \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/deepseek-ai/deepseek-v4-pro-0813:2.1.2-variant \
  download-to-cache -p "$NIM_MODEL_PROFILE"

For details on discovering, selecting, and managing profiles, refer to Model Download.

Run NIM#

DeepSeek-V4-Pro-0813 is a reasoning model with native tool calling. The NIM default for both --reasoning-parser and --tool-call-parser is auto, which selects the DeepSeek V4 parsers and leaves them enabled. Do not add these flags to the run command.

This model requires eight GPUs (TP=8). Increase shared memory so the NVIDIA Collective Communication Library (NCCL) can form the tensor-parallel group:

docker run --gpus=all \
  --shm-size=32g \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_NAME=deepseek-ai/DeepSeek-V4-Pro-0813 \
  -e NIM_SERVED_MODEL_NAME=deepseek-ai/DeepSeek-V4-Pro-0813 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/deepseek-ai/deepseek-v4-pro-0813:2.1.2-variant

With the default parsers enabled, choices[].message.content contains only the final answer, and any reasoning tokens are returned separately in choices[].message.reasoning_content. The rest of this guide assumes this launch.

DeepSeek-V4-Pro-0813 runs on a single node with eight GPUs. Most packaged profiles start successfully with the command above and do not require NIM_PASSTHROUGH_ARGS. For information about how NIM_PASSTHROUGH_ARGS is processed, refer to Advanced Configuration.

Model Identifier#

Every request must send the served model identifier in the model field. The launch command above sets it explicitly with NIM_SERVED_MODEL_NAME, so the identifier matches the Hugging Face model ID: deepseek-ai/DeepSeek-V4-Pro-0813.

Variable	Purpose
`NIM_SERVED_MODEL_NAME`	The identifier the server registers and reports through `/v1/models`. This is what clients send in the `model` field.
`NIM_MODEL_NAME`	The model identity the NIM reports through its metadata endpoints. Keep it identical to `NIM_SERVED_MODEL_NAME`.

Choose any identifier that suits your deployment. Set both variables to the same value and use it in every request. To confirm the served model, run the following:

curl -s http://localhost:8000/v1/models | jq -r '.data[].id'

The returned value is the only one the model field accepts.

Note

These variables control only the identifier the service advertises. They do not change which weights are downloaded. The weights are resolved from the model manifest inside the image.

Setting these variables explicitly keeps your deployment stable across releases. Your clients continue to use the value you chose.

Recommended Runtime Settings#

The following settings cover profiles that need explicit runtime tuning beyond the standard launch command.

Packaged Profile Defaults#

Every packaged profile is FP8 with tensor-parallel size 8. On H200, B200, and H20-3e, the matching throughput tuning config sets mem_fraction_static to 0.90 and enables DSpark (speculative_algorithm: DSPARK). The H20-3e throughput config also sets DeepEP MoE all-to-all, FlashInfer MXFP4 MoE, and chunked prefill.

You do not need to repeat those flags in NIM_PASSTHROUGH_ARGS when the auto-selected profile already applies them. To inspect the selected profile, check the startup logs for the SGLang launch line and the /v1/metadata endpoint.

Context Length#

Packaged profiles do not set --context-length. When those flags are unset, SGLang sizes the served window from remaining GPU memory after the weights load. The result depends on GPU type, memory headroom, mem_fraction_static, and other runtime settings, and is typically well below 1M.

After the container is ready, use the startup logs to determine the window SGLang actually allocated.

To cap the window explicitly and free KV-cache memory for concurrency, pass --context-length in NIM_PASSTHROUGH_ARGS or set NIM_MAX_MODEL_LEN:

export NIM_PASSTHROUGH_ARGS="--context-length 131072"

Note

A larger context window requires significantly more KV cache memory, which reduces the number of requests that can be served concurrently. Validate GPU memory headroom for your target hardware before serving production traffic. For high and max reasoning effort, DeepSeek recommends leaving enough output budget for long traces (up to hundreds of thousands of tokens in the upstream guidance).

Resolve Out-of-Memory Errors#

The packaged H200, B200, and H20-3e throughput profiles have been validated and do not experience an out-of-memory (OOM) error at startup or under the tested load when you run the standard launch on matching hardware. You do not need extra memory flags for those profiles.

Use the following guidance only if you run the fallback profile, or if you replace the packaged runtime with passthrough arguments (NIM_PASSTHROUGH_ARGS, NIM_MAX_MODEL_LEN, and related overrides) instead of the profile defaults. In those cases the container can fail during startup or under load with a CUDA OOM error. The following options reduce the memory required per GPU, in order of preference:

Confirm you have eight GPUs. There is no packaged TP < 8 profile, including fallback. Refer to the support matrix for the profiles available for DeepSeek-V4-Pro-0813.
Cap the context window. If the auto-sized window is too large for remaining memory, set an explicit --context-length so less KV cache is reserved:
```
export NIM_PASSTHROUGH_ARGS="--context-length 131072"
```
Lower GPU memory utilization. If other processes share the GPU, reduce the fraction of GPU memory the NIM reserves so there is more headroom (NIM_KV_CACHE_PERCENT maps to SGLang --mem-fraction-static):
```
export NIM_PASSTHROUGH_ARGS="--mem-fraction-static 0.85"
```

You can combine these settings in a single NIM_PASSTHROUGH_ARGS string. For information about how NIM_PASSTHROUGH_ARGS is processed, refer to Advanced Configuration.

Interact with the API#

The NIM exposes OpenAI-compatible inference endpoints:

Chat Completions: /v1/chat/completions
Text Completions: /v1/completions
Responses: /v1/responses

Tip

Chat Completions, Text Completions, and Responses support streaming.

Important

Reasoning and visible output share one max_tokens budget. DeepSeek-V4-Pro-0813 is a reasoning model. Reasoning tokens and visible answer tokens both count against max_tokens. If the reasoning trace fills the budget first, the response returns with finish_reason: "length" and no final answer.

Follow these rules in every example:

For concise or structured examples where you do not need a long reasoning trace, set "reasoning_effort": "low" so more of the max_tokens budget is available for the visible answer.
For examples that explicitly demonstrate reasoning, size max_tokens large enough for both the reasoning trace and the final answer, and use "reasoning_effort": "high" or "reasoning_effort": "max" as needed.

Send a Chat Completion Request#

After the server is running, you can send a request to the chat completion endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
    "messages": [
      {
        "role": "user",
        "content": "Hello! How are you?"
      }
    ],
    "max_tokens": 128,
    "temperature": 1.0,
    "top_p": 1.0,
    "reasoning_effort": "low"
  }'

The reasoning_effort: "low" setting keeps this introductory example brief so the full max_tokens budget is available for the visible answer. Raise reasoning_effort to high or max for harder problems, but size max_tokens accordingly.

The response has the following general format. The exact generated text can vary by sampling settings and runtime configuration. Because the default reasoning parser is enabled, the reasoning_content field contains the parsed reasoning trace. The field can be empty or null when the request uses a low reasoning effort and the model emits little or no reasoning. If you set the parser to off, the reasoning text appears in content instead.

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 0,
  "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I am doing well, thanks for asking. How can I help you today?",
        "reasoning_content": "We need to answer the user. User says \"Hello! How are you?\". Need to respond in a friendly way."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 53,
    "total_tokens": 63,
    "reasoning_tokens": 34
  }
}

Hold a Multi-Turn Conversation#

The Chat Completions endpoint is stateless. To carry context across turns, resend the full conversation in the messages array, alternating user and assistant turns after an optional system message:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "My favorite color is blue."},
      {"role": "assistant", "content": "Noted — your favorite color is blue."},
      {"role": "user", "content": "What did I just tell you?"}
    ],
    "max_tokens": 128,
    "temperature": 1.0,
    "reasoning_effort": "low"
  }'

Note

Append only the assistant content from prior turns to the history. The reasoning parser is enabled by default; do not feed the reasoning_content field back into messages.

Use the OpenAI Python SDK#

You can direct the OpenAI Python SDK at the NIM endpoint by setting base_url to the local /v1 API path and providing any non-empty API key:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-used",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Pro-0813",
    messages=[{"role": "user", "content": "Summarize GPU computing in one sentence."}],
    max_tokens=128,
    temperature=1.0,
    extra_body={"reasoning_effort": "low"},
)

print(response.choices[0].message.content)

The extra_body={"reasoning_effort": "low"} argument keeps the one-sentence summary short. Omit it or set reasoning_effort to high or max when you want the model to reason longer before answering, and raise max_tokens to leave room for both the reasoning trace and the final answer.

Control Reasoning Effort#

The default launch already enables the DeepSeek V4 reasoning parser, so parsed reasoning is available in choices[].message.reasoning_content. To increase reasoning effort, set reasoning_effort in the request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
    "messages": [
      {
        "role": "user",
        "content": "I have a 3x3 grid of integers. Rows sum to 15, 18, 21. Columns sum to 12, 20, 22. Center is 7, top-left is 2. Find one valid grid."
      }
    ],
    "max_tokens": 4096,
    "temperature": 1.0,
    "reasoning_effort": "high"
  }'

reasoning_effort controls how much deliberation the model spends before answering (low, high, or max). max_tokens still caps the combined total of reasoning tokens and visible answer tokens. If a response returns with finish_reason: "length" and an empty final answer, raise max_tokens or lower reasoning_effort and retry.

Enable Tool Calling and MCP Workflows#

The default launch already enables the DeepSeek V4 reasoning parser and tool-call parser (auto). You do not add parser flags for OpenAI-compatible tool calling. If you set NIM_PASSTHROUGH_ARGS for other runtime options, do not replace the default parsers unless you intend to change that behavior.

The following request provides multiple tool choices and lets the model choose which one to call:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
    "messages": [
      {
        "role": "user",
        "content": "What is the weather in Santa Clara, CA?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather for a city.",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {"type": "string"}
            },
            "required": ["location"]
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "search_docs",
          "description": "Search internal documentation.",
          "parameters": {
            "type": "object",
            "properties": {
              "query": {"type": "string"}
            },
            "required": ["query"]
          }
        }
      }
    ],
    "tool_choice": "auto",
    "max_tokens": 256,
    "reasoning_effort": "low"
  }'

A successful tool-calling response includes a tool_calls array under choices[0].message. Your application executes the selected tool and sends the tool result back to the model in a follow-up Chat Completions request.

For MCP, connect to MCP servers in your client application, convert the MCP tool schemas to the OpenAI tools format, and pass them to /v1/chat/completions. The NIM container does not connect to MCP servers directly. For details and LangChain and LangGraph examples, refer to Tool Calling and MCP Integration.

Structured JSON Output#

For structured-output use cases, request JSON mode through the OpenAI-compatible response_format parameter and validate the response with your preferred schema library, such as Pydantic:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
    "messages": [
      {
        "role": "user",
        "content": "Return JSON with keys name and purpose for NVIDIA NIM."
      }
    ],
    "response_format": {"type": "json_object"},
    "max_tokens": 128,
    "temperature": 1.0,
    "reasoning_effort": "low"
  }'

For structured-output use cases, keep reasoning_effort at low so the max_tokens budget produces the JSON payload rather than a long reasoning trace followed by truncated JSON. If you raise reasoning effort, raise max_tokens to accommodate both the reasoning trace and the JSON output.

Frameworks that support OpenAI-compatible chat completions, such as LangChain, LangGraph, LlamaIndex, Pipecat, OpenCode, and similar agent frameworks, can use the local NIM endpoint by setting their base URL to http://localhost:8000/v1 and using the served model name.

Verify Health Endpoints#

Verify that the NIM container is running by checking its health endpoints. By default, these endpoints are served on port 8000. If you set NIM_HEALTH_PORT, use that port instead.

Live Endpoint#

Check whether the server is running:

curl -v http://localhost:8000/v1/health/live

Example response:

GET /v1/health/live HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "live",
  "status": "live"
}

Ready Endpoint#

Check whether the model is fully loaded and ready for inference:

curl -v http://localhost:8000/v1/health/ready

Example response:

GET /v1/health/ready HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 63
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "ready",
  "status": "ready"
}

Streaming#

To receive responses incrementally as they are generated, enable streaming by adding "stream": true to your request payload. Streaming is supported across the /v1/chat/completions, /v1/completions, and /v1/responses endpoints.

When streaming is enabled, the API returns a sequence of Server-Sent Events (SSE).

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
    "messages": [
      {
        "role": "user",
        "content": "Write a short poem about a robot."
      }
    ],
    "max_tokens": 100,
    "stream": true,
    "reasoning_effort": "low"
  }'

The reasoning_effort: "low" setting keeps this streaming example focused on visible poem tokens. Without it, a long reasoning trace can consume the max_tokens: 100 budget first and the stream can end with finish_reason: "length" before producing visible poem text.

For Chat Completions and Text Completions, the response streams back in chunks. Each chunk contains a data JSON object. These streams terminate with a data: [DONE] message:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"deepseek-ai/DeepSeek-V4-Pro-0813","choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"deepseek-ai/DeepSeek-V4-Pro-0813","choices":[{"index":0,"delta":{"reasoning_content":"Hmm, the user asked"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"deepseek-ai/DeepSeek-V4-Pro-0813","choices":[{"index":0,"delta":{"content":"Steel"},"logprobs":null,"finish_reason":null}]}

...

data: [DONE]

For the Responses API, stream events use typed SSE events such as response.output_text.delta and terminate with response.completed.