Is this page helpful?

Get Started with DeepSeek-V4.1-Flash-1017#

Deploy DeepSeek-V4.1-Flash-1017 using the SGLang Model-Free NIM Experimental Collection. The collection provides a model-free container and a separate serving recipe resource. Supply model weights at runtime and load the recipe for your GPU and topology. For the supported configurations, refer to DeepSeek-V4.1-Flash-1017.

Use container and recipe version 2.1.6-variant-deepseek-v4.1-flash-1017 together. The container does not include model weights. Download the DeepSeek AI checkpoint separately, as shown below.

Prerequisites#

Before deploying the container, ensure your environment meets the following requirements:

Hardware Requirements#

The following are the minimum required specifications for supported hardware components:

Requirement	Specification
CPU	AMD64, ARM64
GPU	Refer to the Support Matrix

Software Requirements#

Minimum required versions for supported software components.

Requirement	Specification
Operating System	Ubuntu 22.04 LTS or later recommended
Container Toolkit	1.14.0 or later
CUDA SDK	12.9 or later
GPU Driver	580 or later
Docker	24.0 or later

Operating System#

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK#

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers#

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker#

Docker is required to run the containerized NIM services.

Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.
Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:
```
sudo groupadd docker
sudo usermod -aG docker $USER
```
Log out and back in for the group change to take effect.

Container Toolkit#

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

Install the toolkit by following the NVIDIA Container Toolkit installation guide.
Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.
Restart the Docker daemon after configuration:
```
sudo systemctl restart docker
```

NIM Container Access#

To download and deploy NIM containers, you need one of the following:

A free NVIDIA Developer Program membership.
An NVIDIA AI Enterprise license. To request a free 90-day evaluation license, refer to Ways to Get Started With NVIDIA AI Enterprise and Activate Your NVIDIA AI Enterprise License.

Generate Access Credentials#

NGC#

An NGC Personal API key is not required to access most NVIDIA NIM containers and models hosted on NGC. An NGC API key is only required to download Production Branch (PB) models. This Get Started documentation uses a feature branch model as an example, so creating an NGC Personal API key is optional.

If you want to create an NGC Personal API key, do the following:

Generate the Personal API Key on the Setup API Keys page.
When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM. Always use a Personal API Key.

Hugging Face#

Create a Hugging Face access token with read permissions (or higher, only if your org or private repos require it). This token allows model-free NIM to fetch models directly from Hugging Face.

Verify NVIDIA Runtime Access#

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Install the NGC CLI to download the serving recipe. Provide disk space for approximately 476 GiB (511 GB) of FP8 weights, in addition to the container and runtime cache.

Configure Authentication and Caching#

Accept any model access terms on Hugging Face before downloading weights.

Export a Hugging Face Access Token#

Export a Hugging Face token to download models from Hugging Face.

Export the variable in your shell (temporary), replacing <token-value> with your actual token:
```
export HF_TOKEN="<token-value>"
```

Optional: Persist the variable in your shell profile for future terminals.

If using bash:

echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc

If using zsh:

echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc

Verify the variable is set:
```
echo "$HF_TOKEN"
```

Export an NGC API Key#

Note

An NGC API key is only required to download Production Branch (PB) models. You can skip this section if you did not create an NGC API key.

Export the variable in your shell (temporary), replacing <VALUE> with your actual API key:
```
export NGC_API_KEY=<VALUE>
```

Optional: Persist the variable in your shell profile for future terminals.

If using bash:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc

If using zsh:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc

Verify the variable is set:
```
echo "$NGC_API_KEY"
```

Model Cache#

NIM downloads model weights to a cache on the host that you mount into the container. Artifacts persist across restarts, so you do not pull the full model on every run.

Local Cache#

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

If you created an NGC API key, authenticate with the NVIDIA Container Registry using the following command:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Use $oauthtoken as the username and NGC_API_KEY as the password. The $oauthtoken username is a special name that indicates that you will authenticate with an API key and not a user name and password.

Choose the Model Source#

To download the FP8 checkpoint from Hugging Face at startup, set:

export NIM_MODEL_PATH="hf://deepseek-ai/DeepSeek-V4.1-Flash-1017"

NIM saves the downloaded files in the mounted cache. To use an existing local checkpoint instead, mount its directory and set NIM_MODEL_PATH to the path inside the container. Refer to Supported Model Sources for details.

Pull the Container and Download the Recipe#

Pull the published SGLang Model-Free NIM container and download the matching DeepSeek-V4.1-Flash-1017 recipe. Both artifacts use version 2.1.6-variant-deepseek-v4.1-flash-1017:

docker pull nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.6-variant-deepseek-v4.1-flash-1017
ngc registry resource download-version "nim/nvidia/deepseek-v4.1-flash-1017-recipe:2.1.6-variant-deepseek-v4.1-flash-1017"
export RECIPE_DIR="$PWD/deepseek-v4.1-flash-1017-recipe_v2.1.6-variant-deepseek-v4.1-flash-1017"
find "$RECIPE_DIR" -type f -name '*.yaml'

Select the recipe for your GPU and topology from the downloaded resource. For supported configurations, refer to the Support Matrix.

For example, to use four B200 GPUs, set:

export RECIPE="$RECIPE_DIR/deepseek-v4.1-flash-1017-fp8-b200-4gpu-tp4-ep4-dspark.yaml"
export GPU_COUNT=4
GPU_DOCKER_ARGS=()
chmod 0644 "$RECIPE"

For another configuration, set RECIPE to its downloaded YAML file and GPU_COUNT to the GPU count required by that configuration in the Support Matrix. Make the selected file readable by the container user with chmod 0644 "$RECIPE".

For GB200, configure an IMEX domain on the host spanning one tray with four GPUs. After selecting the GB200 recipe and setting GPU_COUNT=4, set the additional Docker options required for that configuration:

GPU_DOCKER_ARGS=(-e NVIDIA_IMEX_CHANNELS=0 --ulimit memlock=-1)

For B200, H200, and H20-3e, leave GPU_DOCKER_ARGS empty (GPU_DOCKER_ARGS=()). Run the launch command below in the same Bash shell where you set these options.

All four recipes use the following validated settings:

Recipe setting	Value
`context_length`	`1048576` (1M tokens)
`speculative_algorithm`	`DSPARK` (DSpark speculative decoding)
DSpark block size	`5`
`mem_fraction_static`	`0.75`
`max_running_requests`	`64`
`chunked_prefill_size`	`4096`

For this model, settings in the recipe take precedence over the corresponding NIM_* environment variables. Use the provided, validated recipes without modifying these settings; changes can significantly degrade performance.

Run NIM#

Mount the cache and selected recipe, and load the recipe with NIM_CONFIG_FILE:

docker run --rm --gpus "$GPU_COUNT" --ipc=host --shm-size=32g \
  "${GPU_DOCKER_ARGS[@]}" \
  -e NIM_MODEL_PATH="$NIM_MODEL_PATH" \
  -e HF_TOKEN \
  -e NIM_CONFIG_FILE=/opt/nim/recipe.yaml \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$RECIPE:/opt/nim/recipe.yaml:ro" \
  -p 8000:8000 \
  nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.6-variant-deepseek-v4.1-flash-1017

Keep the selected recipe loaded to reproduce the validated configuration.

Verify Readiness#

Wait for the model to load before sending inference requests. Readiness returns HTTP 200 when the model is ready:

curl -f http://localhost:8000/v1/health/ready
curl -s http://localhost:8000/v1/models

The recipes set served_model_name to deepseek-ai/DeepSeek-V4.1-Flash-1017. Use that identifier in API requests. Setting NIM_SERVED_MODEL_NAME in the launch command does not change the identifier supplied by these recipes.

Send a Chat Completion Request#

Test the running NIM with the recipe’s served model identifier, deepseek-ai/DeepSeek-V4.1-Flash-1017, and an output budget that allows room for reasoning:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017",
    "messages": [{"role": "user", "content": "What is 27 multiplied by 38?"}],
    "max_tokens": 2048
  }'

Reasoning tokens consume the output budget. If the response ends with finish_reason: "length", increase max_tokens and retry.

To receive output incrementally, add "stream": true to the request. For additional request formats, refer to API Reference.

Send an Image Request#

Image input is supported through image_url content parts in Chat Completions requests. For example:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image."},
        {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/0/00/Freiburg_Muenster_Sept_2005_02.jpg"}}
      ]
    }],
    "max_tokens": 2048
  }'

Video and audio content parts are not supported and return HTTP 400.

Send an Anthropic Messages Request#

The SGLang container also exposes /v1/messages. To request thinking, send:

curl http://localhost:8000/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017",
    "max_tokens": 2048,
    "thinking": {"type": "enabled", "budget_tokens": 1024},
    "messages": [{"role": "user", "content": "What is 27 multiplied by 38?"}]
  }'

For tool calling examples, refer to Tool Calling and MCP Integration.