Get Started with Mistral Large 4

This page shows you how to deploy Mistral Large 4 using vLLM Model-Free NIM Experimental. The model-free container downloads model assets from a remote source at startup. For more information, refer to Model-Free NIM. For supported hardware, refer to the Support Matrix.

Mistral Large 4 is a vision-language Mixture-of-Experts model from Mistral. It accepts text and image input, generates text and reasoning output, and supports tool calling and a 1,048,576-token context window.

For the generic NIM LLM and VLM onboarding path, refer to Quickstart. For full endpoint examples, refer to API Reference. For tool calling and MCP integration, refer to Tool Calling and MCP Integration.

Prerequisites

Before deploying a NIM LLM and VLM container, ensure your environment meets the following requirements:

Hardware Requirements

The following are the minimum required specifications for supported hardware components:

Requirement	Specification
CPU	AMD64, ARM64
GPU	Refer to the Support Matrix

Software Requirements

Minimum required versions for supported software components.

Requirement	Specification
Operating System	Ubuntu 22.04 LTS or later recommended
Container Toolkit	1.14.0 or later
CUDA SDK	12.9 or later
GPU Driver	580 or later
Docker	24.0 or later

Operating System

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker

Docker is required to run the containerized NIM services.

1. Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.

2. Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:

   sudo groupadd docker
   sudo usermod -aG docker $USER
   

3. Log out and back in for the group change to take effect.

Container Toolkit

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

1. Install the toolkit by following the NVIDIA Container Toolkit installation guide.

2. Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.

3. Restart the Docker daemon after configuration:

   sudo systemctl restart docker
   

NIM Container Access

To download and deploy NIM containers, you need one of the following:

• A free NVIDIA Developer Program membership.

• An NVIDIA AI Enterprise license. To request a free 90-day evaluation license, refer to Ways to Get Started With NVIDIA AI Enterprise and Activate Your NVIDIA AI Enterprise License.

Generate Access Credentials

An NGC Personal API key is not required to access most NVIDIA NIM containers and models hosted on NGC. An NGC API key is only required to download Production Branch (PB) models. This Get Started documentation uses a feature branch model as an example, so creating an NGC Personal API key is optional.

If you want to create an NGC Personal API key, do the following:

1. Generate the Personal API Key on the Setup API Keys page.

2. When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM. Always use a Personal API Key.

Verify NVIDIA Runtime Access

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuration

Use environment variables to control authentication and model caching.

Hugging Face Authentication

If the model repository requires authentication, obtain a Hugging Face token with read access and accept any model access terms. Export the token before starting the container:

export HF_TOKEN="<your-hugging-face-token>"

Export NGC API Key

Note

An NGC API key is only required to download Production Branch (PB) models. You can skip this section if you did not create an NGC API key.

1. Export the variable in your shell (temporary), replacing <VALUE> with your actual API key:

   export NGC_API_KEY=<VALUE>
   

2. Persist the variable (optional):

   If using bash:

   echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc
   

   If using zsh:

   echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc
   

3. Verify the variable is set:

   echo "$NGC_API_KEY"
   

Model Cache

NIM downloads model weights to a cache on the host that you mount into the container. Artifacts persist across restarts, so you do not pull the full model on every run.

Local Cache

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

To run as a custom user (for example, your host user), pass -u <uid>:0:

docker run --gpus all -u $(id -u):0 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Important

When using -u <uid>, you must include :0 to set GID 0 (for example, -u $(id -u):0). The container’s writable directories are group-owned by GID 0. Without it, the container will fail with PermissionError when writing to cache, config, or log paths.

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Custom Model Source

The experimental vLLM Model-Free NIM container is distributed through NGC. The model weights are supplied separately by Mistral. Select the checkpoint that matches the precision in your serving recipe:

• FP8: mistralai/Mistral-Large-4-1T-A52B

• NVFP4: mistralai/Mistral-Large-4-1T-A52B-NVFP4

Specify a Hugging Face model using the hf:// prefix. Match the checkpoint to the serving recipe you select for your hardware. For FP8:

export NIM_MODEL_PATH="hf://mistralai/Mistral-Large-4-1T-A52B"

For NVFP4:

export NIM_MODEL_PATH="hf://mistralai/Mistral-Large-4-1T-A52B-NVFP4"

For other model locations, refer to supported sources.

NIM downloads the model and caches it in LOCAL_NIM_CACHE. Mount that host directory at /opt/nim/.cache in the container to preserve the weights across container restarts. Provide enough disk space for the full checkpoint and runtime cache files.

Pull the Container Image

Pull the Mistral Large 4 image from vLLM Model-Free NIM Experimental on NGC:

export NIM_LLM_IMAGE="nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.5-mistral-large-4"
docker pull "$NIM_LLM_IMAGE"

Serving Recipe

The container image includes the serving recipes for Mistral Large 4. At startup, NIM matches the checkpoint configuration and the detected GPU to a recipe and applies it, so no recipe download, mount, or NIM_CONFIG_FILE setting is required. The applied settings for each GPU are listed in the Support Matrix. Configurations without a recipe use the default profile.

Each built-in profile targets one GPU product by its PCI device ID. To check which profile NIM selects, run list-model-profiles with the same NIM_MODEL_PATH before serving:

docker run --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -e NIM_MODEL_PATH="$NIM_MODEL_PATH" \
  -e HF_TOKEN \
  "$NIM_LLM_IMAGE" \
  list-model-profiles

The profile for your configuration, for example vllm-nvidia-b200-fp8-tp8-pp1-throughput-2901:10de-8-1, should appear under Compatible with system and runnable. If your GPU is a variant of a supported GPU with a different device ID, that profile is listed under Incompatible with system and NIM falls back to a generic profile that does not serve this model. Pin the built-in profile by setting NIM_MODEL_PROFILE to the profile ID shown in the listing. The ID depends on NIM_MODEL_PATH, so read it from your own listing.

Full Context and Decode Context Parallelism

The H200 NVFP4 and B200, GB200, and VR200 FP8 configurations use TP8 and decode context parallelism 8. The MLA KV cache is not sharded by TP alone: a full-context sequence needs about 46 GiB of KV cache on every rank without DCP. The built-in recipe sets DCP to distribute that cache across the TP ranks. Do not override it. The B200 and GB200 NVFP4 configurations serve the full context with TP8 and no DCP.

The H200 NVFP4 configuration enables Marlin emulation and sets GPU memory utilization to 0.90. Increasing GPU memory utilization without DCP can still fail at startup or run out of memory during prefill. Refer to the known limitation.

Run NIM

The following example uses eight GPUs on one node. Set NIM_MODEL_PATH to the checkpoint before running it:

docker run --rm \
  --gpus 8 \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -p 8000:8000 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -e NIM_MODEL_PATH="$NIM_MODEL_PATH" \
  -e NIM_SERVED_MODEL_NAME="mistralai/Mistral-Large-4-1T-A52B" \
  -e HF_TOKEN \
  "$NIM_LLM_IMAGE" \
  nim-serve

On first start, NIM downloads the checkpoint into LOCAL_NIM_CACHE. Later starts reuse the cached copy.

For GB200 (FP8 or NVFP4) or VR200 FP8, use two nodes with four GPUs each. Set NIM_NUM_COMPUTE_NODES=2 on both nodes, start the head with nim-serve-ray --head, and join the worker with nim-serve-ray --head-address. Follow Multi-Node Deployment for networking, mounts, and the complete head and worker commands.

Override a Recipe Setting

Settings resolve in the following order, from lowest to highest priority: the tuning config file, environment variables, NIM_PASSTHROUGH_ARGS, and nim-serve command-line options. To change one value, pass it on the command line, for example nim-serve --max-num-seqs 128. To replace the built-in recipe entirely, mount your own recipe file read-only and set NIM_CONFIG_FILE to its path in the container.

Model Identifier

Every request must send the served model identifier in the model field. The launch command sets it explicitly with NIM_SERVED_MODEL_NAME, so the identifier is mistralai/Mistral-Large-4-1T-A52B.

Interact with the API

The NIM exposes OpenAI-compatible inference endpoints:

• Chat Completions: /v1/chat/completions

• Text Completions: /v1/completions

• Responses: /v1/responses

For full request and response details, refer to API Reference.

Requests follow the OpenAI-compatible API. To set a random seed, use seed; the Mistral-specific random_seed field is silently ignored. To require a tool call, use tool_choice: "required"; the Mistral-specific value "any" is not supported.

Verify Health Endpoints

Check whether the server is running:

curl -v http://localhost:8000/v1/health/live

Check whether the model is fully loaded and ready for inference:

curl -v http://localhost:8000/v1/health/ready

By default, these endpoints use port 8000. If you set NIM_HEALTH_PORT, use that port instead. Wait until the ready endpoint returns HTTP 200 before sending inference requests.

Send a Chat Completion Request

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-Large-4-1T-A52B",
    "messages": [{"role": "user", "content": "How many prime numbers are there below 30?"}],
    "reasoning_effort": "high",
    "max_tokens": 8192
  }'

Set reasoning_effort to "low", "medium", or "high" to control reasoning effort. The default is "high". The example above sets it explicitly; change it to "low" or "medium" to request a lower reasoning effort. The model always reasons: "none" is not supported and returns HTTP 400. The response returns the reasoning in reasoning and the answer in content. Reasoning tokens count toward max_tokens.

Enable Tool Calling

The following example uses tool_choice: "required" to require a tool call. Use "auto" to let the model decide whether to call a tool.

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-Large-4-1T-A52B",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get the current weather for a city",
      "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
    }}],
    "tool_choice": "required",
    "max_tokens": 8192
  }'

Send an Image with a Text Prompt

Include both text and image_url content parts in the request. The image URL must be reachable from the container:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-Large-4-1T-A52B",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Describe this image."},
      {"type": "image_url", "image_url": {"url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"}}
    ]}],
    "max_tokens": 8192
  }'

Streaming

To receive responses incrementally as they are generated, set stream to true in the request payload. The API returns Server-Sent Events (SSE):

curl -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-Large-4-1T-A52B",
    "messages": [{"role": "user", "content": "Write a short poem about a robot."}],
    "max_tokens": 8192,
    "stream": true
  }'

Chat Completions and Text Completions streams end with a data: [DONE] message. Responses API streams use typed events such as response.output_text.delta and end with response.completed.