> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.6.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.6.1: https://docs.nvidia.com/dynamo/v1.6.1/llms.txt > - v1.6.0: https://docs.nvidia.com/dynamo/v1.6.0/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Mistral Large 4 Each target below is a Dynamo + vLLM deployment of Mistral Large 4, a 1,051B-parameter mixture-of-experts model (54B active) with DSpark speculative decoding and a 1M-token context, served from an NVFP4 checkpoint. Pick your GPU and serving topology; the details below update with the selection. Choose your deployment target GPU GB300 Vera Rubin Topology Aggregated Disaggregated **Precision** NVFP4 weights, FP8 KV **GPUs** 8x GB300 **Parallelism** 2 x TP4 workers **Routing** KV-aware, hard session affinity **Speculative decoding** DSpark, 3 draft tokens **KV cache** Native CPU KV offload (200 GiB per replica), hybrid KV manager off **Precision** NVFP4 weights, FP8 KV **GPUs** 16x GB300 (2P2D) **Parallelism** TP4 per role, decode expert-parallel **Routing** KV-aware, hard session affinity **Speculative decoding** DSpark, 3 draft tokens, both roles **KV cache** MultiConnector: NIXL + native CPU KV offload, hybrid KV manager off **Precision** NVFP4 weights, NVFP4 KV **GPUs** 4x VR200 (1 worker) **Parallelism** TP4 **Routing** Default router (single worker) **Speculative decoding** DSpark, 3 draft tokens **KV cache** Native CPU KV offload, hybrid KV manager off **Precision** NVFP4 weights, NVFP4 KV **GPUs** 8x VR200 (1P1D) **Parallelism** TP4 per role **Routing** KV-aware, hard session affinity **Speculative decoding** DSpark, 3 draft tokens, both roles **KV cache** MultiConnector: NIXL + CPU KV offload on both roles, hybrid KV manager off ## Prerequisites * A Kubernetes cluster with the Dynamo platform installed (`nvidia.com/v1beta1` operator) and the GPUs for the selected target. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). * The NVIDIA DRA driver with ComputeDomain support (multi-node NVLink) on GB300 and Vera Rubin racks. * A `shared-model-cache` PersistentVolumeClaim that holds the NVFP4 checkpoint (`params.json`, `tekken.json` and all weight shards in one directory). `recipes/mistral-large-4/model-cache/` holds the PVC and a Job that downloads the checkpoint from Hugging Face into `/shared-model-cache/mistral-large-4-nvfp4`; the Job pulls `mistralai/Mistral-Large-4-1T-A52B-NVFP4`, so create an `hf-token-secret` first. ## Deploy Apply the manifest for the selected target: ```bash kubectl apply -f recipes/mistral-large-4/vllm/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/mistral-large-4/vllm/disagg-gb300-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/mistral-large-4/vllm/agg-vr-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/mistral-large-4/vllm/disagg-vr-agentic/deploy.yaml -n ${NAMESPACE} ``` The first launch loads the weights and, on the Vera Rubin targets, compiles the MoE kernels at first use; give it tens of minutes. Keep the model-cache and JIT-cache mounts persistent across restarts. The recipes run with the vLLM hybrid KV cache manager off. The published `deploy.yaml` recipes are generic base recipes agnostic to cluster-specific settings. Adapt to your cluster scheduling, tolerations, and networking by using [kustomizations](https://github.com/ai-dynamo/dynamo/tree/main/recipes/templates/kustomize). ## Smoke Test Forward the frontend and send a request. All recipes serve the model as `mistral-large-4`: ```bash kubectl port-forward svc/-frontend 8000:8000 -n ${NAMESPACE} curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"mistral-large-4","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}' ``` Tool calling and reasoning use the Mistral parsers on the workers and the Dynamo chat processor on the frontend. ## Runtime image All targets use `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.7.0-mistral-large-4-dev.1`, with `runtimeVersionOverride: "1.6.0"`. It carries the released `mistral-common` 1.13.0 that the checkpoint's `tekken.json` reasoning-effort levels require and, for Vera Rubin, the sm_107 FlashInfer MoE and sparse-indexer kernels. ## Vera Rubin settings * **KV cache.** `nvfp4_ds_mla` stores the latent KV in NVFP4; it changes numerics, so re-run an accuracy check after any model or image change. Both roles of a disaggregated deployment must use the same dtype. * **Host KV tier.** Aggregated uses `--kv-offloading-size`. Disaggregated wraps the `OffloadingConnector` in a `MultiConnector` next to NIXL on both roles; setting only `--kv-offloading-size` replaces NIXL and silently disables the handoff. * **Disaggregated transfer.** `UCX_CUDA_IPC_ENABLE_MNNVL=y` with `UCX_TLS=cuda_copy,cuda_ipc,tcp` moves KV over NVLink; place both workers in one NVL72 rack. `NIXL_TELEMETRY_ENABLE=y` is required because the operator injects `n`, which makes some vLLM versions fail every transfer. * **Stability.** `VLLM_USE_BREAKABLE_CUDAGRAPH=0`, `DYN_HEALTH_CHECK_ENABLED=false` (the health canary can time out under very long prefills) and `disable_eagle_block_drop` (DSpark counts as eagle and would otherwise drop the trailing prefix block). * **Tool calls.** Workers run `--dyn-tool-call-parser mistral --dyn-reasoning-parser mistral` and the frontend sets `DYN_PARSER_VERSION=2`; keep the Dynamo chat processor on the disaggregated frontend, which also uses `fastokens`. ## GB300 settings * **KV cache dtype.** FP8 latent KV, which is the checkpoint default (`dsa.fp8_kv_cache` in `params.json`); the GB300 recipes do not set `--kv-cache-dtype`. * **Hybrid KV cache manager off.** Both topologies run `--disable-hybrid-kv-cache-manager`. The DSpark draft layers use sliding-window MLA attention; with the manager on, the target and draft layers form two KV cache groups and a prefix hit needs both. With it off they share one group, so a continuation turn can reuse the cached prefix. * **Host KV tier.** Aggregated workers use native CPU offload (`--kv-offloading-size 200`, per replica). Disaggregated workers wrap the `OffloadingConnector` in a `MultiConnector` next to NIXL on both roles; setting only `--kv-offloading-size` would replace NIXL and silently disable the handoff. * **Disaggregated transfer.** Prefill is the NIXL producer and decode the consumer, over UCX with `UCX_CUDA_IPC_ENABLE_MNNVL=yes`, so KV moves over NVLink inside one NVL72 rack. The deployment creates one `ComputeDomain` (`numNodes: 0`) whose channel claim every worker requests. Prefill and decode must use the same block size (64) and speculative configuration. * **Roles.** Prefill runs without CUDA graphs and with synchronous scheduling; decode adds expert parallelism and chunked prefill with a small batched-token budget. * **Resources.** Aggregated workers request 4 GPUs, 64 CPUs and 704 GiB of memory with 264 GiB of shared memory. Disaggregated workers request 4 GPUs, 64 CPUs and 840 GiB with 640 GiB of shared memory. Shared memory and offload both draw on the worker's host memory budget. * **Routing and parsing.** The frontend uses KV routing with hard session affinity, the Dynamo chat processor with `DYN_PARSER_VERSION=2` and `--tokenizer fastokens`. Workers run `--dyn-tool-call-parser mistral --dyn-reasoning-parser mistral`. ## Notes * **Session affinity.** Clients send `x-dynamo-session-id` so the KV router can pin a conversation to a worker. Keep the Dynamo chat processor; the vLLM processor drops this header. * **Speculative decoding.** The recipes use real DSpark acceptance. A synthetic-acceptance alternative is commented next to the flag for performance benchmarking only; never use it for accuracy evaluation. On disaggregated deployments, set it on both roles or neither. * **Disaggregated KV transfer.** Prefill and decode must share block size and speculative configuration. * **Concurrent deployments.** Never run two `DynamoGraphDeployment`s in one namespace where one name is a prefix of the other; the frontend cross-routes requests between them. ## Benchmark `recipes/mistral-large-4/perf/` holds an AIPerf Job that replays the AgentX agentic trace (`inferencex-agentx-mvp`) against the deployment with a 60-minute measured window after the warmup phase (allowed up to 30 minutes). Set `ENDPOINT`, `CONCURRENCY` and the `podAffinity` DGD name for the deployed recipe, then apply it: ```bash kubectl apply -f recipes/mistral-large-4/perf/perf.yaml -n ${NAMESPACE} kubectl logs -f job/mistral-large-4-agentx-aiperf -n ${NAMESPACE} ``` Run several concurrency values; the shipped gate is decode tok/s/user P50 above 50 and TTFT P50 below 2 seconds. See the [perf README](https://github.com/ai-dynamo/dynamo/blob/main/recipes/mistral-large-4/perf/README.md) for artifacts and options. ## Source * Model recipe: [recipes/mistral-large-4](https://github.com/ai-dynamo/dynamo/tree/main/recipes/mistral-large-4) - Selected recipe: [GB300 agg deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/mistral-large-4/vllm/agg-gb300-agentic/deploy.yaml) * Selected recipe: [GB300 disagg deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/mistral-large-4/vllm/disagg-gb300-agentic/deploy.yaml) - Selected recipe: [Vera Rubin (VR200) agg deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/mistral-large-4/vllm/agg-vr-agentic/deploy.yaml) * Selected recipe: [Vera Rubin (VR200) disagg deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/mistral-large-4/vllm/disagg-vr-agentic/deploy.yaml)