Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for this NIM LLM and VLM release.

Release 2.1.5#

Highlights#

This release delivers NIM updates, a new model, profile selection changes, and upgrades the inference backends to vLLM 0.31.0 and SGLang 0.5.21.

NIM Updates#

This release includes updated 2.1.5 versions of the following NIM containers. The following table lists them by NIM type and category:

Updated NIM Containers#

NIM Type

Category

NIMs

Model-specific NIM

LLM

deepseek-v4-flash-0731, deepseek-v4-pro-0813, glm-5.2, gpt-oss-120b, gpt-oss-20b, llama-3.1-70b-instruct, llama-3.1-8b-instruct, llama-3.3-70b-instruct, llama-3.3-nemotron-super-49b-v1.5, nemotron-3-nano, nemotron-3-super-120b-a12b, nemotron-3-ultra-550b-a55b, nemotron-3.5-lightning-30b-a3b, starcoder2-7b

VLM

gemma-4-26b-a4b-it, gemma-4-31b-it, inkling, kimi-k2.6, mistral-small-4-119b-2603, nemotron-3-nano-omni-30b-a3b-reasoning, nemotron-3.5-content-safety, nemotron-parse-v2.0, qwen3.5-122b-a10b, qwen3.5-397b-a17b

Model-free NIM

vLLM backend

apriel-nemotron-15b-thinker, codestral-2501, gpt-oss-20b, mistral-medium-3.5-128b

SGLang backend

glm-5.3

Each model-specific NIM includes validated quantization profiles and optimized runtime configurations for curated model weights. Refer to the Support Matrix for NIMs for each NIM’s supported profiles and verified GPUs, and use the filter at the top of that page to find the profiles for your GPU.

The model-free NIMs are served by the nvidia/vllm-model-free-nim container, which uses vLLM 0.31.0, and the nvidia/sglang-model-free-nim container, which uses SGLang 0.5.21. Each container is validated with the models that the table lists for its backend, and the containers include throughput tuning configurations for every listed model except gpt-oss-20b. To serve codestral-2501, use a single GPU with at least 80 GB of memory. Refer to About Model-Free NIM for deployment instructions and to the Model-Free NIM section of the support matrix for the validated models and tuning configurations, including codestral-2501 and glm-5.3.

New Models#

This release adds the following model:

  • Kimi K3.1 Pro is served by the experimental vLLM model-free NIM container, nvidia/vllm-model-free-nim-experimental, with the public MXFP4 checkpoint and 2.1.6-variant-kimi-k3.1-pro serving recipes. Refer to Kimi K3.1 Pro for supported configurations, and to Kimi K3.1 Pro for deployment requirements.

Tuned Profiles for Fine-Tuned Checkpoints and Model Stores#

When NIM_MODEL_PATH points to a fine-tune of a model-specific NIM’s model or to a model store from create-model-store, the NIM now serves its own profiles. It uses the profiles for the checkpoint’s precision and applies their tuned engine arguments. Previously, the NIM replaced its profiles with generic profiles, so the tuned engine arguments were not applied. This change also applies to model stores that earlier releases created.

The NIM handles these checkpoints as follows:

  • The NIM reads the checkpoint’s precision from config.json and, for ModelOpt checkpoints, from hf_quant_config.json. If the NIM has no profile for that precision, it serves generic profiles, as in earlier releases.

  • Profile names and IDs are the same as with NVIDIA’s weights, so NIM_MODEL_PROFILE accepts them. To serve the profile that a model store was created from, set NIM_MODEL_PROFILE to the profile ID that you passed to create-model-store. If an earlier release created the model store, that profile ID might have changed in this release. In that case, set NIM_MODEL_PROFILE to the 2.1.5 profile ID or name that list-model-profiles prints. Refer to Support Changes.

  • Without NIM_MODEL_PROFILE, the NIM selects one of its profiles for your hardware, as it does for NVIDIA’s weights. The selected profile can differ from the profile that a model store was created from.

  • NIM_MODEL_PATH must point to a fine-tune of the NIM’s own model. The NIM does not check the model architecture.

Refer to Serving a Fine-Tuned Checkpoint on a Model-Specific NIM and Air-Gap Deployment. With some NVFP4 checkpoints, such as mixed-precision checkpoints and checkpoints without a config.json file, the NIM cannot serve its NVFP4 profiles through NIM_MODEL_PATH. Refer to the known issues for nemotron-3-ultra-550b-a55b, deepseek-v4-flash-0731, deepseek-v4-pro-0813, and mistral-small-4-119b-2603.

Advisory Memory Estimates in Profile Selection#

GPU memory estimates are advisory in profile selection. When no profile is estimated to fit in GPU memory, NIM_MODEL_PROFILE=default selects a hardware-compatible profile, and NIM LLM and VLM attempts to start it.

For single-node vLLM and SGLang deployments that use tensor or pipeline parallelism, profile selection works as follows:

  • NIM LLM and VLM still prefers profiles that are estimated to fit in GPU memory. When no profile is estimated to fit, automatic selection and NIM_MODEL_PROFILE=default select a profile that passes the hardware, precision, topology, and feature checks.

  • When the selected profile is estimated not to fit in GPU memory, or its estimate is unavailable, NIM LLM and VLM logs a warning before it downloads the model weights. It then attempts to start the profile. This applies to automatic selection, to NIM_MODEL_PROFILE=default, and to an explicit profile ID or name.

  • NIM LLM and VLM does not shorten the context length automatically.

  • If you do not change a profile’s memory settings, its memory estimate is at least the minimum GPU memory that the profile declares.

  • Hardware, precision, topology, feature, and configuration checks still apply, and the inference backend can still run out of memory at startup.

Multi-node deployments and deployments that use additional data or expert parallelism keep their existing memory checks. Refer to Model Profiles and Selection and Troubleshooting GPU Memory Out-of-Memory Errors.

vLLM 0.31.0#

The vLLM inference backend is updated from 0.30.0 to vLLM 0.31.0. This version includes the following changes that can affect your deployment:

  • When you disable CUDA graphs with --enforce-eager or NIM_DISABLE_CUDA_GRAPH=1, vLLM also skips the startup JIT kernel warmup. Triton kernels then compile during the first requests, so those requests take longer.

  • A LoRA adapter whose name matches the served model name is rejected when it is loaded. Do not give an adapter directory the same name as the served model (NIM_SERVED_MODEL_NAME).

  • Per-request multimodal arguments, media_io_kwargs and mm_processor_kwargs, are disabled by default. Requests that set them return HTTP 400 unless the server runs with --trust-request-mm-kwargs. The qwen3.5-122b-a10b, qwen3.5-397b-a17b, and nemotron-3-nano-omni-30b-a3b-reasoning NIMs enable this option in all of their profiles, so they accept these arguments as in release 2.1.4. Refer to Security Fixes.

  • The slow tokenizer mode (--tokenizer-mode slow) is removed.

  • The outlines structured-output backend rejects json_object requests.

  • When the server runs with --enable-scale-out, a /inference/v1/generate request that sets max_tokens to null and min_tokens above the context length minus the prompt length is rejected. Previously, such a request could stop vLLM from processing new requests.

  • The --enable-mamba-fine-grained-prefix-cache argument is renamed to --enable-mamba-shared-prefix-checkpoint.

  • The --collect-detailed-traces argument no longer accepts a comma-separated list. Pass the modules as separate values, such as --collect-detailed-traces model worker.

  • The VLLM_PLE_CPU_OFFLOAD environment variable is removed. Instead of VLLM_PLE_CPU_OFFLOAD=0, pass --engram-config '{"cpu_offload": false}' to keep Engram embedding tables in GPU memory.

If you pass the new vLLM --log-level or --logging-config argument, NIM LLM and VLM logs a warning and ignores it. Use NIM_LOG_LEVEL and NIM_JSONL_LOGGING to configure logging. Refer to Advanced Configuration.

SGLang 0.5.21#

The SGLang inference backend is updated from 0.5.20 to SGLang 0.5.21. This version includes the following changes that can affect your deployment:

  • The Rust radix tree core is the default radix cache implementation, and the legacy radix caches for sliding window attention (SWA) and Mamba models are removed. This change affects hybrid SWA models, such as Inkling. To use the Python implementation of the radix tree core, set SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python.

  • The FlashInfer fused finalize step for mixture-of-experts (MoE) layers is off by default, which can change NVFP4 MoE performance. To restore it, set SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=1.

  • A /v1/chat/completions request that sets chat_template inside chat_template_kwargs returns HTTP 400 unless the server runs with --trust-request-chat-template. Other chat_template_kwargs keys, such as enable_thinking, do not cause this error.

  • JSON schema keywords that XGrammar cannot enforce now cause the request to fail instead of being ignored. These keywords include if, then, else, not, multipleOf, uniqueItems, contains, and unsupported string format values.

  • In non-streaming responses, tool_calls[].index is the position of the call in the response, as it is in streaming responses. Previously, it could be the position of the tool in the request’s tools list instead.

Security Fixes#

This release fixes 81 unique security issues across the vLLM and SGLang images (3 critical, 18 high, 59 medium, and 1 low).

vLLM Images#

This release fixes 62 security issues in the vLLM images (3 critical, 15 high, 43 medium, and 1 low).

  • The Go standard library and the golang.org/x/net module that are compiled into Mooncake’s libetcd_wrapper.so are updated from Go 1.26.8 to 1.26.9 and from v0.58.0 to v0.60.0, addressing 13 issues (2 critical, 9 high, and 2 medium).

  • lmcache is updated from 0.5.5 to 0.5.6, addressing a critical issue. The flaw allows unauthenticated remote code execution through the network transport of the LMCache multiprocess server, which NIM LLM and VLM does not start.

  • aiohttp is updated from 3.14.3 to 3.14.5, and the copy that Ray vendors is updated from 3.14.3 to 3.14.4, addressing 8 issues (3 high, 4 medium, and 1 low).

  • vLLM is updated from 0.30.0 to 0.31.0, addressing 3 issues (2 high and 1 medium). All three concern multimodal requests, and CVE-2026-114053 applies only when the server runs with --trust-remote-code. The fixes for CVE-2026-114053 and CVE-2026-114055 reject per-request multimodal arguments unless the server runs with --trust-request-mm-kwargs. All profiles of qwen3.5-122b-a10b, qwen3.5-397b-a17b, and nemotron-3-nano-omni-30b-a3b-reasoning enable this option, so clients of these NIMs can still set these arguments. nemotron-3-nano-omni-30b-a3b-reasoning runs with --trust-remote-code in all of its profiles, and some profiles of the Qwen3.5 NIMs also run with it. Serve these NIMs only to trusted clients.

  • The FFmpeg libraries that PyAV and PyNvVideoCodec use are rebuilt from FFmpeg 9.0.2 with the upstream fix that limits RTSP redirects, addressing a high-severity issue.

  • Other packages are updated or removed to address medium-severity issues:

    • PyJWT, from 2.14.0 to 2.15.1. The Ubuntu python3-jwt package (2.7.0-1ubuntu0.1), an older copy of PyJWT, is removed. These changes address 17 issues.

    • libexpat1 and related packages, from 2.6.1-2ubuntu0.5 to 2.6.1-2ubuntu0.6 (11 issues).

    • curl and related packages, from 8.5.0-2ubuntu10.13 to 8.5.0-2ubuntu10.15 (3 issues).

    • sudo, from 1.9.15p5-3ubuntu5.24.04.2 to 1.9.15p5-3ubuntu5.24.04.4 (2 issues).

    • multidict, the copy that Ray vendors, from 6.7.1 to 6.9.1 (1 issue).

    • uv, from 0.12.17 to 0.12.22 (1 issue).

    • nginx, from 1.31.3 to 1.31.6 (1 issue).

Show Fixed Security Issues

SGLang Images#

This release fixes 60 security issues in the SGLang images (2 critical, 16 high, 41 medium, and 1 low).

  • The Go standard library and the golang.org/x/net module that are compiled into Mooncake’s libetcd_wrapper.so are updated from Go 1.26.8 to 1.26.9 and from v0.58.0 to v0.60.0, addressing 13 issues (2 critical, 9 high, and 2 medium).

  • nltk is removed, addressing 2 high-severity issues. It was installed only as a dependency of cosmos-guardrail, which is also removed. NIM LLM and VLM does not use either at runtime.

  • The copy of aiohttp that Ray vendors is updated from 3.14.3 to 3.14.4, addressing 8 issues (3 high, 4 medium, and 1 low).

  • oauthlib is updated from 3.3.1 to 4.0.0, addressing 3 issues (1 high and 2 medium).

  • The FFmpeg libraries that PyAV uses to decode audio input are rebuilt from FFmpeg 9.0.2 with the upstream fix that limits RTSP redirects, and the static FFmpeg 4.2.2 binary that imageio-ffmpeg bundles is removed. These changes address a high-severity issue.

  • Other packages are updated to address medium-severity issues:

    • PyJWT, from 2.14.0 to 2.15.1 (2 issues).

    • libexpat1 and related packages, from 2.6.1-2ubuntu0.4 to 2.6.1-2ubuntu0.6 (23 issues).

    • curl and related packages, from 8.5.0-2ubuntu10.13 to 8.5.0-2ubuntu10.15 (3 issues).

    • libxml2, from 2.9.14+dfsg-1.3ubuntu3.8 to 2.9.14+dfsg-1.3ubuntu3.10 (1 issue).

    • libevent-core-2.1-7t64 and related packages, from 2.1.12-stable-9ubuntu2.1 to 2.1.12-stable-9ubuntu2.2 (1 issue).

    • multidict, the copy that Ray vendors, from 6.7.1 to 6.9.1 (1 issue).

    • uv, from 0.12.16 to 0.12.21 (1 issue).

    • nginx, from 1.31.3 to 1.31.6 (1 issue).

Show Fixed Security Issues

Bug Fixes#

This release includes the following fixes:

  • The inference backends in the vLLM and SGLang images now load cuDNN 9.27.0.42. Previously, cuDNN could release the code of a runtime-compiled kernel while a captured CUDA graph still used it. Replaying the graph could then return incorrect output without reporting an error, or fail with CUDA error 715.

  • On the vLLM images, EAGLE3 speculative decoding now supports draft models that use only the final hidden state of the target model, that is, drafts whose config.json sets use_aux_hidden_state to false in eagle_config. Previously, the inference backend ignored this setting, and these drafts failed with a hidden-state size mismatch. Refer to Speculative Decoding.

  • On systems with unified GPU and CPU memory, such as NVIDIA DGX Spark, the inference backend can now use up to 75% of unified memory instead of 50%. An explicit --gpu-memory-utilization value still takes precedence.

  • When NIM_MODEL_PATH points to a model store that create-model-store created from a model-specific NIM, the NIM now serves its own profiles with their tuned engine arguments. NIM_MODEL_PROFILE also accepts the profile ID that the model store was created from. Previously, the NIM served generic profiles from the model store and rejected that profile ID. Without the tuned engine arguments, some deployments ran out of GPU memory at startup. Refer to Air-Gap Deployment.

  • On the vLLM images, model downloads now retry when the source responds with HTTP 429 or returns a transient error, such as a server error or a reset connection. Previously, a download of a large model from Hugging Face could fail with an HTTP 429 error. The retries apply to Hugging Face, NGC, Amazon S3, Google Cloud Storage, ModelScope, and HTTPS sources. Each rate-limited download operation can now wait up to 15 minutes before it fails. Startup can run about three of these operations, so retries can delay startup by up to about 45 minutes. Allow for this time and the model load time in your container startup probes. The SGLang images already included these retries in release 2.1.4. In this release, a download from Amazon S3 can still fail without a retry when a connection drops during the transfer. Refer to the known issues for gpt-oss-120b and mistral-small-4-119b-2603.

  • When NIM_DISABLE_WORKSPACE_HASH_CHECK is set to true, NIM LLM and VLM now logs a SECURITY warning that names the variable and the profile whose model files are not verified. Previously, NIM LLM and VLM skipped the check without logging it.

Support Changes#

This release changes support for the following configurations:

  • The vllm-nvfp4-tp4-pp1 profile of mistral-small-4-119b-2603 is not supported on NVIDIA B300-SXM6-AC and NVIDIA GB200 in this release, and these combinations are omitted from the Support Matrix for NIMs. Release 2.1.4 listed these combinations with a known issue. On these GPUs, sustained chat traffic can delay /v1/health/ready responses beyond a five-second probe timeout. The profile remains supported on the other GPUs that the Support Matrix for NIMs lists for it. Refer to the known issue.

  • For nemotron-3-nano-omni-30b-a3b-reasoning, the generic NVFP4 TP1 profile (vllm-nvfp4-tp1-pp1) is not listed for NVIDIA RTX PRO 4500 Blackwell Server Edition in the Support Matrix for NIMs for this release. Release 2.1.4 listed this combination. On this GPU, use the default profile selection, which selects the throughput-optimized NVFP4 TP1 profile 642f356201892f339146d30c52e92ada94de23ac0914583dae769f3f15b84c51. Refer to the known issue.

  • For nemotron-3-super-120b-a12b, the BF16 TP4 LoRA profile (vllm-bf16-tp4-pp1-lora) is supported on NVIDIA H100-NVL in this release. Release 2.1.4 did not list this combination. On this GPU, LoRA was validated with --max-lora-rank set to 64 and the default --max-loras value. Refer to the known issue.

  • The Support Matrix for NIMs no longer lists the following profile and GPU combinations. Release 2.1.4 listed them, but they were not validated:

    • vllm-bf16-tp2-pp1-lora of llama-3.3-70b-instruct on NVIDIA H100-NVL.

    • vllm-fp8-tp2-pp1, vllm-fp8-tp4-pp1, and vllm-fp8-tp8-pp1 of mistral-small-4-119b-2603 on NVIDIA H200-NVL.

    • vllm-bf16-tp1-pp1 of starcoder2-7b on NVIDIA A100-80GB-PCIe.

  • Optimized profiles changed for the following NIMs. The Support Matrix for NIMs lists the optimized profiles of each NIM:

    • nemotron-3.5-lightning: the throughput profiles on NVIDIA B200 and NVIDIA H200 are BF16 TP1. Release 2.1.4 used NVFP4 on NVIDIA B200 and NVFP4 (W4A16) with FP8 on NVIDIA H200.

    • gemma-4-26b-a4b-it: this release has no optimized profile for NVIDIA GB200, and a BF16 TP1 throughput profile is new on NVIDIA H200.

    • qwen3.5-122b-a10b: on NVIDIA H200, the FP8 TP2 text throughput profile is replaced by an FP8 TP2 profile tuned for image-text requests and a new NVFP4 (W4A16) TP1 throughput profile. The FP8 TP2 latency profile remains.

    • starcoder2-7b: the only optimized profile is BF16 TP1 throughput on NVIDIA H200. Release 2.1.4 also listed BF16 TP1 and TP2 throughput profiles on other GPUs, including NVIDIA A100, NVIDIA B200, and NVIDIA H100.

  • For nemotron-3.5-lightning, the Support Matrix for NIMs now lists the GPUs that are validated for each profile in this release:

    • NVIDIA H100-NVL is listed for the BF16 TP1, TP2, and TP4 profiles and the W4A16 profiles.

    • The BF16 TP1 profiles are also listed on NVIDIA A100-SXM4-80GB, NVIDIA GB10, and NVIDIA H100-80GB-HBM3, and the W4A16 profiles on NVIDIA A100-SXM4-80GB. For NVIDIA GB10, refer to the known issue.

    • The BF16 TP8 profiles are listed only on NVIDIA A100-SXM4-80GB, NVIDIA H100-80GB-HBM3, NVIDIA L40S, and NVIDIA RTX PRO 6000 Blackwell Server Edition. Release 2.1.4 also listed them on NVIDIA B200, NVIDIA B300-SXM6-AC, NVIDIA GB200, NVIDIA GB300, and NVIDIA H200.

    • NVIDIA GB10 is listed only for TP1 profiles. Release 2.1.4 also listed TP2 and larger profiles on NVIDIA GB10.

  • Profile IDs changed for several NIMs in this release, so a profile ID from release 2.1.4 might not exist in the 2.1.5 container. For example, the nemotron-parse-v2.0 BF16 TP1 throughput profile with FP8 KV cache on NVIDIA B200 has the ID df9fbe0520a0f6b97dd45964cdbeae90679bfab547052a494247b9c95de38f98, which replaces the 2.1.4 ID 98443c5c72f2afd75a056660e8e71799db695e32cdd1ca09fa0e4d5050f29472. If you set NIM_MODEL_PROFILE to a 2.1.4 profile ID, run list-model-profiles in the 2.1.5 container and set NIM_MODEL_PROFILE to the profile ID or name that it prints. For profile selection, refer to Model Profiles and Selection.

  • The following qwen3.5-122b-a10b LoRA profile and GPU combinations are excluded from the Support Matrix for NIMs in this release. With default settings, they run out of GPU memory (CUDA out-of-memory error) during engine initialization, preventing server startup. The affected configurations are:

    Qwen3.5-122B-A10B LoRA Configurations Excluded From Support Due to Startup Out-of-Memory Errors#

    Profile

    GPUs

    vllm-fp8-tp1-pp1-lora

    NVIDIA GH200-144G-HBM3e, NVIDIA H200, NVIDIA H200-NVL

    vllm-fp8-tp2-pp1-lora

    NVIDIA H100-80GB-HBM3

    vllm-fp8-tp4-pp1-lora

    NVIDIA L40S

    vllm-nvfp4-tp1-pp1-lora

    NVIDIA RTX PRO 6000 Blackwell Server Edition

    vllm-nvfp4-tp4-pp1-lora

    NVIDIA RTX PRO 4500 Blackwell Server Edition

  • The following qwen3.5-122b-a10b LoRA profile and GPU combinations are also excluded from the Support Matrix for NIMs in this release. With default settings, the frontend multimodal decoder reserves 2.05 GiB of GPU memory, exceeding the available KV-cache budget and preventing server startup. The affected configurations are:

    Qwen3.5-122B-A10B LoRA Configurations Excluded From Support Due to the Multimodal Memory Reservation#

    Profile

    GPUs

    vllm-bf16-tp2-pp1-lora

    NVIDIA GH200-144G-HBM3e, NVIDIA H200, NVIDIA H200-NVL

    vllm-bf16-tp4-pp1-lora

    NVIDIA A100-SXM4-80GB, NVIDIA H100-80GB-HBM3

    vllm-bf16-tp1-pp1-lora

    NVIDIA B300-SXM6-AC

    vllm-bf16-tp8-pp1-lora

    NVIDIA L40S

    vllm-fp8-tp8-pp1-lora

    NVIDIA RTX PRO 4500 Blackwell Server Edition

    Of the combinations in these two tables, release 2.1.4 listed vllm-fp8-tp1-pp1-lora and vllm-bf16-tp2-pp1-lora on NVIDIA GH200-144G-HBM3e, NVIDIA H200, and NVIDIA H200-NVL with a known issue. This release does not support these combinations.

  • qwen3.5-397b-a17b does not support LoRA in this release. Its LoRA profiles (vllm-bf16-tp4-pp1-lora, vllm-bf16-tp8-pp1-lora, vllm-fp8-tp2-pp1-lora, vllm-fp8-tp4-pp1-lora, vllm-fp8-tp8-pp1-lora, vllm-nvfp4-tp1-pp1-lora, vllm-nvfp4-tp2-pp1-lora, vllm-nvfp4-tp4-pp1-lora, and vllm-nvfp4-tp8-pp1-lora) are not listed in the Support Matrix for NIMs for 2.1.5. The container still includes these profiles, but they are not supported in this release. Use the non-LoRA profiles. This changes the support coverage from release 2.1.4, which listed LoRA profiles for this model on a subset of GPUs. The non-LoRA profiles are unaffected.

  • qwen3.5-122b-a10b supports text and image input only in this release. Video input is not supported on any GPU or profile. Release 2.1.4 did not restrict video input for this model. As in release 2.1.4, qwen3.5-397b-a17b also supports text and image input only. Refer to the known issues for qwen3.5-122b-a10b and qwen3.5-397b-a17b.

Known Issues#

This release includes the following known issues and limitations. For general diagnostic steps, rather than model-specific workarounds, refer to Troubleshooting GPU Memory Out-of-Memory Errors and Troubleshooting Request Timeouts and Responses.

Show Known Issues and Limitations

Jump to: deepseek-v4-flash-0731, deepseek-v4-pro-0813, gemma-4-31b-it, glm-5.2, gpt-oss-120b, gpt-oss-20b, inkling, kimi-k2.6, llama-3.1-8b-instruct, llama-3.3-nemotron-super-49b-v1.5, mistral-small-4-119b-2603, nemotron-3-nano, nemotron-3-nano-omni-30b-a3b-reasoning, nemotron-3-super-120b-a12b, nemotron-3-ultra-550b-a55b, nemotron-3.5-content-safety, nemotron-3.5-lightning, nemotron-parse-v2.0, qwen3.5-122b-a10b, qwen3.5-397b-a17b, SGLang Model-Free NIM, vLLM Model-Free NIM, variant containers.

deepseek-v4-flash-0731

  • For deepseek-v4-flash-0731, the NVFP4 TP4 profile (d2371848) cannot be deployed on NVIDIA B200 from a model store. When NIM_MODEL_PATH points to a model store created for this profile with create-model-store, the NIM reads the weight precision of this mixed-precision NVFP4 checkpoint as FP8. list-model-profiles then reports no compatible and runnable profile and lists only the NVIDIA H200 FP8 TP4 profile (e5199f4a) as incompatible with the system. Deployments that do not set NIM_MODEL_PATH are not affected, and the FP8 TP4 profile on NVIDIA H200 starts from a model store. No workaround has been validated for NIM_MODEL_PATH deployments of the NVFP4 profile on NVIDIA B200. Refer to Air-Gap Deployment.

deepseek-v4-pro-0813

  • For deepseek-v4-pro-0813, the FP8 TP8 throughput profile on NVIDIA H200 (2cf4354f) can fail to start when the container runs with a memory limit, such as a Kubernetes resources.limits.memory setting. This profile enables the SGLang hierarchical cache. The SGLang version in this release counts the container’s file page cache as used memory when it sizes the host cache pools, so startup can stop with ValueError: Not enough host memory for V4 paged pool swa. The server then exits and the container restarts repeatedly. The failure was observed in a KServe deployment with a 256 GiB memory limit. In validation, Helm deployments of this profile on NVIDIA H200 without a container memory limit started and served normally. The NVFP4 TP8 profile on NVIDIA B200 (f6e709b4) does not use the hierarchical cache. No workaround has been validated for memory-limited deployments of this profile.

  • For deepseek-v4-pro-0813, the NVFP4 TP8 throughput profile on NVIDIA B200 (f6e709b4) cannot be deployed from a model store. When NIM_MODEL_PATH points to a model store created for this profile with create-model-store, the NIM reads the weight precision of this mixed-precision NVFP4 checkpoint as FP8. list-model-profiles then does not list f6e709b4 and lists only the FP8 TP8 throughput profile for NVIDIA H200 (2cf4354f) as compatible and runnable. Deployments that do not set NIM_MODEL_PATH are not affected. In validation, Helm deployments of f6e709b4 on NVIDIA B200 with a read-write PVC, with a pre-downloaded model, and in an air-gapped installation started and served normally. No workaround has been validated for NIM_MODEL_PATH deployments of the NVFP4 profile on NVIDIA B200. Refer to Air-Gap Deployment.

gemma-4-31b-it

  • gemma-4-31b-it can fail to start on NVIDIA RTX PRO 6000 Blackwell Server Edition with NIM_MODEL_PROFILE=default. Default selection picks the BF16 TP1 profile (vllm-bf16-tp1-pp1), which requires more GPU memory than this GPU provides, instead of an NVFP4 TP1 profile. Workaround: Set NIM_MODEL_PROFILE=5f2613453901ce34041f9390d1eb035bceabba0af65af73be562fa5cda704a48, the NVFP4 TP1 (vllm-nvfp4-tp1-pp1) optimized profile listed for this GPU in the Support Matrix for NIMs.

glm-5.2

  • For glm-5.2, the throughput profiles that enable the SGLang hierarchical cache can fail to start when the container runs with a memory limit, such as a Kubernetes resources.limits.memory setting. The affected profiles are FP8 TP8 on NVIDIA H200 (28a9b5fd), which requests an 80 GB host cache per GPU (640 GB in total), and NVFP4 TP4 on NVIDIA B200 (53546dab), which requests a 128 GB host cache per GPU (512 GB in total). When the memory limit leaves less free host memory than the cache requests, startup stops with ValueError: Not enough host memory available, the server exits, and the container restarts repeatedly. The failure was observed in KServe deployments with a 384 GiB memory limit. In validation, the NVIDIA H200 profile started and served normally in deployments without a container memory limit, and the NVFP4 TP4 profile on NVIDIA B200 (a6f62247), which does not use the hierarchical cache, started under the same 384 GiB limit. No memory-limit value has been validated for the affected profiles, and no workaround has been validated for memory-limited deployments of them.

  • On glm-5.2, chat requests that set response_format.type to json_schema can intermittently return incomplete JSON on the sglang-nvfp4-tp4-pp1 profile on NVIDIA B300-SXM6-AC. This was observed during release 2.1.4 validation. After generating part of a JSON object, the model can emit only whitespace until it reaches max_tokens, returning finish_reason of length. There is no validated server-side workaround. Validate the response content in your application and retry requests that end with finish_reason of length without a complete JSON object.

gpt-oss-120b

  • On gpt-oss-120b, setting --max-lora-rank above 128 can fail engine startup with a fused_moe_lora_one_shot supports max_lora_rank<=128 assertion. This limitation also affects prior releases. Workaround: Set --max-lora-rank to 128 or lower and use adapters whose rank does not exceed the configured limit. Refer to Out of Memory During LoRA Adapter Allocation for the distinction between rank limits and memory failures.

  • gpt-oss-120b MXFP4 LoRA TP4 (vllm-mxfp4-tp4-pp1-lora) exhausts GPU memory at startup on NVIDIA L40S with --max-lora-rank 64 --max-loras 8. This limitation also affects prior releases. Workaround: Leave --max-loras at its default. This does not establish limits for other ranks or TP sizes; refer to Out of Memory During LoRA Adapter Allocation.

  • On gpt-oss-120b, streaming and non-streaming chat requests stop at a requested stop string and return finish_reason=stop, but the matched stop string, or the whole token that contains it, can remain at the end of the returned content or reasoning field. This happens with the default include_stop_str_in_output value and when it is set to false, and also affects prior releases. On the TP1 throughput profiles that use Eagle3 speculative decoding (H200 and B200), the returned text can also continue for a few tokens past the matched stop string, and the stop string can be absent even when include_stop_str_in_output is true. Do not assume that a requested stop string is excluded from, or included in, either field.

  • On gpt-oss-120b, when NIM_REPOSITORY_OVERRIDE points to an S3 bucket, a single connection that drops while NIM downloads the model files at startup stops the download without a retry. The log shows Download failed after 1 attempts with an AWS DispatchFailure that contains IncompleteMessage, followed by Error downloading manifest, and the container exits before the server starts. The failure is intermittent: the same configuration downloaded and served successfully on other attempts in this release. If startup fails with this error, start the container again. Refer to Model Download.

  • The gpt-oss-120b TP1 throughput profile on NVIDIA A100-SXM4-80GB has a maximum sequence length of 32,768 tokens (input plus output). For longer contexts, select a supported TP2, TP4, or TP8 profile.

gpt-oss-20b

  • gpt-oss-20b text completions (/v1/completions) can continue until max_tokens (finish_reason=length) without generating the requested stop string, in streaming and non-streaming modes, for example by producing reasoning-style text or a long counting sequence such as 2,3,4,... instead of the expected short continuation of the prompt. A stop string that the model generates is honored. Chat requests with stop strings are affected differently; refer to gpt-oss-20b chat requests with stop strings. Refer to Completion Stop Strings Are Not Honored for diagnosis.

  • The gpt-oss-20b LoRA (feat_lora) MXFP4 TP1 profile cannot start on NVIDIA A10G (24 GB) at the full context length of 131,072 tokens because the available memory cannot hold the KV cache for even one request at that length. Other GPUs and other gpt-oss-20b profiles on A10G are unaffected. This also affects prior releases. Workaround: Use a GPU with more memory, or lower NIM_MAX_MODEL_LEN to a value that fits the available memory. Refer to Troubleshooting GPU Memory Out-of-Memory Errors.

  • The fused mixture-of-experts kernels used by gpt-oss-20b support a maximum LoRA rank of 128. Setting --max-lora-rank above 128 causes startup to fail with a fused_moe_lora_one_shot supports max_lora_rank<=128 assertion error. This limit applies even if all deployed adapters have rank 128 or lower. The configured value determines the buffer allocation before any adapter loads. This limitation also affects prior releases. Workaround: Set --max-lora-rank to 128 or lower and use adapters with rank 128 or lower.

  • On gpt-oss-20b, LoRA (feat_lora) MXFP4 profiles can intermittently fail requests instead of returning an answer. /v1/responses requests can return HTTP 500 (unexpected tokens remaining in message header) or HTTP 400 (Unknown channel: None), non-streaming /v1/chat/completions requests can return HTTP 500, and streaming /v1/chat/completions requests can finish with empty content. The behavior is confined to the LoRA profiles and is observed across their supported GPUs; non-LoRA profiles are unaffected. Workaround: Use a non-LoRA profile where LoRA adapters are not required. If LoRA is required, handle HTTP 500 and HTTP 400 responses and empty content in your application. Handling these responses detects the problem but does not restore the missing answer.

  • On gpt-oss-20b, LoRA (feat_lora) MXFP4 profiles can return reasoning-formatted output in which the harmony reasoning channel leaks control tokens (for example <|constrain|>) into the response content, and generation can reach max_tokens without emitting final answer content. The behavior is confined to the LoRA profiles and is observed across their supported GPUs; non-LoRA profiles are unaffected. Workaround: Use a non-LoRA profile where LoRA adapters are not required. If LoRA is required, validate the response content and handle invalid or missing answers in your application. Validation detects the problem but does not restore the missing answer.

  • gpt-oss-20b /v1/chat/completions requests that set stop strings can stop while the model is still reasoning, before it writes an answer. The response ends with finish_reason=stop and no answer: non-streaming responses have content set to null, and streaming responses send no content deltas. The matched stop string can remain in the reasoning field even when include_stop_str_in_output is false or omitted. A short stop string that matches part of the model’s internal channel header (for example, a matching analysis) can instead end the response immediately with raw control text such as <|channel|>analysis in content. This occurs on all supported GPUs, for both LoRA and non-LoRA profiles. No workaround is validated. If your application sets stop strings, treat a response with finish_reason=stop and a null, empty, or control-text content as a missing answer; this detects the problem but does not restore the answer.

inkling

  • For inkling, fetching slow or unreachable image or audio URLs can delay inference requests and /v1/health/ready responses for about 30 seconds per URL. Decoding very large images, including inline images, can also cause delays. Sustained high-concurrency image requests, such as 256 concurrent requests, can delay /v1/health/ready responses by more than 5 seconds; this was observed on NVIDIA B300-SXM6-AC and NVIDIA H200 GPUs. A liveness probe that targets /v1/health/ready with a short timeout, such as 5 seconds, can restart the pod during these delays. This also affects prior releases. Workaround: Point the liveness probe at /v1/health/live, which is the Helm chart default and which the NIM proxy answers independently of the model server. Use inline media or fast, reachable hosts to reduce download delays, and downscale very large images. These measures do not eliminate all preprocessing delays. This workaround has not been validated with the high-concurrency image benchmark in this release.

  • For inkling, requests that set two or more stop strings can continue past the earliest stop string in the output on the throughput profiles that use EAGLE speculative decoding: 8502d8a44bc24573a17d5e1b20381cc2c74669280ddb9cc3fde765cf15bbb27a on NVIDIA B200 and 4a30cb634a0f720cc28572ec4592876d255762f2bfe7bf0d2dceff6890f1f5f4 on NVIDIA H200. The response then includes that stop string and the text after it, in content or reasoning_content, and still ends with finish_reason set to stop. This was observed with streaming chat requests. Non-streaming requests and text completions use the same stop handling and can also be affected. The issue was not observed on profiles that do not use speculative decoding, or on requests that set a single stop string. Workaround: Set one stop string per request, or truncate the returned text at the first occurrence of any requested stop string in your application. Truncating in the application has not been validated in this release. Refer to Completion Stop Strings Are Not Honored.

  • For inkling on NVIDIA H200 GPUs, the NVFP4 TP8 profiles b71876219f49544d9e42967621ef19a1858458b6bcd333f6c5d55b92f0f27fba and abe667223aac8d65d80dbb2ae304516d41a3d2d3bc3d89f463fa254e100876a9 can intermittently return repetitive, empty, or incoherent output with HTTP 200. These profiles do not use speculative decoding. The issue was not observed on the H200 throughput profile, which uses EAGLE speculative decoding. This also affects prior releases. Workaround: When LoRA adapters are not required, use the H200 throughput profile, which has its own multiple stop strings limitation. No unaffected H200 profile with LoRA support has been identified. Validate response content in your application.

  • For inkling, chat requests return HTTP 500 instead of HTTP 400 when an image URL is empty, is not an HTTP or HTTPS URL or supported inline data, cannot be downloaded, or points to content that is not an image, such as an HTML page. Empty audio URLs also return HTTP 500. This also affects prior releases. Workaround: Validate image URLs in your client before sending requests. Reject empty values, send only reachable HTTP or HTTPS URLs or supported inline data, and confirm that each URL returns image content.

  • For inkling, chat requests with an image in a format that the model server cannot decode, such as a camera raw file, or with audio that cannot be decoded return HTTP 500 instead of HTTP 400. This also affects prior releases. Workaround: Validate image and audio formats in your client before sending requests. For example, confirm that each image decodes locally, or convert it to PNG or JPEG.

  • For inkling, the structured_outputs request field is ignored. Its JSON, grammar, regular expression, and whitespace constraints are not applied, and requests can return unconstrained prose with HTTP 200. This also affects prior releases. To request JSON output, use response_format with type set to json_schema. Refer to JSON schema output limitation.

  • For inkling, chat requests that use response_format with type set to json_schema can return incomplete JSON with HTTP 200. Output can contain a long sequence of whitespace and reach max_tokens before closing the JSON object, with finish_reason set to length. This was observed with text-only and image-and-text requests on NVIDIA B200, B300-SXM6-AC, and H200 GPUs. This also affects prior releases, where it was also observed on NVIDIA GB300 GPUs. The issue was not observed on NVIDIA GB300 GPUs in testing for this release, but GB300 GPUs can still be affected. Workaround: Check finish_reason and validate the returned JSON. Retrying can succeed, but does not guarantee valid output.

  • For inkling, requests with some out-of-range values for n, temperature, or top_logprobs can return HTTP 200 instead of HTTP 400. Chat requests can also accept a tools[].type string other than function. This also affects prior releases. Workaround: Validate sampling parameters in your application and set tools[].type to function.

  • For inkling, a chat request that sets min_tokens above the remaining context budget without setting max_tokens can generate until it exhausts the context budget. The server does not reject this request with HTTP 400, and the response can take a long time. This also affects prior releases. Workaround: Set max_tokens or max_completion_tokens whenever you set min_tokens, and keep min_tokens at or below that limit.

  • For inkling, retrieving or canceling a nonexistent response ID through the Responses API returns HTTP 400 instead of HTTP 404 when the response store is disabled. Requests are rejected before the response ID lookup. This also affects prior releases. Workaround: Check the error response body to identify the cause. An HTTP 400 response alone does not establish that a response ID does not exist.

  • For inkling, the response store is disabled in the default serving configuration. Multi-turn requests that use previous_response_id therefore return HTTP 400. This also affects prior releases. Workaround: Maintain conversation context in your application and include the conversation history in each request.

  • For inkling, /v1/chat/completions/batch, /inference/v1/generate, /generative_scoring, and the Llama Stack /inference/chat_completion endpoint are unavailable and return HTTP 404. This also affects prior releases. Workaround: Use the supported /v1/chat/completions and /v1/completions endpoints for generation.

  • For inkling, reasoning is enabled by default, with a default reasoning_effort of 0.9. Reasoning tokens count toward max_tokens. The chat_template_kwargs.enable_thinking and include_reasoning fields do not disable reasoning or suppress reasoning_content. Request-level chat template overrides are also ignored. This also affects prior releases. Workaround: Use reasoning_effort to control reasoning effort and omit reasoning_content from your application display when it is not needed. Hiding this field does not reduce token usage.

  • For inkling, unsupported request fields are ignored rather than rejected. These include prompt_logprobs, return_tokens_as_token_ids, include_stop_str_in_output, request_id, mm_processor_kwargs, and media_io_kwargs. Requests can return HTTP 200 even when these fields contain invalid values. This also affects prior releases. Workaround: Do not rely on these fields to control generation or validate input.

  • For inkling, chat requests that set response_format to a value other than a JSON object return HTTP 500 instead of HTTP 400. This also affects prior releases. Workaround: Send response_format as a JSON object.

  • For inkling, /detokenize requests with negative or out-of-range token IDs return HTTP 500 instead of HTTP 400. This also affects prior releases. Workaround: Send token IDs within the model vocabulary.

kimi-k2.6

  • kimi-k2.6 text completions can ignore stop strings on NVIDIA B200 and B300-SXM6-AC in streaming and non-streaming modes. Chat is unaffected on all supported GPUs. Workaround: Use NVIDIA GB200, GB300, or H200, where completions are unaffected, when completion stop strings are required. Refer to Completion Stop Strings Are Not Honored.

  • kimi-k2.6 /v1/responses requests with empty input on NVIDIA GB300 can exceed 60 seconds. Nonempty requests and NVIDIA B200, B300-SXM6-AC, GB200, and H200 are unaffected. Use nonempty input or an unaffected GPU if prompt completion is required; no fix is available for the affected case. Refer to Requests Exceed the Client Timeout.

  • kimi-k2.6 supports text and image input only. Video input to /v1/chat/completions returns HTTP 500 on all supported GPUs with both NVFP4 TP4 and INT4 TP8 profiles. Workaround: Use text or image input; video input has no workaround in this release.

  • For kimi-k2.6, the NVFP4 TP4 throughput profile for NVIDIA B200 (b197f6a800db21269b7388d2ea7d82cb0e66ed5c66d3278d7049a52c687f86fa) places a 200 GiB native CPU KV-offload pool in /dev/shm. The pool is backed by host memory, so the host needs at least 200 GiB of free memory for it. If /dev/shm is smaller than the pool, such as with Docker’s default size or --shm-size=16GB, the engine fails at startup with Insufficient space in /dev/shm for CPU KV offload shared region. The other kimi-k2.6 profiles do not use native KV offload. Workaround: For this profile, run the container with --shm-size=300g, the shared-memory size this release was validated with. The NIM Helm chart mounts /dev/shm as a memory-backed volume without a size limit, and Helm deployments of this profile started and served requests in this release’s testing. If you set a sizeLimit on the /dev/shm volume, such as the 16Gi in the KServe example, raise it to at least 300Gi, and make sure any pod memory limit also leaves room for the 200 GiB pool.

  • For kimi-k2.6, the NVFP4 TP4 throughput profile for NVIDIA GB200 (25868c0d2f0529d801a9e5aea19d16ae15cbed8e7d065dfcdbf8cdc2fd451d86) can intermittently fail at startup. During engine warmup, while FlashInfer autotunes its NVFP4 mixture-of-experts kernels, one tensor-parallel worker fails with CUDA error: unspecified launch failure. The other workers then stall, and the server neither exits nor becomes ready until it is stopped. Workaround: If the NIM does not become ready and its log shows this error, stop the container or pod and start the NIM again. In this release’s testing, a new start of this profile on a different NVIDIA GB200 system became ready and served requests. Restarting on the same system has not been validated.

llama-3.1-8b-instruct

  • llama-3.1-8b-instruct NVFP4 TP1 (vllm-nvfp4-tp1-pp1) has higher single-request decode latency on NVIDIA RTX PRO 6000 Blackwell Server Edition and NVIDIA RTX PRO 4500 Blackwell Server Edition than in the previous release. At concurrency 1, inter-token latency is about 7-11% higher on RTX PRO 6000 and about 4% higher on RTX PRO 4500. Requests complete correctly, throughput at high concurrency is unchanged, and NVIDIA B200 and B300 are unaffected.

llama-3.3-nemotron-super-49b-v1.5

  • llama-3.3-nemotron-super-49b-v1.5 NVFP4 TP1 on NVIDIA GB10 can exceed client timeouts for large generations while the server remains healthy. Chat and smaller or capped completions are unaffected. Workaround: Cap output, increase the client timeout, or use a faster GPU as described in Requests Exceed the Client Timeout.

  • For llama-3.3-nemotron-super-49b-v1.5, the NVFP4 TP8 profile (vllm-nvfp4-tp8-pp1) has API and output-quality limitations on NVIDIA B200, NVIDIA B300-SXM6-AC, NVIDIA RTX PRO 4500 Blackwell Server Edition, and NVIDIA RTX PRO 6000 Blackwell Server Edition. On these GPUs, requests that use logprobs or top_logprobs can return HTTP 400 or invalid log probabilities, and some requests can generate repetitive, unrelated text that runs to the token limit. Requests that set stop strings, including requests that set include_stop_str_in_output, can then end with a finish_reason of length without the requested stop string. On NVIDIA B200, long responses such as reasoning output are the most affected: in accuracy testing for this release, no response from this profile produced a usable answer. On NVIDIA B200 and NVIDIA B300-SXM6-AC, structured-output requests that use response_format can return HTTP 500, and Responses API requests to /v1/responses can time out. The affected profile ID is 3692d8d0a442c5c2b2e3bbad4fc55861b0f46ce5c90208444bfeff3f5750be98. This also affects prior releases. Workaround: Select another profile for your GPU from the Support Matrix for NIMs, and validate the required APIs and output quality before deployment. Setting max_tokens limits output length but does not restore output quality or stop-string termination.

  • Some llama-3.3-nemotron-super-49b-v1.5 profiles can fail to start at the default context length of 131,072 tokens on GPUs with insufficient available memory, because the KV cache cannot hold one request of that length. Affected combinations are FP8 TP1 with or without LoRA on NVIDIA H100 80GB HBM3 and NVFP4 TP2 with LoRA on NVIDIA RTX PRO 4500 Blackwell Server Edition. The FP8 TP1 combinations on NVIDIA H100 80GB HBM3 are not listed as verified configurations in the Support Matrix for NIMs. This also affects prior releases. Workaround: For NVFP4 TP2 with LoRA on NVIDIA RTX PRO 4500 Blackwell Server Edition, set NIM_MAX_MODEL_LEN to a value that fits the available memory, or select another profile. For FP8 TP1 on NVIDIA H100 80GB HBM3, select a GPU and profile combination listed in the Support Matrix for NIMs. Validate the deployment after you change the context length.

  • During release 2.1.4 validation of llama-3.3-nemotron-super-49b-v1.5 with NVIDIA NIM Operator, recreating a NIMCache resource with the same name intermittently prevented the deployment from becoming ready. The operator reported a missing cache pod during reconciliation. Workaround: Use a unique name for the new NIMCache, or wait for the previous cache and its resources to be fully removed before reusing the name.

mistral-small-4-119b-2603

  • Explicit NIM_MODEL_PROFILE selection for mistral-small-4-119b-2603 bypasses GPU compatibility checks, so an unsupported or incorrect SKU profile can start and report healthy. Automatic selection still enforces the checks. Workaround: Use automatic selection or a profile that list-model-profiles reports as compatible. Refer to the Support Matrix for NIMs.

  • For mistral-small-4-119b-2603, /v1/chat/completions requests return HTTP 400 when they specify chat_template or chat_template_kwargs, including enable_thinking. The Mistral tokenizer also requires tool-call IDs in the message history to contain exactly nine alphanumeric characters (a-z, A-Z, or 0-9). Other IDs return HTTP 400. These restrictions also affect release 2.1.4. Workaround: Omit chat template overrides and use tool-call IDs that satisfy the tokenizer requirements.

  • For mistral-small-4-119b-2603, /v1/chat/completions requests that set per-request media_io_kwargs or mm_processor_kwargs return HTTP 400 (Per-request media_io_kwargs are disabled by default. Start the server with --trust-request-mm-kwargs). The 2.1.5 server (vLLM 0.31.0) disables per-request multimodal kwargs by default; this is new relative to release 2.1.4. Default multimodal chat and completion requests are unaffected. Workaround: Omit per-request media_io_kwargs and mm_processor_kwargs and use the model’s default media handling.

  • For mistral-small-4-119b-2603, POST /inference/v1/generate returns HTTP 404 on every tested profile and GPU in release 2.1.5. Release 2.1.4 has the same limitation. Workaround: Use /v1/completions or /v1/chat/completions for generation.

  • For mistral-small-4-119b-2603, /v1/chat/completions requests that force a tool call with tool_choice set to "required" or to a named function can return HTTP 200 with a text answer and no tool_calls when the prompt does not itself ask for a tool. The text answer can continue until max_tokens is reached. Testing observed this on NVFP4 and FP8 profiles on every tested GPU in release 2.1.5, and less often in release 2.1.4. Requests with tool_choice set to "auto" or "none" passed in testing. No workaround has been validated.

  • For mistral-small-4-119b-2603, the vllm-nvfp4-tp4-pp1 profile is not supported on NVIDIA B300-SXM6-AC and NVIDIA GB200 GPUs in this release, and these combinations are omitted from the Support Matrix for NIMs. On these GPUs, sustained chat traffic can delay /v1/health/ready responses beyond a five-second probe timeout while chat requests continue to succeed. Testing observed this with 100 concurrent image chat requests on NVIDIA B300-SXM6-AC GPUs, and on NVIDIA GB200 GPUs in release 2.1.4. On NVIDIA B300-SXM6-AC GPUs, testing also observed it with 32 concurrent text requests of 2,000 input and 2,000 output tokens, in one of two test runs. When the liveness probe also targets /v1/health/ready, repeated probe failures can restart the pod and cause in-flight requests to fail with HTTP 500. Release 2.1.4 documented this issue as a known issue on these GPUs. The vllm-nvfp4-tp4-pp1 profile remains supported on NVIDIA B200, NVIDIA GB300, and NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs.

  • For mistral-small-4-119b-2603, sustained text traffic can delay /v1/health/ready responses beyond a five-second probe timeout while chat requests continue to succeed. Testing observed this on the vllm-fp8-tp2-pp1 profile on NVIDIA H200 GPUs and on the NVFP4 TP2 throughput profile with FP8 KV cache on NVIDIA B200 GPUs with 128 concurrent requests and a 131,072-token generation limit. When the liveness probe also targets /v1/health/ready, repeated probe failures can restart the pod and cause in-flight requests to fail with HTTP 500. Lower-concurrency workloads on the same profiles and GPUs completed. No workaround has been validated for these workloads.

  • For mistral-small-4-119b-2603, the NIM does not apply its own NVFP4 profiles when NIM_MODEL_PATH points to the NVFP4 checkpoint or to a model store created from an NVFP4 profile with create-model-store. The NVFP4 checkpoint uses the native Mistral format (params.json with no config.json), so the NIM cannot read its weight precision and serves generic profiles instead. The NIM starts and reports ready, but the profile’s tuned settings are not applied: the context length is the checkpoint maximum of 1,048,576 tokens instead of 262,144, and the NVFP4 TP2 throughput profile on NVIDIA B200 GPUs runs without its FP8 KV cache and batching settings. list-model-profiles lists only generic profile IDs, so NIM_MODEL_PROFILE cannot select the profile that was passed to create-model-store -p. This affects the vllm-nvfp4-tp1-pp1, vllm-nvfp4-tp2-pp1, vllm-nvfp4-tp4-pp1, and vllm-nvfp4-tp8-pp1 profiles and the NVFP4 TP2 throughput profile on NVIDIA B200 GPUs. FP8 profiles and deployments that do not set NIM_MODEL_PATH are not affected. No workaround has been validated for NIM_MODEL_PATH deployments of the NVFP4 profiles. Refer to Air-Gap Deployment.

  • For mistral-small-4-119b-2603, when NIM_REPOSITORY_OVERRIDE points to an S3 bucket, a single connection that drops while NIM downloads the model files at startup stops the download without a retry. The log shows Download failed after 1 attempts with an AWS DispatchFailure that contains IncompleteMessage, followed by Error downloading manifest, and the container exits before the server starts. The failure is intermittent: the same S3 repository override downloaded and served successfully on other test runs in this release. If startup fails with this error, start the container again. Refer to Model Download.

  • For mistral-small-4-119b-2603, /v1/chat/completions requests that set response_format to {"type": "json_object"} can return a JSON array instead of a JSON object. For this response format, the model server constrains output to either a JSON object or a JSON array. Testing observed a top-level array on every tested profile and GPU in release 2.1.5, and on most tested profiles and GPUs in release 2.1.4. No workaround has been validated.

  • For mistral-small-4-119b-2603, sustained high-concurrency image chat traffic can delay /v1/health/ready responses beyond a five-second probe timeout while chat requests continue to succeed. Testing observed this with 100 concurrent image chat requests on the vllm-nvfp4-tp2-pp1 profile on NVIDIA B200 GPUs and on the vllm-nvfp4-tp4-pp1 profile on NVIDIA B300-SXM6-AC GPUs, which is not supported on these GPUs in this release. Other profiles of this model were not tested under this load and can also be affected. When the liveness probe also targets /v1/health/ready, repeated probe failures can restart the pod and cause in-flight requests to fail with HTTP 500. No workaround has been validated.

  • For mistral-small-4-119b-2603, the vllm-nvfp4-tp1-pp1 profile is not supported on NVIDIA RTX PRO 6000 Blackwell Server Edition and is omitted from the Support Matrix for NIMs, but the container still includes this profile. If you select it with NIM_MODEL_PROFILE on that GPU, startup fails with a CUDA out-of-memory error while allocating the KV cache at the default maximum model length of 262,144 tokens. Workaround: Deploy on a GPU that the Support Matrix for NIMs lists for this model.

nemotron-3-nano

  • Some nemotron-3-nano /v1/responses requests exceed 60 seconds: empty input on NVFP4 TP1/NVIDIA B200 or FP8 TP1 LoRA/NVIDIA H100-NVL, and a short nonempty prompt on NVFP4 TP1/NVIDIA GB300-WS. Completion with a longer timeout is unverified. Refer to Requests Exceed the Client Timeout for diagnosis.

  • The fused mixture-of-experts kernels used by nemotron-3-nano support a maximum LoRA rank of 128. Setting --max-lora-rank above 128 causes startup to fail with a fused_moe_lora_one_shot supports max_lora_rank<=128 assertion error, even when every deployed adapter has rank 128 or lower, because the configured value sizes the LoRA buffers before any adapter loads. This also affects prior releases. Workaround: Set --max-lora-rank to 128 or lower, and use adapters whose rank does not exceed it.

  • With dynamic LoRA enabled through NIM_PEFT_REFRESH_INTERVAL, nemotron-3-nano can fail to load an adapter whose files are still being copied into NIM_PEFT_SOURCE when the directory watcher polls. The load fails with a safetensors incomplete metadata, file not fully covered error. On profiles with tensor parallelism of 2 or more, the next load attempt can report the same error even after the copy has finished. Workaround: Add an adapter folder to NIM_PEFT_SOURCE only after all of its files are completely written, for example by copying it to a staging directory outside NIM_PEFT_SOURCE on the same file system and then moving it into NIM_PEFT_SOURCE. This workaround has not been validated in this release.

  • During release 2.1.4 validation of nemotron-3-nano with NVIDIA NIM Operator, recreating a NIMCache resource with the same name intermittently prevented the deployment from becoming ready. The operator reported a missing cache pod during reconciliation. Workaround: Use a unique name for the new NIMCache, or wait for the previous cache and its resources to be fully removed before reusing the name.

nemotron-3-nano-omni-30b-a3b-reasoning

  • nemotron-3-nano-omni-30b-a3b-reasoning accuracy on long-form reasoning workloads is not validated on NVIDIA B300-SXM6-AC for the NVFP4 TP1 profiles (vllm-nvfp4-tp1-pp1 and the B300 throughput-optimized NVFP4 TP1 profile). The GPQA Diamond accuracy benchmark did not complete on either profile in this release. Under sustained long reasoning generation on NVIDIA B300-SXM6-AC, /v1/health/ready can take longer than 5 seconds to respond while /v1/chat/completions continues to return HTTP 200. If a deployment’s liveness probe targets /v1/health/ready with a short timeout, such as 5 seconds, the orchestrator can restart a server that is still processing requests. The NIM Helm chart’s default liveness probe uses /v1/health/live instead. Both NVFP4 TP1 profiles completed the shorter OCRBench accuracy benchmark on NVIDIA B300-SXM6-AC. The BF16 and FP8 profiles completed GPQA Diamond on NVIDIA B300-SXM6-AC in this release but also showed delayed /v1/health/ready responses, so they can be affected. This also affects prior releases. Workaround: Point the liveness probe at /v1/health/live, or increase its timeout and failure threshold. This configuration has not been validated with the benchmark on NVIDIA B300-SXM6-AC. Alternatively, use NVIDIA B200, where all NVFP4 TP1 profiles completed the benchmark in this release. Refer to Requests Exceed the Client Timeout.

  • For nemotron-3-nano-omni-30b-a3b-reasoning, video input on NVIDIA RTX PRO 4500 Blackwell Server Edition can stop the NIM server. While decoding a video request, the hardware video decoder reports cuvidMapVideoFrame(...) returned error 208, and the server process aborts instead of failing only that request. Requests in flight at that time fail. In this release’s testing, this occurred with both NVFP4 TP1 profiles on this GPU during the video-input API tests, and it did not occur on NVIDIA RTX PRO 6000 Blackwell Server Edition in the same tests. The same decoder error has also stopped other NIMs on other GPUs, so other GPUs can be affected.

    Affected Configurations

    Profile ID prefix

    Precision

    Tensor parallel size

    GPU

    642f3562

    NVFP4

    1

    NVIDIA RTX PRO 4500 Blackwell Server Edition

    b0c36da4

    NVFP4

    1

    NVIDIA RTX PRO 4500 Blackwell Server Edition

    In this release’s testing, the server restarted in its pod and completed startup again within about 4 minutes. No workaround that prevents the abort has been validated.

  • For nemotron-3-nano-omni-30b-a3b-reasoning, the generic NVFP4 TP1 profile (vllm-nvfp4-tp1-pp1, profile ID prefix b0c36da4) is not listed for NVIDIA RTX PRO 4500 Blackwell Server Edition in the Support Matrix for NIMs. If you pin this profile on that GPU, the NIM can intermittently fail to start with ValueError: max_num_seqs (128) exceeds available Mamba cache blocks. Workaround: Use the default profile selection, which picks the RTX PRO 4500 throughput-optimized NVFP4 TP1 profile (642f356201892f339146d30c52e92ada94de23ac0914583dae769f3f15b84c51), or set NIM_MODEL_PROFILE to that profile ID.

nemotron-3-super-120b-a12b

  • nemotron-3-super-120b-a12b rejects requests that set a nonzero min_p or a non-empty logit_bias with HTTP 400 on NVIDIA B200 with the NVFP4 TP4 throughput-optimized profile (6bd9e1c75425eeee3f8e0ebe562409116f3bd728e668c11c9d175a176a2da95e). This profile enables speculative decoding by default, and vLLM does not yet support these parameters with speculative decoding. Requests that omit them are unaffected. Workaround: Set NIM_MODEL_PROFILE=f7c0416d2f9ed6f6ca30634833156f1c9df0171955ed4c7213505569962c8323, the NVFP4 TP4 (vllm-nvfp4-tp4-pp1) profile. It runs without speculative decoding and without the throughput tuning of the optimized profile.

  • nemotron-3-super-120b-a12b rejects requests that set a nonzero min_p or a non-empty logit_bias with HTTP 400 on NVIDIA H200 with the FP8 TP4 throughput-optimized profile (0e13d08b10121c8c2e41f03f6eb67fdf6f9261ba7e5a30e16f79709ff3622c6f). This profile enables speculative decoding by default, and vLLM does not yet support these parameters with speculative decoding. Requests that omit them are unaffected. Workaround: Set NIM_SPECDEC_ENABLE=0 to disable speculative decoding on this profile.

  • nemotron-3-super-120b-a12b LoRA profiles, including the BF16 TP4 LoRA profile (vllm-bf16-tp4-pp1-lora), do not support --max-lora-rank values above 128 in this release, because the fused mixture-of-experts LoRA kernels support a maximum LoRA rank of 128. On NVIDIA H100-NVL with the BF16 TP4 LoRA profile, --max-lora-rank 256 also exceeds GPU memory: startup fails during memory profiling with CUBLAS_STATUS_ALLOC_FAILED, even with a low-rank adapter, because the configured value sizes the LoRA buffers before any adapter loads. Workaround: Set --max-lora-rank to 128 or lower, and use adapters whose rank does not exceed it. LoRA buffer memory grows with both --max-lora-rank and --max-loras. On NVIDIA H100-NVL with the BF16 TP4 LoRA profile, --max-lora-rank 64 with the default --max-loras is validated; --max-lora-rank 128 is not validated. Refer to Out of Memory During LoRA Adapter Allocation.

nemotron-3-ultra-550b-a55b

  • nemotron-3-ultra-550b-a55b text completions at temperature=0 can repeat until max_tokens without emitting the requested stop string, in streaming and non-streaming modes. Affected GPUs are NVIDIA B200, B300-SXM6-AC, GB200, GB300, H100 80GB HBM3, H100-NVL, and H200, including optimized NVFP4 TP4 profiles. Workaround: Set max_tokens to bound generated output; this does not correct the repetitive response.

  • For nemotron-3-ultra-550b-a55b, the NIM does not apply its own NVFP4 profiles when NIM_MODEL_PATH points to the NVFP4 checkpoint or to a model store created from an NVFP4 profile with create-model-store. The NIM cannot read the weight precision of this mixed-precision NVFP4 checkpoint, so it serves generic profiles instead. The NIM starts and reports ready, but the profile’s tuned settings are not applied: MTP speculative decoding is disabled, and the profile’s GPU memory utilization and maximum number of sequences fall back to generic defaults. This was observed with the NVFP4 TP4 throughput profiles on NVIDIA B200 and H200 GPUs and applies to every NVFP4 profile of this model. Deployments that do not set NIM_MODEL_PATH are not affected. No workaround has been validated for NIM_MODEL_PATH deployments of the NVFP4 profiles. Refer to Air-Gap Deployment.

  • For nemotron-3-ultra-550b-a55b, the NVFP4 profiles on NVIDIA B200, B300-SXM6-AC, GB200, and GB300 GPUs can intermittently fail at startup. During engine warmup, while FlashInfer autotunes its NVFP4 mixture-of-experts kernels, one tensor-parallel worker fails with CUDA error: unspecified launch failure. The server then either exits without becoming ready or stalls until it is stopped. In this release’s testing of this model, this occurred once, on NVIDIA B200, with the NVFP4 TP4 throughput profile (9d06a0e06444481c655310f7521afda72b34916d99427c3264c99c081aa2be5c), and was not observed on the other GPUs. Workaround: If the NIM does not become ready and its log shows this error, stop the container or pod and start the NIM again. In this release’s testing, a new start of the same build on a different NVIDIA B200 system became ready and served requests. Restarting on the same system has not been validated.

nemotron-3.5-content-safety

  • For nemotron-3.5-content-safety, some /v1/chat/completions requests with image input return HTTP 500 instead of a response or an HTTP 400 error. This was observed with streaming and non-streaming requests that contain two or more images (requests with 2 and with 10 images were tested) and with requests whose message contains an image but no text part. A non-streaming multi-turn request whose first user turn contained one image and a text prompt also returned HTTP 500; the same request with streaming returned HTTP 200. Single-turn requests with one image and a text prompt returned HTTP 200 in both modes. In this release, image input was tested only with the BF16 TP1 profile for NVIDIA A100-SXM4-40GB (ecc0c366), so profiles for other GPUs can also be affected. No workaround has been validated.

nemotron-3.5-lightning

  • nemotron-3.5-lightning BF16 TP1 (vllm-bf16-tp1-pp1) on NVIDIA GB10 can take much longer to start than on NVIDIA H100-80GB-HBM3, and the startup time varies from run to run with the same configuration. In this release’s testing on NVIDIA GB10 with default settings, the server took between about 15 and 35 minutes to become ready, compared with under 3 minutes on NVIDIA H100-80GB-HBM3. A large and variable part of this time is spent while the inference engine allocates memory for the KV cache. In this release’s testing, API requests succeeded once the server was ready, but accuracy on NVIDIA GB10 was not validated for this profile. Workaround: If your deployment sets its own startup probe or readiness timeout, allow at least as long as the NIM Helm chart’s default startup probe, about 60 minutes, which is longer than any startup observed in this release’s testing.

nemotron-parse-v2.0

  • On nemotron-parse-v2.0, an optical character recognition (OCR) request with a blank or empty input image can return a successful (200) response that contains hallucinated document content, such as a fabricated title and garbled body text, instead of an empty result. This was observed on NVIDIA A100 80GB, B200, GB200, H100 80GB HBM3, H200, and L40S GPUs, and it also affects prior releases. No workaround has been validated.

qwen3.5-122b-a10b

  • qwen3.5-122b-a10b forced-tool requests can return HTTP 200 with empty tool_calls. Affected GPUs include NVIDIA B200, GB200, H200, and H100 80GB HBM3. Most forced requests succeed, and unforced selection is unaffected. Workaround: Use tool_choice=auto if optional tool selection is acceptable. Refer to A Forced Tool Call Is Missing.

  • qwen3.5-122b-a10b supports text and image input only. Video input is unsupported on all GPUs and profiles in this release. During validation, video requests to /v1/chat/completions that included a 4K video could trigger an unrecoverable NVDEC error (cuvidMapVideoFrame returned error 208) that terminated the server on NVIDIA H100-80GB-HBM3, H200, RTX PRO 4500 Blackwell Server Edition, and RTX PRO 6000 Blackwell Server Edition with BF16, FP8, and NVFP4 profiles. Workaround: Use text or image input.

  • qwen3.5-122b-a10b NVFP4 TP8 LoRA (vllm-nvfp4-tp8-pp1-lora) exhausts GPU memory at startup on NVIDIA RTX PRO 4500 Blackwell Server Edition when started with --max-loras 8 (rank 16) or with --max-lora-rank 256. Workaround: Use the default LoRA settings, which work on this GPU, as do --max-loras 2 at rank 16 and --max-lora-rank 64 with one adapter slot (--max-loras 1). Refer to Out of Memory During LoRA Adapter Allocation.

  • qwen3.5-122b-a10b returns HTTP 404 for /inference/v1/generate on all profiles and GPUs. This limitation also affects release 2.1.4. Workaround: For text generation, use /v1/chat/completions or /v1/completions instead.

qwen3.5-397b-a17b

  • qwen3.5-397b-a17b chat requests with tool_choice=required or a named function can return HTTP 200 with empty tool_calls across multiple profile and GPU combinations. Workaround: Use tool_choice=auto if optional tool selection is acceptable. Refer to A Forced Tool Call Is Missing.

  • qwen3.5-397b-a17b NVFP4 TP4 on NVIDIA B200 returned reasoning without a final answer for an image chat request with temperature=0.7, frequency_penalty=0.5, presence_penalty=0.5, and max_tokens=2000. The HTTP 200 response had content=null and finish_reason=stop.

  • For qwen3.5-397b-a17b, the NVFP4 TP2 throughput profiles vllm-nvfp4-b200-tp2-throughput (NVIDIA B200, f2475206) and vllm-nvfp4-h200-tp2-throughput (NVIDIA H200, 9fff2d89) place a native CPU KV-offload pool in /dev/shm: 160 GiB on NVIDIA B200 and 100 GiB on NVIDIA H200. The pool is backed by host memory. With a smaller shared-memory size, such as --shm-size=16GB, the engine fails at startup with Insufficient space in /dev/shm for CPU KV offload shared region. Workaround: For these two profiles, run the container with --shm-size=200g on NVIDIA B200 or --shm-size=128g on NVIDIA H200, or with --ipc=host. On Kubernetes, size the shared-memory volume mounted at /dev/shm the same way. These are the shared-memory sizes this release was validated with.

  • qwen3.5-397b-a17b supports text and image input only. Video input is unsupported on all GPUs and profiles. In this release’s testing on NVIDIA H200, video input triggered an unrecoverable NVDEC error that terminated the server.

  • qwen3.5-397b-a17b can fail to become ready on its first Kubernetes deployment with the NIM Helm chart’s default startup probe when the model cache is empty. The NIM downloads the selected profile from NGC inside the main container, and the chart’s default startup probe (initialDelaySeconds: 60, periodSeconds: 30, failureThreshold: 120, about 61 minutes) restarts the container if the download and model load have not finished by then. The model artifacts are about 251 GB for NVFP4 profiles, 406 GB for FP8 profiles, and 807 GB for BF16 profiles. In this release’s testing, model loading took about 11 minutes on NVIDIA GB300 once the cache was populated, so a cold deployment of an NVFP4 profile needs roughly 670 Mbit/s of sustained download throughput (FP8 about 1.1 Gbit/s, BF16 about 2.2 Gbit/s) to finish within the default window. In the affected test runs, files downloaded before a restart were downloaded again after it, and the deployment did not become ready. Workaround: Pre-populate the model cache volume for the selected profile before deploying, and size persistence.size for the profile; the chart default of 50Gi is too small for this model. Raising startupProbe.failureThreshold so that initialDelaySeconds + failureThreshold × periodSeconds exceeds the expected download time has not been validated in this release.

  • qwen3.5-397b-a17b returns HTTP 404 for /inference/v1/generate on all profiles and GPUs. This limitation carries forward from release 2.1.4. Workaround: For text generation, use /v1/chat/completions or /v1/completions instead.

  • For qwen3.5-397b-a17b, the profiles that enable MTP speculative decoding reject requests that set min_p or logit_bias with HTTP 400 (The min_p and logit_bias sampling parameters are not yet supported with speculative decoding). These profiles are vllm-nvfp4-b200-tp2-throughput (NVIDIA B200), vllm-nvfp4-h200-tp2-throughput (NVIDIA H200), vllm-fp8-h200-tp4-pp1-image-text-throughput (NVIDIA H200), and vllm-nvfp4-gb300-tp4-pp1-image-text-throughput (NVIDIA GB300). Workaround: Omit min_p and logit_bias from requests to these profiles.

SGLang Model-Free NIM

  • In the SGLang Model-Free NIM, download-to-cache --all with a remote NIM_MODEL_PATH, such as an hf:// URI, downloads the same checkpoint once for each generated profile. This repeats the transfer of the model weights and can cause cache preparation jobs to exceed their time limits. This also affects Kubernetes and KServe jobs that run this command, and it also affects release 2.1.4. Workaround: Instead of --all, download only the profile that you deploy by running download-to-cache -p <PROFILE_HASH> with the same NIM_MODEL_PATH, using a profile hash from list-model-profiles. This transfers the checkpoint once. Serving from a cache prepared this way has not been validated for this NIM in this release. For air-gapped systems, you can also use a local model directory as described in Model-Free NIM with a Local Path.

  • In the SGLang Model-Free NIM, explicitly selecting a generic profile with NIM_MODEL_PROFILE can fail to start with a CUDA out-of-memory error during CUDA graph capture, even though list-model-profiles reports the profile as compatible. The generic profile starts SGLang with a static memory fraction of 0.92 and the model’s default context length, and too little GPU memory remains for graph capture. This was observed with nvidia/GLM-5.3-NVFP4 and the generic TP4 profile (sglang-tp4-pp1) on four NVIDIA B200 GPUs, and with zai-org/GLM-5.3 and the generic TP2 PP4 profile (sglang-tp2-pp4) on eight NVIDIA H200 GPUs. For these models on these GPUs, automatic profile selection is not affected: it selects the tuned NVIDIA B200 NVFP4 TP4 throughput profile and the tuned NVIDIA H200 FP8 TP8 throughput profile, respectively, and each starts and serves requests. Workaround: For nvidia/GLM-5.3-NVFP4 on four NVIDIA B200 GPUs and zai-org/GLM-5.3 on eight NVIDIA H200 GPUs, leave NIM_MODEL_PROFILE unset so that the NIM selects the tuned profile automatically.

vLLM Model-Free NIM

  • When the vLLM Model-Free NIM serves gpt-oss-20b, streaming and non-streaming /v1/chat/completions requests that set stop strings can keep the matched stop string in the returned reasoning or content field, even when include_stop_str_in_output is false or omitted. If the stop string matches while the model is still reasoning, the response can return no answer (content is null). This was observed on every GPU tested and also affects prior releases. No workaround is validated. Do not assume that a requested stop string is excluded from either field.

  • In the vLLM Model-Free NIM, download-to-cache --all with a remote NIM_MODEL_PATH, such as an ngc:// URI, downloads the same checkpoint once for each generated profile. This repeats the transfer of the model weights and can cause cache preparation jobs to exceed their time limits. This also affects Kubernetes and KServe jobs that run this command. Workaround: Instead of running download-to-cache --all, start NIM on a system with network access with NIM_MODEL_PATH set and a persistent cache volume. At startup, NIM fetches the model files only for the selected profile and reuses them from the cache on later restarts. For air-gapped systems, use a local model directory as described in Model-Free NIM with a Local Path. Startup from an empty cache has not been validated as a replacement for cache preparation jobs in this release.

The following known issues apply to variant containers that NVIDIA released separately from release 2.1.5: the qwen3.8-27b, glm-5.3-flash, and Qwen3.8-Flash-Next containers built on release 2.1.4, the experimental SGLang Model-Free NIM container for MiniMax M3.1 Flash, and the experimental vLLM Model-Free NIM container for Kimi K3.1 Pro. They do not describe the release 2.1.5 containers.

  • For MiniMax M3.1 Flash served by the experimental SGLang Model-Free NIM container 2.1.7-variant, json_schema output can contain incorrect values or incomplete JSON when a field type differs from the model’s natural answer. Workaround: Match schema field types to the expected answer, such as using integer for an integer result.

  • For Kimi K3.1 Pro, /openapi.json advertises POST /generative_scoring. Earlier development builds returned HTTP 404 for this endpoint. NIM 2.1.6-variant-kimi-k3.1-pro includes a fix.

  • For Kimi K3.1 Pro, a readiness GET /v1/health/ready request sent with a JSON body can cause the next GET /v1/metrics request to return HTTP 400. NIM 2.1.6-variant-kimi-k3.1-pro includes a fix. Workaround: Send readiness probes without a request body. Refer to Kimi K3.1 Pro.

  • For Kimi K3.1 Pro on NIM 2.1.6-variant-kimi-k3.1-pro, audio and video input and NIM_DISABLE_MODEL_DOWNLOAD are not supported. The environment variable is deprecated for this model and release; it must not be used to prevent checkpoint-file writes. Mount the downloaded checkpoint read-only. Refer to Kimi K3.1 Pro.

  • On qwen3.8-27b, the image pins --max-num-seqs 256 in its default NIM_PASSTHROUGH_ARGS. The cap is required: without it, vLLM selects 1,024 on GPUs with 70 GB or more of memory, the Mamba cache of this hybrid model runs out of blocks, and the server fails at startup with max_num_seqs exceeds available Mamba cache blocks. Passthrough arguments take precedence over the values a profile declares, so the optimized profiles run at 256 instead of their own: 24 for B200 NVFP4 TP1 and B200 FP8 TP2, 48 for B200 BF16 TP2, 22 for H200 NVFP4 TP2, and 32 for H200 FP8 TP2 and H200 BF16 TP2. Workaround: To use the value a profile declares, append --max-num-seqs <value> to the container command after the image name (for example docker run ... nvcr.io/nim/qwen/qwen3.8-27b:2.1.4-variant --max-num-seqs 24), which overrides the cap and keeps every other flag the image sets. Do not set NIM_PASSTHROUGH_ARGS instead. A user value replaces the whole default string, and without --exclude-tools-when-tool-choice-none, tool_choice: "none" requests return empty content again. Run the container with --dry-run to confirm the effective value, which must not exceed the available Mamba cache blocks.

  • On qwen3.8-27b, a fatal startup error, such as an invalid NIM_MODEL_PROFILE value or a profile that is not compatible with the GPU, stops the container with exit status 0 instead of a non-zero status. A script or orchestrator that checks only the exit status does not detect the failure. Workaround: Check the container log for NIM-LLM process died and the error message that precedes it. A fix is planned for a later release.

  • On qwen3.8-27b, time to first token rises sharply at concurrency 64 or higher on long-prefill workloads (8K and 64K input) on NVIDIA L40S, NVIDIA A100 40GB, and NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. Throughput at concurrency 32 or lower is at parity with a standalone vLLM deployment. Workaround: Keep concurrency at 32 or lower per GPU set on these GPUs or add replicas.

  • glm-5.3-flash does not support NVIDIA RTX PRO 6000 Blackwell, RTX PRO 4500 Blackwell, A100, L40S, GH200, A10G, or GB10. SGLang 0.5.20 has no sparse-attention (DSA) backend for the RTX PRO Blackwell architecture, the sparse-attention kernels need Hopper (SM90) or newer, the Arm64 SGLang package omits the FlashAttention 3 kernels that GH200 decoding uses, and the model does not fit in GPU memory on one A10G or GB10 node. Workaround: Use a GPU listed in GLM-5.3-Flash.

  • glm-5.3-flash does not support multi-node deployment. Workaround: Use a single-node GPU configuration from GLM-5.3-Flash.

  • For glm-5.3-flash, video input is not validated. Workaround: Use text or image input.

  • For glm-5.3-flash, the throughput-optimized NVFP4 profile on four NVIDIA GB200 GPUs keeps 100 GB of KV cache per GPU in host memory. Workaround: Provide about 400 GB of free host RAM and run the container with --shm-size=64g.

  • For glm-5.3-flash, reasoning cannot be turned off. Setting chat_template_kwargs.enable_thinking to false has no effect, and reasoning tokens count toward max_tokens. Workaround: To shorten reasoning, set reasoning_effort to low. At the default effort, use a max_tokens of at least 8192. Refer to Control Reasoning.

  • For glm-5.3-flash, stop sequences also match text in the reasoning. If a stop sequence appears while the model is reasoning, the response ends before the answer and content is empty. Workaround: Choose stop sequences that are unlikely to appear in the reasoning, or check content before using the response.

  • For glm-5.3-flash, when generation stops at a string from stop_sequences, the Anthropic Messages API (/v1/messages) reports stop_reason as end_turn instead of stop_sequence and omits the stop_sequence field. This affects streaming and non-streaming requests. Generation still stops before the matched string. Workaround: To detect which stop sequence matched, use /v1/chat/completions with stop, which reports the matched string in matched_stop.

  • For glm-5.3-flash, a Responses API (/v1/responses) request that includes an image and omits max_output_tokens returns HTTP 400 with a context-length error. Workaround: Set max_output_tokens on Responses API requests that include images.

  • For Qwen3.8-Flash-Next, the tuned FP8 recipes for NVIDIA B200 and NVIDIA H200 select tensor parallel 2 and expert parallel 2 with MTP speculative decoding and use two GPUs. Do not set NIM_TENSOR_PARALLEL_SIZE and do not pass the --tensor-parallel-size flag for these recipes; the image sets neither. A value of NIM_TENSOR_PARALLEL_SIZE=4, or the flag --tensor-parallel-size 4 appended to the container command, overrides the recipe’s tensor-parallel size but keeps its expert-parallel size, and the NIM exits at startup with ValueError: expert_parallel_size (2) must be divisible by tensor_parallel_size (4) to derive data_parallel_size. Workaround: Remove (unset) NIM_TENSOR_PARALLEL_SIZE from the deployment environment and remove --tensor-parallel-size from the container command. Reducing the number of visible GPUs does not clear the override. With the variable unset and the flag removed, the NIM starts with 2 or 4 visible GPUs and applies the recipe parallelism. Other GPUs this model supports are unaffected.

  • For Qwen3.8-Flash-Next, the NVIDIA H200 tuned FP8 recipe (TP2, EP2, MTP, two GPUs) trades time to first token (TTFT) for throughput above a concurrency of 64. Measured with a chat workload (2,000 input / 2,000 output tokens, greedy decoding), aggregate output throughput is 3,143 tokens/s at concurrency 32 (TTFT p50 0.19 s) and 4,072 tokens/s at 64 (TTFT p50 1.8 s). At concurrency 128 and 256, throughput rises only to 4,309 and 4,344 tokens/s while TTFT p50 rises to 30 s and 88 s. Image workloads behave the same way.

  • For Qwen3.8-Flash-Next, the vLLM Model-Free NIM image sets NIM_PASSTHROUGH_ARGS to --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-num-seqs 256 --no-enable-log-requests --disable-uvicorn-access-log --exclude-tools-when-tool-choice-none. A NIM_PASSTHROUGH_ARGS value set in the container environment replaces this whole string instead of extending it. A replacement that omits the default flags drops the reasoning parser, the tool-call parser, --exclude-tools-when-tool-choice-none, and the two logging flags. Without the parsers, the NIM returns reasoning text in content instead of message.reasoning, and a request with tools returns HTTP 400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set. Without the two logging flags, image-request throughput is lower. Workaround: If you set NIM_PASSTHROUGH_ARGS, start from the full default string and append your flags to it. To add one flag and keep the rest, append the flag to the container command after the image reference instead (for example docker run ... ${NIM_LLM_IMAGE} --max-num-seqs 128). Run the container with --dry-run to print the effective vllm serve command and confirm the flags. To restore per-request log lines, add --enable-log-requests. Request-level visibility also remains available through the nginx access log and the X-Request-Id header. Refer to Logging and Observability.

  • For Qwen3.8-Flash-Next, the Docker default shared memory of 64 MB is too small for the tuned FP8 recipes for NVIDIA B200 and NVIDIA H200 (tensor parallel 2). The container logs Insufficient space in /dev/shm and stops. Workaround: Add --shm-size=16GB to docker run.

  • For Qwen3.8-Flash-Next, a fatal startup error, such as too little shared memory, a missing NIM_MODEL_PATH, an unsupported tensor-parallel override, or an unknown NIM_MODEL_PROFILE value, logs an error but stops the container with exit status 0 instead of a non-zero status. A --dry-run prints NIM-LLM process died (exit code: 0) after a successful dry run. Restart policies, Kubernetes Jobs, and CI pipelines that check only the exit status treat the failed start as a success. Workaround: Check the container log for ERROR lines and poll /v1/health/ready instead of relying on the exit status.

  • For Qwen3.8-Flash-Next, when NIM_MODEL_PATH points to a Hugging Face repository that cannot be resolved (for example, a typo in the name, or no access with HTTP 401 or 404) and the cache directory already holds a model, the NIM logs a WARNING that it falls back to the cached manifest, reaches ready, and serves the cached model under the configured NIM_SERVED_MODEL_NAME. /v1/metadata reports the URI of the cached model, not the requested one. Workaround: Check the startup log for falling back to the cached manifest and compare the model URI in /v1/metadata with the intended model. Use a separate cache directory for each model.

  • For Qwen3.8-Flash-Next, NIM_MODEL_PROFILE rejects generic profile names, such as vllm-fp8-tp2-pp1, and the value default with no matching profile_id or profile description is found in manifest. Only a profile ID from the Support Matrix for NIMs or the full profile description (for example vllm-nvidia-b200-fp8-tp2-pp1-ep2-throughput-2901:10de-2-1) is accepted. The list-model-profiles and download-to-cache commands fail with Could not determine path of NIM manifest when NIM_MODEL_PATH is not set. Workaround: Set NIM_MODEL_PROFILE to the profile ID for your GPU from the Support Matrix for NIMs, or read the profile description from --dry-run or /v1/metadata and pass it in full. Set NIM_MODEL_PATH before you run the helper commands.

  • For Qwen3.8-Flash-Next running on an H200 NVL GPU with the FP8 profile (TP4), long-context throughput (64k input / 400 output) is 17% to 26% below plain vLLM at concurrency 4 to 32. Short-context (8k/1k), chat and image workloads on this SKU are at parity or faster.

For information about past updates and older versions, refer to the previous release notes.