Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for this NIM LLM and VLM release.

Release 2.1.4#

Highlights#

This release delivers NIM performance improvements, new models, new features, and updates to vLLM and SGLang.

Performance Improvements#

This release delivers throughput optimizations for 28 NIMs, up to 2.1x relative to the upstream open-source engine. Actual gains depend on the NIM, the workload, and the GPU. The following table lists these NIMs by NIM type and category:

NIMs With Throughput Optimizations#

NIM Type

Category

NIMs

Model-specific NIM

LLM

deepseek-v4-flash-0731, deepseek-v4-pro-0813, glm-5.2, gpt-oss-120b, gpt-oss-20b, llama-3.1-70b-instruct, llama-3.1-8b-instruct, llama-3.3-70b-instruct, llama-3.3-nemotron-super-49b-v1.5, nemotron-3-nano, nemotron-3-super-120b-a12b, nemotron-3-ultra-550b-a55b, nemotron-3.5-lightning, starcoder2-7b

VLM

gemma-4-26b-a4b-it, gemma-4-31b-it, inkling, kimi-k2.6, mistral-small-4-119b-2603, nemotron-3-nano-omni-30b-a3b-reasoning, nemotron-3.5-content-safety, nemotron-parse-v2.0, qwen3.5-122b-a10b, qwen3.5-397b-a17b

Model-free NIM

vLLM backend

apriel-nemotron-15b-thinker, codestral-2501, mistral-medium-3.5-128b

SGLang backend

glm-5.3

Refer to the Support Matrix for NIMs for each NIM’s optimized profiles, and use the filter at the top of that page to find the profiles for your GPU.

New Models#

This release adds the following models:

  • glm-5.3 is served by the SGLang model-free NIM container, nvidia/sglang-model-free-nim. Refer to GLM-5.3 for supported configurations.

  • mistral-medium-3.5-128b is served by the vLLM model-free NIM container, nvidia/vllm-model-free-nim. Refer to Mistral-Medium-3.5-128B for supported configurations.

  • MiniMax-M3.1-Flash is served by the experimental SGLang model-free NIM container nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.7-variant with a matching 2.1.7-variant serving recipe. Refer to MiniMax M3.1 Flash for supported configurations, and to Get Started with MiniMax M3.1 Flash for deployment steps.

  • nemotron-3.5-bolt-e4b is published as the nvidia/nemotron-3.5-bolt-e4b-4gb container. It accepts text, image, and audio input; video input is not supported. Refer to nemotron-3.5-bolt-e4b for supported configurations.

Model Updates#

  • glm-5.3-flash is served by the SGLang model-free NIM container nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.4-variant-glm-5.3-flash. Refer to GLM-5.3-Flash for supported configurations, and to Get Started with GLM-5.3-Flash for deployment steps.

  • The meta/muse-glimmer NIM is updated in this release. Refer to muse-glimmer for supported configurations.

  • The nvidia/nemotron-3.5-content-safety NIM is updated in this release. Refer to nemotron-3.5-content-safety for supported configurations.

  • The nvidia/nemotron-parse-v2.0 NIM is updated in this release. Refer to nemotron-parse-v2.0 for supported configurations.

  • The qwen/qwen3.8-27b NIM is updated in this release. Refer to qwen3.8-27b for supported configurations.

  • qwen3.8-flash-next is served by the vLLM model-free NIM container nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-qwen3.8-flash-next. Refer to Qwen3.8-Flash-Next for supported configurations, and to Get Started with Qwen3.8-Flash-Next for deployment steps.

New Features#

This release includes the following new features:

  • The SGLang model-free NIM is published as the nvidia/sglang-model-free-nim container. Refer to About Model-Free NIM.

  • The Helm chart includes an SGLang values overlay, values-sglang.yaml, for deployments that use the shared chart with an SGLang image. NIM_KVCACHE_PERCENT maps to SGLang --mem-fraction-static. Setting NIM_ENABLE_KV_CACHE_REUSE to false maps to --disable-radix-cache.

  • Container images now include the NVIDIA license, and a notice and disclaimer about the external materials that NIM retrieves or accesses, at /opt/nim/NIM_LLM_LICENSE.

vLLM 0.30.0#

The vLLM inference backend is updated from 0.28.0 to vLLM 0.30.0. This version includes the following changes:

  • Model Runner V2 is the default for all models.

  • FlashInfer all-reduce is enabled by default for tensor-parallel deployments.

  • Prefix caching for hybrid Mamba models checkpoints the prefill state internally, which shortens time to first token.

The container carries a backport of vLLM pull request 53175, which enables an FP8 key-value (KV) cache for Gemma 4 on Hopper GPUs. The container also no longer replaces vLLM’s video decoder. Video requests use the upstream loader, including its decoder-pool limit and decoder memory reservation.

SGLang 0.5.20#

The SGLang inference backend is updated from 0.5.18 to SGLang 0.5.20. Mooncake, the KV cache transfer engine, is updated from v0.3.12.post1 to v0.3.13.post1.

Security Fixes#

This release fixes 28 unique security issues across the vLLM and SGLang images (4 critical, 10 high, 13 medium, and 1 low).

vLLM Images#

This release fixes 12 security issues in the vLLM images (1 high and 11 medium).

  • python3.12 and related packages are updated from 3.12.3-1ubuntu0.16 to 3.12.3-1ubuntu0.17, addressing CVE-2026-15308 (NVD: High; Ubuntu: Medium) and CVE-2026-4360 (Medium). CVE-2026-15308 is a CPU denial-of-service vulnerability in the Python HTML parser, html.parser.HTMLParser, that affects code that parses untrusted HTML.

  • Other packages are updated to address medium-severity issues:

SGLang Images#

This release fixes 28 security issues in the SGLang images (4 critical, 10 high, 13 medium, and 1 low).

  • nltk is updated from 3.9.1 to 3.10.3, addressing CVE-2026-79657, CVE-2026-79675, CVE-2026-78683, and CVE-2025-14009 (all Critical).

  • python3.12 is updated to 3.12.3-1ubuntu0.17, addressing CVE-2026-15308 (NVD: High; Ubuntu: Medium) and CVE-2026-4360 (Medium). CVE-2026-15308 is a CPU denial-of-service vulnerability in the Python HTML parser, html.parser.HTMLParser, that affects code that parses untrusted HTML.

  • openssl and related packages are updated to 3.0.13-0ubuntu3.16, and the cryptography Python package is updated to 50.0.2 to replace its bundled OpenSSL, addressing CVE-2026-84782 (High). The flaw is an out-of-bounds read in DTLS handshake retransmission, so it is reachable only over a DTLS connection, where a peer can read heap memory or crash the process.

  • virtualenv is removed, addressing CVE-2024-53899 (High) and CVE-2026-22702 (Medium). It was installed only as a dependency of the pre-commit developer tool, which is also removed. NIM does not use either at runtime.

  • The FFmpeg libraries that PyAV uses to decode audio input are rebuilt from FFmpeg 9.0.2, addressing CVE-2026-64830, CVE-2026-64835, and CVE-2026-70628 (all High).

  • The rebuilt FFmpeg leaves out Vulkan and NVDEC hardware decoding, muxers, and network protocols, addressing CVE-2026-64831, CVE-2026-64832, CVE-2026-64833, and CVE-2026-64834 (all High). opencv-python-headless and decord2, which carried additional copies of FFmpeg, are removed.

  • Other packages are updated to address medium- and low-severity issues:

The SGLang images also remove the vulnerable nsight-compute-2025.3.1 package, its CUDA metapackage, and leftover files. This removal is not counted in the 28 CVE fixes listed in this section.

Bug Fixes#

This release includes the following fixes:

  • Tensor-parallel workers no longer crash at startup on single-node NVIDIA B200 systems with NVSwitch when the multi-node NVLink cluster ID is empty. NIM now treats an empty ID as a single-node system.

  • Inference requests can now carry a body of up to 2 GB by default. Previously, the proxy rejected bodies larger than 10 MB with an HTTP 413 error that did not explain how to raise the limit. This limit affected multimodal chat requests with inline image or video data. Management endpoints keep the 10 MB limit. To set a lower inference limit, set NIM_PROXY_CLIENT_MAX_BODY_SIZE.

  • On the SGLang backend, profile selection and the memory estimate now use the same test for whether a deployment will serve LoRA adapters. Previously, those paths could disagree, so a deployment that set NIM_PEFT_SOURCE, DYN_LORA_ENABLED, or --enable-lora could be assigned a profile that did not reserve memory for the adapters.

  • The vLLM images now keep the base image’s NCCL library inside the NIM virtual environment. Previously, the audio dependencies installed an older NCCL that the process loaded instead, which could affect multi-GPU communication.

  • Flags in NIM_PASSTHROUGH_ARGS are now recognized on hosts without a GPU. Previously, vLLM argument discovery returned no known arguments when a GPU build of vLLM ran on a host without a GPU, so passthrough flags were not recognized.

  • On qwen3.8-27b, a chat completion request that set tool_choice to "none" while tools were present returned empty content. The image now sets --exclude-tools-when-tool-choice-none in its default NIM_PASSTHROUGH_ARGS, and the request returns a normal text answer. If you override NIM_PASSTHROUGH_ARGS, keep this flag in your value.

Support Changes#

This release changes support for the following configurations:

  • The following qwen3.5-122b-a10b LoRA profile and GPU combinations are excluded from the 2.1.4 Support Matrix for NIMs. With default settings, the frontend multimodal decoder reserves 2.05 GiB of GPU memory, exceeding the available KV-cache budget and preventing server startup. The affected configurations are:

    Qwen3.5-122B-A10B LoRA Configurations Excluded From 2.1.4 Support#

    Profile

    GPUs

    vllm-bf16-tp4-pp1-lora

    NVIDIA A100-SXM4-80GB, NVIDIA H100-80GB-HBM3

    vllm-bf16-tp8-pp1-lora

    NVIDIA L40S

    vllm-bf16-tp1-pp1-lora

    NVIDIA B300-SXM6-AC

    vllm-fp8-tp8-pp1-lora

    NVIDIA RTX PRO 4500 Blackwell Server Edition

  • The following qwen3.5-122b-a10b LoRA profile and GPU combinations are also excluded from the 2.1.4 Support Matrix for NIMs. With default settings, they run out of GPU memory (CUDA out-of-memory error) during engine initialization, preventing server startup. The affected configurations are:

    Qwen3.5-122B-A10B LoRA Configurations Excluded From 2.1.4 Support Due to Startup Out-of-Memory Errors#

    Profile

    GPUs

    vllm-fp8-tp2-pp1-lora

    NVIDIA H100-80GB-HBM3

    vllm-fp8-tp4-pp1-lora

    NVIDIA L40S

    vllm-nvfp4-tp1-pp1-lora

    NVIDIA RTX PRO 6000 Blackwell Server Edition

    vllm-nvfp4-tp4-pp1-lora

    NVIDIA RTX PRO 4500 Blackwell Server Edition

Known Issues#

This release includes the following known issues and limitations. For general diagnostic steps, rather than model-specific workarounds, refer to Troubleshooting GPU Memory Out-of-Memory Errors and Troubleshooting Request Timeouts and Responses.

Show Known Issues and Limitations

  • For MiniMax M3.1 Flash served by the experimental SGLang Model-Free NIM container 2.1.7-variant, json_schema output can contain incorrect values or incomplete JSON when a field type differs from the model’s natural answer. Workaround: Match schema field types to the expected answer, such as using integer for an integer result.

  • muse-glimmer supports text and image input only. A request containing video_url can return HTTP 200 while the video frames are not passed to the model. Audio input, sent as audio_url or input_audio, is handled the same way: the request returns HTTP 200 and the model does not receive the audio. Workaround: Use text or image input.

  • Constrained output is unreliable for muse-glimmer, which reasons before every answer. A request that uses response_format with a json_schema or json_object type, or the regex parameter, can return any of the following:

    • A value that is well-formed but wrong.

    • A schema-valid object filled with placeholder values, such as {"city": "...", "country": "..."}.

    • With json_object, an object that is valid JSON but unusable, such as {"": ""}, or keys that contain reasoning text or the <|eom|> token.

    • An empty content, when the output-token budget is too small for the reasoning section. Refer to Muse Glimmer for budget guidance.

    The empty-content case returns finish_reason: "length". The other cases return HTTP 200 with finish_reason: "stop", so a client cannot detect the failure from the response envelope. For json_schema and regex, upstream SGLang 0.5.20 reproduces the same behavior on the same hardware, model, and requests. Workaround: Validate constrained responses against the request before you use them, or send the request without a constraint and parse the answer from content.

  • muse-glimmer does not enforce a forced tool choice. A request that sets tool_choice to "required" or names a function can return HTTP 200 with finish_reason: "stop", a text answer in content, and no tool_calls when the model decides that no tool is needed. The model also returns at most one tool call per turn, so a task that needs several calls completes over several turns. tool_choice: "auto" and "none", multi-turn tool use, and streaming tool calls work. Workaround: Check that finish_reason is "tool_calls" before you rely on a forced call. If it is not, retry with an instruction in the prompt that names the tool to call, or handle the text answer in your client.

  • The muse-glimmer B200 NVFP4 Turbo configuration disables the vision tower for its tuned agentic-coding recipe. Image input is unsupported on that configuration. Workaround: Select a B200 BF16 profile when image input is needed.

  • The context for muse-glimmer is 131,072 tokens except on H100 80GB HBM3, H100 NVL, H200, and H200 NVL when using NVFP4, and on A100-SXM4-40GB and RTX PRO 4500, which are limited to 65,536 tokens.

  • muse-glimmer on GB10 supports NVFP4 only. The 128 GB of unified memory is shared with the CPU, and the candidate profile limits the usable memory fraction to 0.35.

  • A fatal startup error can end the muse-glimmer container with exit code 0. This has been observed after a profile-selection failure, where NIM_MODEL_PROFILE matches no profile in the manifest, and after a fatal out-of-memory error during engine initialization. In both cases the server logs the error and shuts down with exit code 0. Failure-based restart policies do not restart the container for this exit code. Kubernetes Deployments and StatefulSets use restartPolicy: Always, which restarts containers regardless of their exit code. Workaround: Treat /v1/health/ready rather than the container exit status as the signal that the server started, and check the startup log for ERROR when a container exits early.

  • muse-glimmer logs import warnings and exception tracebacks for optional multimodal processors during startup. These are expected: the container has no video decoder, so the processors that depend on one cannot load. The server starts and serves normally.

  • On nemotron-parse-v2, a blank or empty input image can cause an optical character recognition (OCR) request to return a successful (200) response containing hallucinated document content (for example, a fabricated section heading and body text) instead of an empty or no-content result. There is no validated workaround at this time.

  • nemotron-3-nano-omni-30b-a3b-reasoning accuracy on long-generation reasoning workloads has not been validated on NVIDIA B300-SXM6-AC for the vllm-nvfp4-tp1-pp1, vllm-fp8-tp1-pp1, and vllm-bf16-tp2-pp1 profiles. Under sustained long-form generation on these B300-SXM6-AC profiles, the server’s /v1/health/ready readiness endpoint can stop responding within the deployment readiness-probe timeout while /v1/chat/completions and the liveness endpoint continue to return HTTP 200, which can cause an orchestrator to recycle an otherwise-healthy pod. Standard chat and completion requests and shorter generations are unaffected, and these three profiles are unaffected on the other GPUs they support; all three are supported on NVIDIA B200. Workaround: For long-form reasoning workloads on NVIDIA B300-SXM6-AC, deploy an unaffected GPU that supports the profile — all three profiles (vllm-nvfp4-tp1-pp1, vllm-fp8-tp1-pp1, vllm-bf16-tp2-pp1) are supported on NVIDIA B200, and vllm-fp8-tp1-pp1 and vllm-bf16-tp2-pp1 are additionally supported on NVIDIA H200 — or raise the deployment’s readiness-probe timeout and failure threshold so a healthy server is not recycled during long generations. Refer to Requests Exceed the Client Timeout.

  • On gpt-oss-20b, LoRA (feat_lora) MXFP4 profiles can return reasoning-formatted output in which the harmony reasoning channel leaks control tokens (for example <|constrain|>) into the response content, and generation can reach max_tokens without emitting final answer content. The behavior is confined to the LoRA profiles and is observed across their supported GPUs; non-LoRA profiles are unaffected. Workaround: Use a non-LoRA profile where LoRA adapters are not required. If LoRA is required, validate the response content and handle invalid or missing answers in your application. Validation detects the problem but does not restore the missing answer.

  • The fused mixture-of-experts kernels used by gpt-oss-20b support a maximum LoRA rank of 128. Setting --max-lora-rank above 128 causes startup to fail with a fused_moe_lora_one_shot supports max_lora_rank<=128 assertion error. This limit applies even if all deployed adapters have rank 128 or lower. The configured value determines the buffer allocation before any adapter loads. This limitation also affects prior releases. Workaround: Set --max-lora-rank to 128 or lower and use adapters with rank 128 or lower.

  • The gpt-oss-20b LoRA (feat_lora) MXFP4 profile with tensor parallelism of one (TP1) cannot start on NVIDIA A10G (24 GB) at the full context length of 131,072 tokens. The available memory cannot hold the key-value (KV) cache for even one request at that length. This startup limitation does not affect the profile on other supported GPUs or other gpt-oss-20b profiles on A10G. It also affects prior releases. Workaround: Use a GPU with more memory, or lower --max-model-len (or NIM_MAX_MODEL_LEN) to a value that fits the available memory. Lowering this setting limits the combined input and output tokens per request. Refer to Troubleshooting GPU Memory Out-of-Memory Errors for guidance on choosing a value.

  • gpt-oss-20b requests to /v1/completions can produce prolonged reasoning-style output and reach max_tokens without generating the expected completion or requested stop string. This affects streaming and non-streaming requests across supported GPUs and carries forward from prior releases. The stop condition cannot trigger if the string is not generated. Use /v1/chat/completions for chat-formatted prompts. Set max_tokens to bound output and validate completion output in your application; these steps do not correct the generation behavior. Refer to Completion Stop Strings Are Not Honored for diagnosis.

  • On gpt-oss-120b, a requested stop string can appear in reasoning_content returned by streaming and non-streaming chat requests. Check the final response content separately from reasoning_content; do not assume that requested stop strings are excluded from reasoning output.

  • On gpt-oss-120b, setting --max-lora-rank above 128 can fail engine startup with a fused_moe_lora_one_shot supports max_lora_rank<=128 assertion. Workaround: Set --max-lora-rank to 128 or lower and use adapters whose rank does not exceed the configured limit.

  • The gpt-oss-120b TP1 throughput profile on NVIDIA A100-SXM4-80GB has a maximum sequence length of 32,768 tokens (input plus output). For longer contexts, select a supported TP2, TP4, or TP8 profile.

  • qwen3.5-397b-a17b has the following API limitations on all profiles and GPUs:

    • /inference/v1/generate returns HTTP 404. For text generation, use /v1/chat/completions or /v1/completions instead.

    • Invalid sampling or structured-output parameters, such as an out-of-range n, top_k, or repetition_penalty, or an empty guided-JSON or regex constraint, can return HTTP 200 instead of HTTP 400. Validate parameters before sending requests.

    • Some image chat responses have empty or null content. Check for empty content and handle it in your application.

  • qwen3.5-397b-a17b supports text and image input only. Video input is unsupported on all GPUs and profiles, and on NVIDIA B200 with the vllm-nvfp4-tp8-pp1 profile, video input can trigger an unrecoverable NVDEC error that terminates the server.

  • On qwen3.8-27b, the image pins --max-num-seqs 256 in its default NIM_PASSTHROUGH_ARGS. The cap is required: without it, vLLM selects 1,024 on GPUs with 70 GB or more of memory, the Mamba cache of this hybrid model runs out of blocks, and the server fails at startup with max_num_seqs exceeds available Mamba cache blocks. Passthrough arguments take precedence over the values a profile declares, so the optimized profiles run at 256 instead of their own: 24 for B200 NVFP4 TP1 and B200 FP8 TP2, 48 for B200 BF16 TP2, 22 for H200 NVFP4 TP2, and 32 for H200 FP8 TP2 and H200 BF16 TP2. Workaround: To use the value a profile declares, append --max-num-seqs <value> to the container command after the image name (for example docker run ... nvcr.io/nim/qwen/qwen3.8-27b:2.1.4-variant --max-num-seqs 24), which overrides the cap and keeps every other flag the image sets. Do not set NIM_PASSTHROUGH_ARGS instead. A user value replaces the whole default string, and without --exclude-tools-when-tool-choice-none, tool_choice: "none" requests return empty content again. Run the container with --dry-run to confirm the effective value, which must not exceed the available Mamba cache blocks.

  • On qwen3.8-27b, a fatal startup error, such as an invalid NIM_MODEL_PROFILE value or a profile that is not compatible with the GPU, stops the container with exit status 0 instead of a non-zero status. A script or orchestrator that checks only the exit status does not detect the failure. Workaround: Check the container log for NIM-LLM process died and the error message that precedes it. A fix is planned for a later release.

  • On qwen3.8-27b, time to first token rises sharply at concurrency 64 or higher on long-prefill workloads (8K and 64K input) on NVIDIA L40S, NVIDIA A100 40GB, and NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. Throughput at concurrency 32 or lower is at parity with a standalone vLLM deployment. Workaround: Keep concurrency at 32 or lower per GPU set on these GPUs or add replicas.

  • On qwen3.5-122b-a10b, the following LoRA profiles can fail to start with default settings on NVIDIA GH200-144G-HBM3e, NVIDIA H200, and NVIDIA H200-NVL. These profile and GPU combinations remain in the Support Matrix for NIMs.

    • vllm-bf16-tp2-pp1-lora: The frontend multimodal decoder reserves 2.05 GiB of GPU memory, exceeding the available KV-cache budget and preventing server startup.

    • vllm-fp8-tp1-pp1-lora: A CUDA out-of-memory error during engine initialization prevents server startup.

  • On qwen3.5-122b-a10b, chat requests with high-resolution or large video input can return HTTP 500 with a CUDA out-of-memory error during video decoding. The following table lists the affected configurations and inputs observed during validation. There is no validated workaround.

    Affected Configurations

    Profile ID Prefix

    Precision

    Tensor Parallel Size

    LoRA

    GPU

    Failing Video Input

    8c83a5a2

    BF16

    4

    Disabled

    NVIDIA B200, NVIDIA GB200

    8K

    989aa6cc

    BF16

    2

    Disabled

    NVIDIA GB200

    8K

    c06f2ba3

    BF16

    4

    Enabled

    NVIDIA GB200

    8K

    7897bbe8

    FP8

    4

    Enabled

    NVIDIA B200

    8K

    d1cef367

    FP8

    4

    Disabled

    NVIDIA GB200

    8K

    d1cef367

    FP8

    4

    Disabled

    NVIDIA H100-80GB-HBM3

    4K; 200 MB 1080p file; multiple-video request

    c475df81

    FP8

    2

    Disabled

    NVIDIA H200

    4K

    0e618ee0

    FP8

    8

    Disabled

    NVIDIA L40S

    4K

    f10440ae

    NVFP4

    4

    Disabled

    NVIDIA B200, NVIDIA GB200

    8K

    d69a1f92

    NVFP4

    4

    Enabled

    NVIDIA B200

    8K

    5c40112d

    NVFP4

    2

    Enabled

    NVIDIA GB200

    8K; 4K under memory pressure

    68c0d967

    NVFP4

    1

    Enabled

    NVIDIA GB200

    8K

    27f35f44

    NVFP4

    8

    Disabled

    NVIDIA RTX PRO 4500 Blackwell Server Edition

    4K

  • For llama-3.3-nemotron-super-49b-v1.5, some chat and completion requests that use include_stop_str_in_output generate repetitive text without producing the requested stop string. Generation reaches the token limit and returns a finish_reason of length. This behavior occurs with BF16, FP8, and NVFP4 profiles. Set max_tokens to limit the generated output, and validate the output for your application. Limiting output does not restore stop-string termination.

  • For llama-3.3-nemotron-super-49b-v1.5, the NVFP4 TP8 profile on NVIDIA B200 has API and output-quality limitations. Requests that use logprobs or top_logprobs can return HTTP 400, structured-output requests that use response_format can return HTTP 500, and Responses API requests to /v1/responses can time out. Some requests also produce repetitive or unrelated output. The affected profile ID is 3692d8d0a442c5c2b2e3bbad4fc55861b0f46ce5c90208444bfeff3f5750be98. Workaround: Select another GPU and profile combination from the Support Matrix for NIMs, and validate the required APIs and output quality before deployment.

  • Manually selecting some llama-3.3-nemotron-super-49b-v1.5 profiles on GPUs with insufficient available memory can cause startup to fail at the default context length of 131,072 tokens. Affected combinations include FP8 TP1 with or without LoRA on NVIDIA H100 80GB HBM3 and NVIDIA GB10, NVFP4 TP1 with LoRA on NVIDIA GB10, and NVFP4 TP2 with LoRA on NVIDIA RTX PRO 4500 Blackwell Server Edition. The FP8 TP1 combinations on NVIDIA H100 80GB HBM3 and NVIDIA GB10 are not listed as verified configurations in the Support Matrix for NIMs. Workaround: Select a validated GPU and profile combination. If you manually configure a shorter context, set NIM_MAX_MODEL_LEN to a value that fits the available memory and validate the deployment.

  • During validation of llama-3.3-nemotron-super-49b-v1.5 with NVIDIA NIM Operator, recreating a NIMCache resource with the same name intermittently prevented the deployment from becoming ready. The operator reported a missing cache pod during reconciliation. Workaround: Use a unique name for the new NIMCache, or wait for the previous cache and its resources to be fully removed before reusing the name.

  • For mistral-small-4-119b-2603 on NVIDIA B200 and B300-SXM6-AC GPUs, the vllm-fp8-tp4-pp1 profile can produce repetitive output in /v1/completions requests that use stop strings. Generation can reach the token limit and return finish_reason set to length instead of stop. This occurs with include_stop_str_in_output set to either true or false. The behavior was not observed on NVFP4 profiles or on the FP8 TP4 profile on NVIDIA GB200 and GB300 GPUs. Workaround: For workloads that rely on stop strings, use one of these unaffected profiles listed in the Support Matrix for NIMs.

  • For mistral-small-4-119b-2603, /v1/chat/completions requests return HTTP 400 when they specify chat_template or chat_template_kwargs, including enable_thinking. The Mistral tokenizer also requires tool-call IDs in the message history to contain exactly nine alphanumeric characters (a-z, A-Z, or 0-9). Other IDs return HTTP 400. These restrictions also affect release 2.0.13. Workaround: Omit chat template overrides and use tool-call IDs that satisfy the tokenizer requirements.

  • For mistral-small-4-119b-2603, POST /inference/v1/generate returns HTTP 404 on every tested profile and GPU in release 2.1.4. This endpoint worked in release 2.0.13. Workaround: Use /v1/completions or /v1/chat/completions for generation.

  • For mistral-small-4-119b-2603, sustained image chat traffic can delay /v1/health/ready responses on the vllm-nvfp4-tp4-pp1 profile on NVIDIA B300-SXM6-AC and GB200 GPUs. Testing with 100 concurrent requests exceeded a five-second probe timeout while chat requests continued to succeed. When the liveness probe also targets /v1/health/ready, repeated probe failures can restart the pod and cause in-flight requests to fail with HTTP 500. The Helm chart default probes and this workload on NVIDIA B200 and GB300 GPUs were not tested. No workaround has been validated for this workload.

  • The fused mixture-of-experts kernels used by nemotron-3-nano support a maximum LoRA adapter rank of 128. Loading a rank-256 adapter can prevent the server from starting. Workaround: Use adapters with rank 128 or lower. Reducing the configured maximum rank does not change an adapter’s rank.

  • Some nemotron-3-nano NVFP4 profiles can produce unrelated or incomplete output, omit a requested stop string, or reach the token limit before producing a final answer. This affects the generic NVFP4 TP1 profile on NVIDIA B200, NVIDIA B300-SXM6-AC, NVIDIA GB200, NVIDIA GB300, and NVIDIA GB300-WS, and the generic NVFP4 TP2 and TP4 profiles on NVIDIA GB200 and NVIDIA GB300. Workaround: Select a validated BF16 or FP8 profile from the Support Matrix for NIMs, and validate the output for your application. Setting max_tokens limits output length but does not correct output quality.

  • During validation of nemotron-3-nano with NVIDIA NIM Operator, recreating a NIMCache resource with the same name intermittently prevented the deployment from becoming ready. The operator reported a missing cache pod during reconciliation. Workaround: Use a unique name for the new NIMCache, or wait for the previous cache and its resources to be fully removed before reusing the name.

  • In the SGLang model-free NIM, download-to-cache --all with a remote NIM_MODEL_PATH, such as an hf:// URI, downloads the same checkpoint for each generated profile. This causes repeated transfers of the model weights and can cause cache preparation jobs to exceed their time limits. This also affects Kubernetes and KServe jobs that run this command.

    Workaround: On a system with network access, start NIM with NIM_MODEL_PATH set and a persistent cache volume. NIM downloads the checkpoint for the selected profile during startup. For air-gapped systems, use a local model directory as described in Model-Free NIM with a Local Path.

  • On glm-5.2, chat requests that set response_format.type to json_schema can intermittently return incomplete JSON on the sglang-nvfp4-tp4 profile on NVIDIA B300-SXM6-AC. After generating part of a JSON object, the model can emit only whitespace until it reaches max_tokens, returning finish_reason of length. There is no validated server-side workaround. Validate the response content in your application and retry requests that end with finish_reason of length without a complete JSON object.

  • glm-5.3-flash does not support NVIDIA RTX PRO 6000 Blackwell, RTX PRO 4500 Blackwell, A100, L40S, GH200, A10G, or GB10. SGLang 0.5.20 has no sparse-attention (DSA) backend for the RTX PRO Blackwell architecture, the sparse-attention kernels need Hopper (SM90) or newer, the Arm64 SGLang package omits the FlashAttention 3 kernels that GH200 decoding uses, and the model does not fit in GPU memory on one A10G or GB10 node. Workaround: Use a GPU listed in GLM-5.3-Flash.

  • glm-5.3-flash does not support multi-node deployment. Workaround: Use a single-node GPU configuration from GLM-5.3-Flash.

  • For glm-5.3-flash, video input is not validated. Workaround: Use text or image input.

  • For glm-5.3-flash, the throughput-optimized NVFP4 profile on four NVIDIA GB200 GPUs keeps 100 GB of KV cache per GPU in host memory. Workaround: Provide about 400 GB of free host RAM and run the container with --shm-size=64g.

  • For glm-5.3-flash, reasoning cannot be turned off. Setting chat_template_kwargs.enable_thinking to false has no effect, and reasoning tokens count toward max_tokens. Workaround: To shorten reasoning, set reasoning_effort to low. At the default effort, use a max_tokens of at least 8192. Refer to Control Reasoning.

  • For glm-5.3-flash, stop sequences also match text in the reasoning. If a stop sequence appears while the model is reasoning, the response ends before the answer and content is empty. Workaround: Choose stop sequences that are unlikely to appear in the reasoning, or check content before using the response.

  • For glm-5.3-flash, when generation stops at a string from stop_sequences, the Anthropic Messages API (/v1/messages) reports stop_reason as end_turn instead of stop_sequence and omits the stop_sequence field. This affects streaming and non-streaming requests. Generation still stops before the matched string. Workaround: To detect which stop sequence matched, use /v1/chat/completions with stop, which reports the matched string in matched_stop.

  • For glm-5.3-flash, a Responses API (/v1/responses) request that includes an image and omits max_output_tokens returns HTTP 400 with a context-length error. Workaround: Set max_output_tokens on Responses API requests that include images.

  • For inkling, chat requests with invalid or empty image URLs, unsupported URL schemes, or empty audio URLs return HTTP 500 instead of HTTP 400. Workaround: Validate media inputs before sending requests. Use valid, reachable media URLs or supported inline data.

  • For inkling, chat requests with unsupported image formats or audio that cannot be decoded return HTTP 500 instead of HTTP 400. Workaround: Validate image and audio formats before sending requests.

  • For inkling, the structured_outputs request field is ignored. Its JSON, grammar, regular expression, and whitespace constraints are not applied, and requests can return unconstrained prose with HTTP 200. To request JSON output, use response_format with type set to json_schema. Refer to JSON schema output limitation.

  • For inkling, image-and-text chat requests that use response_format with type set to json_schema can return incomplete JSON with HTTP 200. Output can contain a long sequence of whitespace and reach max_tokens before closing the JSON object, with finish_reason set to length. This was observed with strict schemas on NVIDIA B200, B300-SXM6-AC, GB300, and H200 GPUs. Workaround: Check finish_reason and validate the returned JSON. Retrying can succeed, but does not guarantee valid output.

  • For inkling, requests with some out-of-range values for n, temperature, or top_logprobs can return HTTP 200 instead of HTTP 400. Chat requests can also accept a tools[].type string other than function. Workaround: Validate sampling parameters in your application and set tools[].type to function.

  • For inkling, a chat request that sets min_tokens above the remaining context budget without setting max_tokens can generate until it exhausts the context budget. The server does not reject this request with HTTP 400, and the response can take a long time. Workaround: Set max_tokens or max_completion_tokens whenever you set min_tokens, and keep min_tokens at or below that limit.

  • For inkling, Responses API requests with empty input return HTTP 400 with an initialization error instead of a message identifying the empty input. Workaround: Send nonempty input to /v1/responses.

  • For inkling, retrieving or canceling a nonexistent response ID through the Responses API returns HTTP 400 instead of HTTP 404 when the response store is disabled. Requests are rejected before the response ID lookup. Workaround: Check the error response body to identify the cause. An HTTP 400 response alone does not establish that a response ID does not exist.

  • For inkling, the response store is disabled in the default serving configuration. Multi-turn requests that use previous_response_id therefore return HTTP 400. Workaround: Maintain conversation context in your application and include the conversation history in each request.

  • For inkling, /v1/chat/completions/batch, /inference/v1/generate, /generative_scoring, and the Llama Stack /inference/chat_completion endpoint are unavailable and return HTTP 404. Workaround: Use the supported /v1/chat/completions and /v1/completions endpoints for generation.

  • For inkling, reasoning is enabled by default, with a default reasoning_effort of 0.9. Reasoning tokens count toward max_tokens. The chat_template_kwargs.enable_thinking and include_reasoning fields do not disable reasoning or suppress reasoning_content. Request-level chat template overrides are also ignored. Workaround: Use reasoning_effort to control reasoning effort and omit reasoning_content from your application display when it is not needed. Hiding this field does not reduce token usage.

  • For inkling, unsupported request fields are ignored rather than rejected. These include prompt_logprobs, return_tokens_as_token_ids, include_stop_str_in_output, request_id, mm_processor_kwargs, and media_io_kwargs. Requests can return HTTP 200 even when these fields contain invalid values. Workaround: Do not rely on these fields to control generation or validate input.

  • For inkling, chat requests that set response_format to a value other than a JSON object return HTTP 500 instead of HTTP 400. Workaround: Send response_format as a JSON object.

  • For inkling, /detokenize requests with negative or out-of-range token IDs return HTTP 500 instead of HTTP 400. Workaround: Send token IDs within the model vocabulary.

  • For inkling on NVIDIA H200 GPUs, the NVFP4 TP8 profiles b71876219f49544d9e42967621ef19a1858458b6bcd333f6c5d55b92f0f27fba and abe667223aac8d65d80dbb2ae304516d41a3d2d3bc3d89f463fa254e100876a9 can intermittently return repetitive, empty, or incoherent output with HTTP 200. These profiles do not use speculative decoding. The issue was not observed on the H200 throughput profile, which uses EAGLE speculative decoding. Workaround: When LoRA adapters are not required, use the H200 throughput profile. No unaffected H200 profile with LoRA support has been identified. Validate response content in your application.

  • For inkling, fetching slow or unreachable image or audio URLs can delay inference requests and /v1/health/ready responses for about 30 seconds per URL. Decoding very large images, including inline images, can also cause delays. A liveness probe that targets /v1/health/ready can restart the pod during these delays. Workaround: Keep the Helm chart default liveness probe on /v1/health/live, which the NIM proxy answers independently of the model server. Use inline media or fast, reachable hosts to reduce download delays, and downscale very large images. These measures do not eliminate all preprocessing delays.

  • For Qwen3.8-Flash-Next, the tuned FP8 recipes for NVIDIA B200 and NVIDIA H200 select tensor parallel 2 and expert parallel 2 with MTP speculative decoding and use two GPUs. Do not set NIM_TENSOR_PARALLEL_SIZE and do not pass the --tensor-parallel-size flag for these recipes; the image sets neither. A value of NIM_TENSOR_PARALLEL_SIZE=4, or the flag --tensor-parallel-size 4 appended to the container command, overrides the recipe’s tensor-parallel size but keeps its expert-parallel size, and the NIM exits at startup with ValueError: expert_parallel_size (2) must be divisible by tensor_parallel_size (4) to derive data_parallel_size. Workaround: Remove (unset) NIM_TENSOR_PARALLEL_SIZE from the deployment environment and remove --tensor-parallel-size from the container command. Reducing the number of visible GPUs does not clear the override. With the variable unset and the flag removed, the NIM starts with 2 or 4 visible GPUs and applies the recipe parallelism. Other GPUs this model supports are unaffected.

  • For Qwen3.8-Flash-Next, the NVIDIA H200 tuned FP8 recipe (TP2, EP2, MTP, two GPUs) trades time to first token (TTFT) for throughput above a concurrency of 64. Measured with a chat workload (2,000 input / 2,000 output tokens, greedy decoding), aggregate output throughput is 3,143 tokens/s at concurrency 32 (TTFT p50 0.19 s) and 4,072 tokens/s at 64 (TTFT p50 1.8 s). At concurrency 128 and 256, throughput rises only to 4,309 and 4,344 tokens/s while TTFT p50 rises to 30 s and 88 s. Image workloads behave the same way.

  • For Qwen3.8-Flash-Next, the vLLM Model-Free NIM image sets NIM_PASSTHROUGH_ARGS to --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-num-seqs 256 --no-enable-log-requests --disable-uvicorn-access-log --exclude-tools-when-tool-choice-none. A NIM_PASSTHROUGH_ARGS value set in the container environment replaces this whole string instead of extending it. A replacement that omits the default flags drops the reasoning parser, the tool-call parser, --exclude-tools-when-tool-choice-none, and the two logging flags. Without the parsers, the NIM returns reasoning text in content instead of message.reasoning, and a request with tools returns HTTP 400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set. Without the two logging flags, image-request throughput is lower. Workaround: If you set NIM_PASSTHROUGH_ARGS, start from the full default string and append your flags to it. To add one flag and keep the rest, append the flag to the container command after the image reference instead (for example docker run ... ${NIM_LLM_IMAGE} --max-num-seqs 128). Run the container with --dry-run to print the effective vllm serve command and confirm the flags. To restore per-request log lines, add --enable-log-requests. Request-level visibility also remains available through the nginx access log and the X-Request-Id header. Refer to Logging and Observability.

  • For Qwen3.8-Flash-Next, the Docker default shared memory of 64 MB is too small for the tuned FP8 recipes for NVIDIA B200 and NVIDIA H200 (tensor parallel 2). The container logs Insufficient space in /dev/shm and stops. Workaround: Add --shm-size=16GB to docker run.

  • For Qwen3.8-Flash-Next, a fatal startup error, such as too little shared memory, a missing NIM_MODEL_PATH, an unsupported tensor-parallel override, or an unknown NIM_MODEL_PROFILE value, logs an error but stops the container with exit status 0 instead of a non-zero status. A --dry-run prints NIM-LLM process died (exit code: 0) after a successful dry run. Restart policies, Kubernetes Jobs, and CI pipelines that check only the exit status treat the failed start as a success. Workaround: Check the container log for ERROR lines and poll /v1/health/ready instead of relying on the exit status.

  • For Qwen3.8-Flash-Next, when NIM_MODEL_PATH points to a Hugging Face repository that cannot be resolved (for example, a typo in the name, or no access with HTTP 401 or 404) and the cache directory already holds a model, the NIM logs a WARNING that it falls back to the cached manifest, reaches ready, and serves the cached model under the configured NIM_SERVED_MODEL_NAME. /v1/metadata reports the URI of the cached model, not the requested one. Workaround: Check the startup log for falling back to the cached manifest and compare the model URI in /v1/metadata with the intended model. Use a separate cache directory for each model.

  • For Qwen3.8-Flash-Next, NIM_MODEL_PROFILE rejects generic profile names, such as vllm-fp8-tp2-pp1, and the value default with no matching profile_id or profile description is found in manifest. Only a profile ID from the Support Matrix for NIMs or the full profile description (for example vllm-nvidia-b200-fp8-tp2-pp1-ep2-throughput-2901:10de-2-1) is accepted. The list-model-profiles and download-to-cache commands fail with Could not determine path of NIM manifest when NIM_MODEL_PATH is not set. Workaround: Set NIM_MODEL_PROFILE to the profile ID for your GPU from the Support Matrix for NIMs, or read the profile description from --dry-run or /v1/metadata and pass it in full. Set NIM_MODEL_PATH before you run the helper commands.

  • For Qwen3.8-Flash-Next running on an H200 NVL GPU with the FP8 profile (TP4), long-context throughput (64k input / 400 output) is 17% to 26% below plain vLLM at concurrency 4 to 32. Short-context (8k/1k), chat and image workloads on this SKU are at parity or faster.

For information about past updates and older versions, refer to the previous release notes.