Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for the NIM LLM Production Branch version 6 (PB6) releases, newest first.

Important

PB releases (indicated by the -pb tag) provide stable, secure AI frameworks and SDKs for mission-critical applications. Refer to NVIDIA AI Enterprise Release Branches for more details.

Release 2.0.4-pb6.3#

NIM LLM 2.0.4-pb6.3 is a point release of the NVIDIA AI Enterprise Production Branch 6 (PB6) for NIM LLM. This release uses the standard NIM LLM 2.0.4 documentation set as its baseline and adds PB-specific release information on this page.

Unless a PB-specific update is listed here, the standard 2.0.4 release notes apply.

Highlights#

The following updates are specific to the 2.0.4-pb6.3 point release:

This release resolves 36 CVEs across the base image and PB-specific packages, including one critical and one high severity fix in the vLLM inference backend. Refer to Security Fixes for more information.

Security Fixes#

This release includes the following security fixes:

  • vLLM — The updated base image includes backported fixes for CVE-2026-48746 (Critical) and CVE-2026-41523 (High) in the vLLM inference backend.

  • System packages — Updated python3.12, curl, gzip, tar, libnghttp2, libtinfo6, and libgcrypt20 to clear known CVEs.

  • Mooncake transfer engine — Rebuilt with Go 1.25.12 to fix CVE-2026-39822, a symlink-escape flaw in the bundled Go standard library.

  • wget — Removed the unused wget package from the image to reduce the container’s vulnerability surface, clearing four Medium-severity CVEs (CVE-2026-58469, CVE-2026-58470, CVE-2026-58471, and CVE-2026-58472).

Security Alerts#

This release includes the following security alerts:

  • Supported model scope: LLM only. PB6 supports large language model (LLM) inference. Vision-language (VLM) / multimodal workloads are not supported. VLM code paths are present in the base image but are not a supported PB6 workload.

  • CVE-2026-44222 (vLLM, Medium): Do not allow untrusted client input with multimodal serving.

  • CVE-2026-27893 (vLLM, High): Do not load VLMs from untrusted sources. Some VLMs automatically enable remote code execution.

Bug Fixes#

No PB-specific bug fixes are documented for this release beyond the security fixes above. For fixes from the baseline release, refer to the standard 2.0.4 release notes.

Release 2.0.4-pb6.2#

NIM LLM 2.0.4-pb6.2 is a point release of the NVIDIA AI Enterprise Production Branch 6 (PB6) for NIM LLM. This release uses the standard NIM LLM 2.0.4 documentation set as its baseline and adds PB-specific release information on this page.

Unless a PB-specific update is listed here, the standard 2.0.4 release notes apply.

Highlights#

The following updates are specific to the 2.0.4-pb6.2 point release:

  • Updated the DLFW vLLM base image to 26.03.04-py3-stage (from 26.03.03-py3-stage). This base includes an upgrade of DALI to 2.1.1 to resolve an FFMPEG licensing issue.

  • Lowered the default NIM_KVCACHE_PERCENT (GPU memory fraction for the KV cache) from 0.92 to 0.9, matching the default of the vLLM 0.17.1 backend that the PB image runs. The previous 0.92 was inherited from the feature branch build (vLLM 0.20.0) and budgeted GPU memory too aggressively for PB, causing unexpected out-of-memory errors during CUDA graph capture on some hardware. You can still override it explicitly with NIM_KVCACHE_PERCENT.

Security Fixes#

This release includes the following security fixes:

  • nginx — Updated from 1.30.1 to 1.30.3, fixing CVE-2026-9256 (Critical) — a heap overflow in ngx_http_rewrite_module.

  • Python packages — Upgraded aiohttp, cryptography, msgpack, python-multipart, PyJWT, pydantic-settings, and uv to clear known CVEs, including High-severity issues in cryptography (GHSA-537c-gmf6-5ccf), msgpack (GHSA-6v7p-g79w-8964), python-multipart (GHSA-5rvq-cxj2-64vf), and PyJWT (GHSA-xgmm-8j9v-c9wx).

  • System packages — Updated libsqlite3-0 to clear Medium-severity CVEs.

  • Base image (DLFW 26.03.04) — Removed the unused vim package to reduce the container’s vulnerability surface.

Security Alerts#

This release includes the following security alerts:

  • Supported model scope: LLM only. PB6 supports large language model (LLM) inference. Vision-language (VLM) / multimodal workloads are not supported. VLM code paths are present in the base image but are not a supported PB6 workload.

  • CVE-2026-44222 (vLLM 0.17.1): Do not allow untrusted client input with multimodal serving.

  • CVE-2026-27893 (vLLM 0.17.1): Do not load VLMs from untrusted sources. Some VLMs (for example, Kimi-K2.5 and NemotronVL) automatically enable remote code execution.

Bug Fixes#

No PB-specific bug fixes are documented for this release beyond the security fixes above. For fixes from the baseline release, refer to the standard 2.0.4 release notes.

Known Issues#

This release includes the following known issues:

  • ModelScope downloads can fail with error decoding response body on slow or unstable networks. When pulling models from a modelscope:// source over a slow or long-haul link, the transfer can drop a chunked HTTP response mid-stream and abort startup at the download step with an error decoding response body error. This is a transient, recoverable read failure originating from the source rather than a NIM defect. NIM makes a single download attempt by default (NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=0), so the first failure ends startup. As a workaround, set a non-zero retry count — for example NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=3 — so NIM retries the download. For details, refer to Reliability on Slow or Unstable Networks in the model download guide.

  • LoRA is not supported with NVFP4-quantized Mixture-of-Experts (MoE) models. For MoE models such as Nemotron-3-Nano, profiles that combine NVFP4 quantization with LoRA (for example, nvfp4 TP1 LoRA) fail to start. vLLM injects LoRA adapters through its Triton-based fused-MoE kernel, but NVFP4 checkpoints dispatch to a specialized NVFP4 experts kernel instead, so engine startup aborts with an AssertionError in vllm/lora/layers/fused_moe.py. This is an upstream vLLM limitation (vLLM 0.17.1). The startup-failure fix is tracked in vLLM PR #34421, and broader native NVFP4 MoE LoRA support (Cutlass kernels) is tracked in vLLM issue #42629. As a workaround, serve these models with a non-NVFP4 profile (for example, BF16 or FP8) when LoRA is required, or use an NVFP4 profile without LoRA.

  • gpt-oss-120b can fail to start on two NVIDIA-A100-SXM4-40GB GPUs with No available memory for the cache blocks. On NVIDIA-A100-SXM4-40GB (40 GB, Ampere) at tensor-parallel 2, gpt-oss-120b’s MXFP4 weights are served through vLLM’s Marlin dequantization kernels (Ampere has no native FP4), whose extra buffers leave too little GPU memory for the KV cache under this release’s default NIM_KVCACHE_PERCENT=0.90 — so the vLLM engine aborts at startup with No available memory for the cache blocks. Reducing the context length (NIM_MAX_MODEL_LEN) does not help, because the shortfall is in the overall memory budget, not the cache size. As a workaround, raise the KV-cache budget with NIM_KVCACHE_PERCENT=0.92, which frees enough memory for the server to start. Larger deployments (tp=4, tp=8) and other GPUs are unaffected.

For baseline known issues and limitations, refer to the standard 2.0.4 release notes.

Release 2.0.4-pb6.1#

NIM LLM 2.0.4-pb6.1 is a point release of the NVIDIA AI Enterprise Production Branch 6 (PB6) for NIM LLM. This release uses the standard NIM LLM 2.0.4 documentation set as its baseline and adds PB-specific release information on this page.

Unless a PB-specific update is listed here, the standard 2.0.4 release notes apply.

Highlights#

The following updates are specific to the 2.0.4-pb6.1 point release:

  • Updated the DLFW vLLM base image to 26.03.03-py3-stage (from 26.03.02-py3-stage). The inference backend remains at vLLM 0.17.1.

  • Reduced the container’s vulnerability surface by removing unused multimedia components from the PB image (see Security Fixes).

The following updates apply to the PB6 documentation set:

  • Updated the Support Matrix with the PB6 model-free and model-specific NIMs.

  • Clarified that PB releases provide stable and secure NVIDIA AI Enterprise release branches for mission-critical deployments.

  • Updated OpenShift SecurityContextConstraints (SCC) guidance for restricted-v2 and restricted-v3.

  • Added guidance for PB6 air-gapped deployments, including runtime cache and telemetry control environment variables.

  • Corrected the documented default for NIM_KVCACHE_PERCENT (GPU memory fraction for the KV cache) to 0.92. Earlier documentation listed it as 0.9.

Security Fixes#

This release includes the following security fixes:

  • nginx — Updated from 1.29.7 to 1.30.1, which fixes the following CVEs:

    • CVE-2026-42945 — “NGINX Rift” RCE in ngx_http_rewrite_module (Critical)

    • CVE-2026-42926, CVE-2026-42934, CVE-2026-42946, CVE-2026-40460, and CVE-2026-40701 (Medium)

  • Mooncake transfer engine — Rebuilt with Go 1.25.11 and upgraded the vulnerable Go module dependencies in the bundled libetcd_wrapper.so:

    • google.golang.org/grpc upgraded to 1.79.3 — gRPC-Go :path authorization bypass (CVE-2026-33186 / GHSA-p77j-4mvh-x3m3)

    • golang.org/x/net upgraded to 0.55.0 — IDNA hostname-validation bypass (CVE-2026-39821 / GO-2026-5026)

    • Go 1.25.11 toolchain also clears CVE-2025-68121, CVE-2025-61732, CVE-2026-25679, and CVE-2026-27142

  • System multimedia libraries — Removed the base image’s system ffmpeg libraries (libavcodec60, libavformat60, libswscale7) and opencv-python-headless from the PB image to reduce CVE and licensing surface. Hardware video decoding and VLM video input are not supported on PB6.

Security Alerts#

This release includes the following security alerts:

  • CVE-2026-42311 (Pillow 12.1.1): For vision language model (VLM) deployments, do not allow untrusted image sources with multimodal serving.

  • CVE-2026-44222 (vLLM 0.17.1): Do not allow untrusted client input with multimodal serving.

  • CVE-2026-27893 (vLLM 0.17.1): Do not load VLMs from untrusted sources. Some VLMs (for example, Kimi-K2.5 and NemotronVL) automatically enable remote code execution.

Bug Fixes#

This release includes the following bug fix:

  • Startup banner now displays the release-appropriate license. The container startup banner reads the container and model license and notice text directly from the on-disk files instead of a static built-in string, so the displayed governing terms match the specific release and bundled model.

For fixes from the baseline release, refer to the standard 2.0.4 release notes.

Known Issues#

This release includes the following known issues:

  • ModelScope downloads can fail with error decoding response body on slow or unstable networks. When pulling models from a modelscope:// source over a slow or long-haul link, the transfer can drop a chunked HTTP response mid-stream and abort startup at the download step with an error decoding response body error. This is a transient, recoverable read failure originating from the source rather than a NIM defect, but NIM makes a single download attempt by default (NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=0), so the first failure ends startup. As a workaround, set a non-zero retry count — for example NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=3 — so NIM retries the download. For details, see Reliability on Slow or Unstable Networks in the model download guide.

  • LoRA is not supported with NVFP4-quantized Mixture-of-Experts (MoE) models. For MoE models such as Nemotron-3-Nano, profiles that combine NVFP4 quantization with LoRA (for example, nvfp4 TP1 LoRA) fail to start. vLLM injects LoRA adapters through its Triton-based fused-MoE kernel, but NVFP4 checkpoints dispatch to a specialized NVFP4 experts kernel instead, so engine startup aborts with an AssertionError in vllm/lora/layers/fused_moe.py. This is an upstream vLLM limitation (vLLM 0.17.1). The startup-failure fix is tracked in vLLM PR #34421, and broader native NVFP4 MoE LoRA support (Cutlass kernels) is tracked in vLLM issue #42629. As a workaround, serve these models with a non-NVFP4 profile (for example, BF16 or FP8) when LoRA is required, or use an NVFP4 profile without LoRA.

  • The default NIM_KVCACHE_PERCENT of 0.92 can cause out-of-memory (OOM) errors on some profiles during CUDA graph capture or warmup. The default budgets 92% of GPU memory for model operations, leaving 8% unreserved. On some profiles — particularly smaller GPUs — this is not enough headroom for CUDA graph capture, which can consume 1 to 5 GB. The OOM appears after weights and KV cache are allocated, during graph capture or sampler warmup. As a workaround, lower NIM_KVCACHE_PERCENT (for example, to 0.85) to leave more memory unreserved, or disable CUDA graphs with NIM_DISABLE_CUDA_GRAPH=1. For details, see During CUDA Graph Compilation or Warmup in the memory troubleshooting guide.

For baseline known issues and limitations, refer to the standard 2.0.4 release notes.

Release 2.0.4-pb6#

NIM LLM 2.0.4-pb6 is the NVIDIA AI Enterprise Production Branch 6 (PB6) release of NIM LLM. This release uses the standard NIM LLM 2.0.4 documentation set as its baseline and adds PB-specific release information on this page.

Unless a PB-specific update is listed here, the standard 2.0.4 release notes apply.

Highlights#

The following updates are specific to the 2.0.4-pb6 documentation set:

  • Updated the Support Matrix with the PB6 model-free and model-specific NIMs.

  • Clarified that PB releases provide stable and secure NVIDIA AI Enterprise release branches for mission-critical deployments.

  • Updated OpenShift SecurityContextConstraints (SCC) guidance for restricted-v2 and restricted-v3.

  • Added guidance for PB6 air-gapped deployments, including runtime cache and telemetry control environment variables.

Security Alerts#

This release includes the following security alerts:

  • CVE-2026-42311 (Pillow 12.1.1): For vision language model (VLM) deployments, do not allow untrusted image sources with multimodal serving.

  • CVE-2026-44222 (vLLM 0.17.1): Do not allow untrusted client input with multimodal serving.

  • CVE-2026-27893 (vLLM 0.17.1): Do not load VLMs from untrusted sources. Some VLMs (for example, Kimi-K2.5 and NemotronVL) automatically enable remote code execution.

Bug Fixes#

No PB-specific known issues are documented as fixed in this release. For fixes from the baseline release, refer to the standard 2.0.4 release notes.

Known Issues#

No PB-specific known issues are documented for this release. For baseline known issues and limitations, refer to the standard 2.0.4 release notes.