Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for each NIM LLM release.

Release 2.0.10#

Highlights#

This release upgrades the inference backend to vLLM 0.26.0, adds speculative-decoding profile defaults and read-only root filesystem support, and refreshes the model-specific and model-free NIM containers to 2.0.10.

vLLM 0.26.0#

The inference backend is updated from vLLM 0.25.1 to vLLM 0.26.0, bringing improvements to functionality, performance, and stability.

Speculative Decoding Profile Defaults#

Profiles can now enable speculative decoding by default on a per-profile basis. The new NIM_SPECDEC_ENABLE environment variable provides a tri-state override: unset follows the profile default, 1 forces speculative decoding on, and 0 forces it off. Refer to Advanced Configuration for details.

Read-Only Root Filesystem Support#

The container now supports running with a read-only /opt/nim by setting the NIM_WRITABLE_ROOT environment variable to a writable path. This enables deployments with readOnlyRootFilesystem security policies.

Audio Modality for Omni-Class Models#

The vLLM container restores audio-input support for omni-class models through vllm[audio] and a source-built PyAV linked against an LGPL-safe, audio-decoders-only FFmpeg.

Model-Specific NIM Updates#

This release includes updated 2.0.10 versions of the following model-specific certified NIM containers:

  • gpt-oss-120b

  • gpt-oss-20b

  • llama-3.1-70b-instruct

  • llama-3.1-8b-instruct

  • llama-3.3-70b-instruct

  • llama-3.3-nemotron-super-49b-v1.5

  • nemotron-3-nano

  • nemotron-3-super-120b-a12b

  • nemotron-3-ultra-550b-a55b

  • starcoder2-7b

Each container ships with curated model weights, validated quantization profiles, and optimal runtime configurations. Refer to the Support Matrix for NIMs for supported profiles and verified GPUs.

This release includes the 2.0.9-variant version of the following model-specific NIM container:

  • nemotron-3.5-lightning-30b-a3b

For more information, refer to Get Started with Nemotron 3.5 Lightning.

Model-Free NIM Update#

This release includes an updated 2.0.10 version of the model-free NIM container. Refer to Model-Free NIM for details.

Security Fixes#

This release includes the following security fixes:

  • OpenCV — Rebuilt the bundled opencv-python-headless package without FFmpeg, removing the royalty-bearing AAC/H.264/HEVC codecs and their associated CVEs from the image. OpenCV image codecs (JPEG, PNG, WebP) are unaffected; OpenCV video I/O is not used by any supported model.

  • PyAV / audio FFmpeg — Replaced the prebuilt av wheel, which vendors a full FFmpeg including the GPL x264 and x265 encoders, with av 18.0.0 built from source against an audio-decoders-only, LGPL-safe FFmpeg 8.1.2. No GPL codecs ship in the image, and audio decoding for omni-class models links against the audited system FFmpeg.

  • nginx — Updated nginx to 1.31.3 to address three CVEs in the bundled reverse proxy.

  • Ray — Updated ray from 2.55.1 to 2.56.1 to address GHSA-hhrp-gw25-jr43.

  • Mooncake — Rebuilt the mooncake wheel with Go toolchain 1.25.12 to address CVE-2026-39822 and updated its gRPC dependency to 1.82.1 to address GHSA-hrxh-6v49-42gf.

Bug Fixes#

This release includes the following bug fixes:

  • A corrupt or oversized video in one request no longer leaves the hardware NVDEC decoder unusable for subsequent requests on the same worker. Invalid videos now return HTTP 400 instead of HTTP 500, and the decoder recovers automatically.

  • VLM video requests route the hardware-accelerated PyNvVideoCodec backend through vLLM’s supported --media-io-kwargs selection path, preserving each model’s built-in frame sampling. Video requests no longer crash with ModuleNotFoundError: No module named 'cv2'.

  • Tool-call parsing for Llama Nemotron models now handles several previously misparsed tool-calling patterns correctly.

  • Model downloads that fail because of guest-access restrictions on gated models now surface a hint identifying the gated model instead of a generic authorization error.

  • Data-parallel attention is now enabled correctly when data_parallel_size is greater than 1 on the SGLang backend.

Known Issues#

This release includes the following known issues and limitations:

  • On NVLink-bridged Hopper GPUs (H100-NVL, H200-NVL, and GH200), tensor-parallel profiles with TP ≥ 2 can crash at startup during CUDA graph capture with a CUDA error: an illegal memory access was encountered. This vLLM regression carries forward from 2.0.9 and is still present in vLLM 0.26.0. It affects the TP ≥ 2 profiles of llama-3.3-70b-instruct, llama-3.1-70b-instruct, llama-3.3-nemotron-super-49b-v1.5, nemotron-3-super-120b-a12b, nemotron-3-nano, gpt-oss-120b, and gpt-oss-20b. A fix is expected in the next release.

    Workaround: Disable the fused allreduce and RMS normalization pass.

    Set the following environment variable before starting the container:

    export NIM_PASSTHROUGH_ARGS="--compilation-config '{\"pass_config\":{\"fuse_allreduce_rms\":false}}'"

  • FP8 profiles of llama-3.3-nemotron-super-49b-v1.5 can fail on NVIDIA B200 with out-of-memory or CUBLAS_STATUS_EXECUTION_FAILED errors caused by a vLLM 0.26 FP8 scaled-matrix-multiply kernel regression on Blackwell. Workaround: Set VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel.

  • The BF16 TP1 profile of llama-3.1-8b-instruct can fail with an out-of-memory error on NVIDIA GB10 because of a vLLM 0.26.0 regression on that unified-memory platform. The FP8 and NVFP4 profiles are not affected. There is no workaround for this issue in NIM 2.0.10.

  • The BF16 TP1 LoRA profile of nemotron-3-super-120b-a12b and the BF16 TP8 LoRA profile of nemotron-3-ultra-550b-a55b crash at startup on Blackwell GPUs (observed on B200, B300-SXM6-AC, GB300, and GB300-WS) with a mixture-of-experts LoRA shape assertion error caused by a vLLM 0.26.0 regression. Other tensor-parallel configurations of the BF16 LoRA profiles, and the base BF16 profiles, are not affected. There is no workaround for this issue in NIM 2.0.10; the affected configurations have been removed from the support matrix and will be restored when the fix lands in the next release.

  • The fused mixture-of-experts kernels used by gpt-oss-120b and gpt-oss-20b support a maximum LoRA rank of 128. Deploying LoRA adapters with rank 256 fails at startup. Use adapters with rank 128 or lower.

  • The LoRA profile of gpt-oss-20b on NVIDIA A10G fails at startup with a KV-cache out-of-memory error at the default 131072-token context length. Workaround: Reduce the context length by setting NIM_MAX_MODEL_LEN to a smaller value.

  • On GB10 GPUs, /v1/responses requests can time out for NVFP4 TP1 profiles of llama-3.3-nemotron-super-49b-v1.5 and the FP8 TP1 LoRA profile of nemotron-3-nano because of the reduced decode throughput on that lower-TDP platform. The server starts and serves normally on larger GPUs. To work around this issue, use a larger GPU.

  • NVFP4 TP8 and TP4 profiles of llama-3.3-nemotron-super-49b-v1.5 and nemotron-3-super-120b-a12b on B300-SXM6-AC GPUs reach Application startup complete but can exceed the deployment readiness or restart time limit during startup. The engine does not crash, but startup can take approximately 27 minutes. If the deployment platform permits it, increase the readiness time limit.

  • Streaming /v1/responses requests to nemotron-3-nano and llama-3.3-nemotron-super-49b-v1.5 can hang when using server-sent events (SSE). Workaround: Use non-streaming requests, or cap max_output_tokens.

  • LoRA profiles can fail engine initialization with a CUDA out-of-memory error during full CUDA graph capture on lower-VRAM GPUs (A100-40GB, L40S, A10G, and RTX PRO 4500 Blackwell Server Edition). This carries forward from 2.0.9 and affects LoRA profiles of llama-3.3-70b-instruct, llama-3.1-70b-instruct, llama-3.3-nemotron-super-49b-v1.5, and gpt-oss-120b. The configurations that still exceed the memory budget after applying the workaround have been removed from the support matrix.

    Workaround: Use piecewise CUDA graph mode.

    Set the following environment variable before starting the container:

    export NIM_PASSTHROUGH_ARGS="--compilation-config '{\"cudagraph_mode\":\"PIECEWISE\"}'"

  • The NIM Operator does not support nemotron-3-ultra-550b-a55b in this release. The model manifest exceeds the Kubernetes 1 MiB ConfigMap limit used by NIMCache. Use another supported deployment method for this model.

  • Deploying llama-3.3-70b-instruct through the NIM Operator (auto_deploy) may time out during provisioning on H100-NVL GPUs. This carries forward from 2.0.9 and is not a 2.0.10 regression. The NIM serves normally through Helm, KServe, and Docker deployment paths.

  • On RTX PRO 6000 GPUs, llama-3.3-70b-instruct requires the FlashInfer attention backend. This carries forward from 2.0.9. Workaround: Set VLLM_ATTENTION_BACKEND=FLASHINFER before starting the container.

  • ModelScope manifest downloads can fail with error decoding response body on slow or unstable networks. The NIM SDK defaults to zero retries (NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=0), so a single transient failure stops the download. Other model sources, including Hugging Face, NGC, S3, GCS, and local storage, are not affected.

    Workaround: Increase the retry count.

    Set the following environment variable before starting the container:

    export NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=5

For information about past updates and older versions, refer to the previous release notes.