Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for each NIM LLM release.

Release 2.0.9#

Highlights#

This release upgrades the inference backend to vLLM 0.25.1, adds expert-parallel and data-parallel deployment controls, and refreshes the model-specific and model-free NIM containers to 2.0.9.

vLLM 0.25.1#

The inference backend is updated from vLLM 0.23.0 to vLLM 0.25.1, bringing improvements to functionality, performance, and stability.

Expert and Data Parallelism#

The NIM configuration, vLLM backend, and Helm chart now support expert-parallel and data-parallel deployment controls. Refer to Advanced Configuration for details.

Runtime Configuration Passthrough#

The runtime_config.json file now accepts a reserved NIM_PASSTHROUGH_ARGS key. The value uses the same CLI-style parsing, type handling, and argument validation as the NIM_PASSTHROUGH_ARGS environment variable. Explicit keys in the same file take precedence over values in the passthrough string. Refer to Advanced Configuration for details.

Benchmarking Documentation#

The documentation now includes a benchmarking reference that describes supported tools, metrics, and recommended workflows. Refer to Benchmarking for details.

Model-Specific NIM Updates#

This release includes updated 2.0.9 versions of the following model-specific certified NIM containers:

  • gpt-oss-120b

  • gpt-oss-20b

  • llama-3.1-70b-instruct

  • llama-3.1-8b-instruct

  • llama-3.3-70b-instruct

  • llama-3.3-nemotron-super-49b-v1.5

  • nemotron-3-nano

  • nemotron-3-super-120b-a12b

  • nemotron-3-ultra-550b-a55b

  • starcoder2-7b

Each container ships with curated model weights, validated quantization profiles, and optimal runtime configurations. Refer to the Support Matrix for NIMs for supported profiles and verified GPUs.

Model-Free NIM Update#

This release includes an updated 2.0.9 version of the model-free NIM container. Refer to Model-Free NIM for details.

Performance#

nemotron-3-ultra-550b-a55b: MTP Speculative Decoding#

MTP speculative decoding with num_speculative_tokens=5 was validated on NVIDIA B200 at TP=4 using the AIPerf SPEED-Bench subroutine across all 11 SPEED-Bench workload categories. The following table reports per-category acceptance rate (AR) and acceptance length (AL). The final column is the cross-category average.

Metric

coding

humanities

math

multilingual

qa

rag

reasoning

roleplay

stem

summarization

writing

avg

Accept Rate

0.69

0.55

0.68

0.72

0.53

0.71

0.57

0.39

0.60

0.61

0.44

0.59

Accept Length

4.45

3.73

4.38

4.60

3.67

4.53

3.87

2.97

3.99

4.07

3.19

3.95

Acceptance rates range from 0.39 (roleplay) to 0.72 (multilingual), reflecting the expected workload sensitivity of speculative decoding.

nemotron-3-ultra-550b-a55b: H100 and H200 Support#

nemotron-3-ultra-550b-a55b is now validated on NVIDIA H100 and H200 GPUs using NVFP4 profiles. H100 and H200 do not include native FP4 tensor cores, so NVFP4 inference runs using software emulation on these GPUs:

  • NVIDIA-H100-80GB-HBM3 and NVIDIA-H100-NVL: vllm-nvfp4-tp8-pp1 (8 GPUs).

  • NVIDIA-H200: vllm-nvfp4-tp4-pp1 (4 GPUs). H200 validation covers this one NVFP4 profile and is less extensive than the Blackwell validation.

Refer to the Support Matrix for NIMs for the full verified GPU list and profile table.

Security Fixes#

This release includes the following security fixes:

  • OpenSSL — Updated the Ubuntu 24.04 openssl and libssl3 packages to 3.0.13-0ubuntu3.11 or later to address CVE-2026-45447 (High).

  • PyNvVideoCodec — Updated PyNvVideoCodec from 2.0.4 to 2.0.5, which bundles FFmpeg 8.1.2, and removed the stale system-level 2.0.4 package to address CVE-2026-8461 (High).

Bug Fixes#

This release includes the following bug fixes:

  • Requests without a body, including POST /v1/responses/{id}/cancel, no longer fail with HTTP 415 when they omit the Content-Type header.

  • VLM video requests use the hardware-accelerated PyNvVideoCodec backend while preserving model-specific frame sampling. A failed NVDEC request no longer leaves the decoder unusable for subsequent requests.

  • A CUDA out-of-memory error during vLLM model execution now fails only the affected requests instead of terminating the engine process.

  • The vLLM CPU backend no longer terminates during structured-output requests because of unsupported pinned-memory allocation.

Known Issues#

This release includes the following known issues and limitations:

  • On NVLink-bridged Hopper GPUs (H100-NVL, H200-NVL, and GH200), tensor-parallel profiles with TP ≥ 2 can crash at startup during CUDA graph capture with a CUDA error: an illegal memory access was encountered. This vLLM 0.25.x regression affects the TP ≥ 2 profiles of llama-3.3-70b-instruct, llama-3.1-70b-instruct, llama-3.3-nemotron-super-49b-v1.5, nemotron-3-super-120b-a12b, nemotron-3-nano, gpt-oss-120b, and gpt-oss-20b.

    Workaround: Disable the fused allreduce and RMS normalization pass.

    Set the following environment variable before starting the container:

    export NIM_PASSTHROUGH_ARGS="--compilation-config '{\"pass_config\":{\"fuse_allreduce_rms\":false}}'"

  • LoRA profiles can fail engine initialization with a CUDA out-of-memory error during full CUDA graph capture on lower-VRAM GPUs (A100-40GB, L40S, A10G, and RTX PRO 4500 Blackwell Server Edition). This vLLM 0.25.x regression affects LoRA profiles of llama-3.3-70b-instruct, llama-3.1-70b-instruct, llama-3.3-nemotron-super-49b-v1.5, and gpt-oss-120b. The configurations that still exceed the memory budget after applying the workaround have been removed from the support matrix.

    Workaround: Use piecewise CUDA graph mode.

    Set the following environment variable before starting the container:

    export NIM_PASSTHROUGH_ARGS="--compilation-config '{\"cudagraph_mode\":\"PIECEWISE\"}'"

  • On GB10 GPUs, /v1/responses requests can time out for NVFP4 TP1 profiles of llama-3.3-nemotron-super-49b-v1.5 and the FP8 TP1 LoRA profile of nemotron-3-nano because of the reduced decode throughput on that lower-TDP platform. The server starts and serves normally on larger GPUs. To work around this issue, use a larger GPU.

  • NVFP4 TP8 and TP4 profiles of llama-3.3-nemotron-super-49b-v1.5 and nemotron-3-super-120b-a12b on B300-SXM6-AC GPUs reach Application startup complete but can exceed the deployment readiness or restart time limit during startup. The engine does not crash, but startup can take approximately 27 minutes. If the deployment platform permits it, increase the readiness time limit.

  • The NIM Operator does not support nemotron-3-ultra-550b-a55b in this release. The model manifest exceeds the Kubernetes 1 MiB ConfigMap limit used by NIMCache. Use another supported deployment method for this model.

  • Deploying llama-3.3-70b-instruct through the NIM Operator (auto_deploy) may time out during provisioning on H100-NVL GPUs. This is a pre-existing provisioning-timeout signature (also present in 2.0.8), not a 2.0.9 regression. The NIM serves normally through Helm, KServe, and Docker deployment paths.

  • On RTX PRO 6000 GPUs, llama-3.3-70b-instruct requires the FlashInfer attention backend. To ensure correct behavior on this GPU, set VLLM_ATTENTION_BACKEND=FLASHINFER before starting the container.

  • ModelScope manifest downloads can fail with error decoding response body on slow or unstable networks. The NIM SDK defaults to zero retries (NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=0), so a single transient failure stops the download. Other model sources, including Hugging Face, NGC, S3, GCS, and local storage, are not affected.

    Workaround: Increase the retry count.

    Set the following environment variable before starting the container:

    export NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=5

For information about past updates and older versions, refer to the previous release notes.