Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for each NIM LLM release.

Release 2.0.8#

Highlights#

This release upgrades the inference backend to vLLM 0.23.0 and refreshes the model-specific and model-free NIM containers to 2.0.8.

vLLM 0.23.0#

The inference backend is updated from vLLM 0.22.1 to vLLM 0.23.0, bringing improvements to functionality, performance, and stability.

Model-Specific NIM Updates#

This release includes updated 2.0.8 versions of the following model-specific certified NIM containers:

  • gpt-oss-120b

  • gpt-oss-20b

  • llama-3.1-70b-instruct

  • llama-3.1-8b-instruct

  • llama-3.3-70b-instruct

  • llama-3.3-nemotron-super-49b-v1.5

  • nemotron-3-nano

  • nemotron-3-super-120b-a12b

  • starcoder2-7b

Each container ships with curated model weights, validated quantization profiles, and optimal runtime configurations. Refer to the Support Matrix for Certified NIMs for supported profiles and verified GPUs.

Model-Free NIM Update#

This release includes an updated 2.0.8 version of the model-free NIM container. Refer to Model-Free NIM for details.

Security Fixes#

This release includes the following security fixes:

  • mooncake-transfer-engine — The bundled libetcd_wrapper.so was rebuilt with Go 1.25.11 and golang.org/x/net 0.55.0 to address CVE-2026-39821 (Critical, golang.org/x/net/idna) along with the rolling Go standard-library and golang.org/x CVE stream in that component.

  • nginx — Updated from 1.30.1 to 1.30.3 to address CVE-2026-42055 (buffer overflow in the proxy and gRPC modules, Medium) and CVE-2026-48142 (buffer overread in the charset module, Low) in the front-door proxy.

Known Issues#

This release includes the following known issues and limitations:

  • NVFP4 profiles on B300-SXM6-AC and GB300 GPUs may return finish_reason: "length" where "stop" is expected, and requests that enable logprobs or top_logprobs may fail with HTTP 400 and the error Out of range float values are not JSON compliant: nan. This affects the NVFP4 profiles of llama-3.1-70b-instruct, llama-3.3-70b-instruct, llama-3.3-nemotron-super-49b-v1.5, and nemotron-3-nano. To work around the logprobs issue, omit logprobs and top_logprobs from the request.

  • FP8 + LoRA profiles fail engine initialization for nemotron-3-super-120b-a12b and nemotron-3-nano. During engine initialization, the vLLM LoRA Triton kernel rejects the FP8 activation dtype with Unsupported lhs dtype fp8e4nv, and the container exits. To work around this issue, use a non-FP8 LoRA profile (BF16 or NVFP4 base), or deploy an FP8 base model profile without LoRA adapters.

  • The fused MoE LoRA path used by nemotron-3-nano limits the maximum LoRA rank to 128. Starting the NIM with --max-lora-rank 256 fails during engine initialization with fused_moe_lora_one_shot supports max_lora_rank<=128; got rank=256, and the container exits. The nemotron-3-nano LoRA profiles remain supported and start normally with a LoRA rank of 128 or lower. To work around this issue, set --max-lora-rank to 128 or lower.

  • nemotron-3-nano LoRA profiles crash on H100, H200, GH200, and Blackwell GPUs with a CUDA illegal memory access error during inference. The vLLM engine terminates, and the server returns HTTP 500 errors for all subsequent requests. Non-LoRA profiles and older GPUs (A100, L40S) are not affected. There is no workaround for this issue.

  • nemotron-3-nano NVFP4 LoRA profiles with TP values greater than one fail to deploy.

  • NVFP4 MoE models (for example, nemotron-3-nano) on GB10 and RTX PRO 4500 Blackwell Server Edition GPUs might crash at startup with CUDA error: misaligned address during full CUDA graph capture. Dense NVFP4 models are unaffected.

    Workaround: Disable FlashInfer NVFP4 MoE kernels.

    Set the following environment variable before starting the container:

    export VLLM_USE_FLASHINFER_MOE_FP4=0

  • FP8 profiles for llama-3.3-nemotron-super-49b-v1.5 fail to start on NVIDIA RTX PRO 4500 and RTX PRO 6000 Blackwell Server Edition GPUs, so the affected configurations are not supported in 2.0.8. On RTX PRO 4500 Blackwell Server Edition, the TP2 FP8 profiles (base and LoRA) are affected; on RTX PRO 6000 Blackwell Server Edition, the TP4 and TP8 FP8 profiles (base and LoRA) are affected and have been removed from the support matrix. During CUDA graph capture, the FlashInfer FP8 GEMM (invoked through vLLM) raises RuntimeError: Plan index 10 is invalid in the cuDNN backend, and the server never becomes ready. There is no workaround other than selecting a non-FP8 (BF16 or NVFP4) profile on these GPUs.

  • An upstream vLLM issue might corrupt MoE base model responses when the same NIM deployment loads LoRA adapters. This issue affects all MoE models served with LoRA adapters loaded on the server. Refer to the upstream vLLM issue for details. To work around this issue, deploy the base model without loading any LoRA adapters.

  • An upstream vLLM issue might cause MoE LoRA responses to produce corrupted output when the base model is an FP8-quantized checkpoint. Refer to the upstream vLLM issue for details.

  • An unreported vLLM issue might cause the vLLM worker to fail on four or eight H100-NVL GPUs with an EngineDeadError.

    Example Error Message

    You might encounter the following error message:

    Worker proc VllmWorker-2 died unexpectedly, shutting down executor.
RuntimeError: cancelled (shm_broadcast.py:677 acquire_read)
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

  • A stale FlashInfer compilation cache might cause a deployment crash on Blackwell GPUs with a Ninja build error referencing a missing source file.

    Example Error Message

    You might encounter the following error message:

    ninja: error: '/usr/local/lib/python3.12/dist-packages/flashinfer/data/csrc/fp4_gemm_cutlass_sm103.cu' ... missing and no known rule to make it
    Workaround: Clear the FlashInfer cache.

    Remove stale cached FlashInfer files on the host before starting the container. These files are nested under the flashinfer directory in the NIM cache mounted to the container.

  • ModelScope manifest downloads can fail with error decoding response body on slow or unstable networks. The NIM SDK defaults to zero retries (NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=0), so a single transient read failure aborts the download. Other model sources (HuggingFace, NGC, S3, GCS, local) are not affected.

    Workaround: Increase the retry count.

    Set the following environment variable before starting the container to allow the download to retry on failure:

    export NIM_MANIFEST_DOWNLOAD_MAX_RETRY_COUNT=5

For information about past updates and older versions, refer to the previous release notes.