Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for this NIM LLM and VLM release.

Release 2.0.13#

Highlights#

This release delivers 13 NIMs with significant throughput optimizations, up to 2.98x depending on the NIM and the workload.

  • nemotron-3-super-120b-a12b

  • nemotron-3.5-lightning

  • nemotron-3-nano-omni-30b-a3b-reasoning

  • kimi-k2.6

  • gpt-oss-120b

  • llama-3.3-nemotron-super-49b-v1.5

  • llama-3.3-70b-instruct

  • llama-3.1-70b-instruct

  • llama-3.1-8b-instruct

  • gemma-4-31b-it

  • gemma-4-26b-a4b-it

  • qwen3.5-122b-a10b

  • qwen3.5-397b-a17b

Refer to the Support Matrix for NIMs for the throughput-optimized profiles for each of these NIMs.

It also updates the inference backend to vLLM 0.28.0.

vLLM 0.28.0#

The vLLM inference backend is updated from 0.27.1 to vLLM 0.28.0, bringing improvements to functionality, performance, and stability. The update applies to both the GPU and CPU container images.

Security Fixes#

This release addresses the following security issues:

  • pip and one related package are updated from 26.1.2 to 26.2.1, addressing GHSA-qwm4-qh6w-59xr and CVE-2026-13346.

  • Updated cuda-compat-13-0 from 580.95.05-0ubuntu1 to 580.159.03-1ubuntu1. The associated host-driver vulnerabilities require a host GPU driver update.

  • google.golang.org/grpc is updated from v1.82.1 to v1.83.2, addressing CVE-2026-84304 and GHSA-vp52-pcj8-j9qc (High).

Bug Fixes#

This release includes the following fixes:

  • The vllm serve launch command handed to the backend is now emitted as an INFO message even at the default log configuration. Previously, this command was visible only when setting NIM_LOG_LEVEL=INFO or using --dry-run. Log pipelines that retain only warning-or-higher records still drop this message.

  • vLLM passthrough now forwards --middleware additively. Previously, --middleware was blocked in NIM_PASSTHROUGH_ARGS, so an operator could not install a custom ASGI middleware class. Refer to Advanced Configuration for the ordering and Dynamo deployment constraints that apply to this flag.

  • In the model name that NIM derives when NIM_SERVED_MODEL_NAME is not set, the URI scheme is now stripped from the model source, and any : in the result is removed or replaced. Previously, a derived name could retain the source scheme or a :-separated tag. Some inference backends interpret : in the model field as a base-model:lora-adapter separator. That interpretation might cause every request to be rejected. Refer to About Model-Free NIM for the derivation rules and examples.

Support Changes#

  • By default, llama-3.3-70b-instruct accepts at most four stop strings per request in this release. A /v1/chat/completions or /v1/completions request that sets more than four stop strings is rejected with an HTTP 400 error (List should have at most 4 items after validation) instead of being served. This is a behavior change in 2.0.13: release 2.0.12 accepted more than four stop strings. The limit comes from the updated inference backend (vLLM 0.28.0) and aligns with the OpenAI API, which documents up to four stop sequences. The change applies to both streaming and non-streaming requests on all supported GPUs. Send at most four stop strings per request.

  • The 2.0.13 container for Nemotron 3.5 Lightning is available as nvcr.io/nim/nvidia/nemotron-3.5-lightning:2.0.13. Refer to nemotron-3.5-lightning for its supported profiles and GPUs.

  • For nemotron-3-super-120b-a12b, the BF16 TP4 LoRA profile is supported on NVIDIA H100-NVL in 2.0.13. The following BF16 LoRA combinations remain unsupported and are omitted from the Support Matrix for NIMs: TP2 on NVIDIA GH200-144G-HBM3e and NVIDIA H200-NVL, and TP8 on NVIDIA L40S.

  • For the NVFP4 profiles of llama-3.3-nemotron-super-49b-v1.5 with LoRA on NVIDIA GB10 (TP1) and NVIDIA RTX PRO 4500 Blackwell Server Edition (TP2), the default 131,072-token context can exceed available KV cache memory and prevent startup. Reduce NIM_MAX_MODEL_LEN to fit the available memory.

  • The following LoRA-enabled profiles of llama-3.3-70b-instruct are not supported on the listed lower-memory GPUs in this release because the server runs out of GPU memory while initializing the vLLM engine for the LoRA configuration:

    • vllm-bf16-tp8-pp1-lora on NVIDIA A100-SXM4-40GB, NVIDIA L40S, and NVIDIA RTX PRO 4500 Blackwell Server Edition

    • vllm-fp8-tp4-pp1-lora and vllm-fp8-tp8-pp1-lora on NVIDIA L40S and NVIDIA RTX PRO 4500 Blackwell Server Edition

    • vllm-nvfp4-tp4-pp1-lora and vllm-nvfp4-tp8-pp1-lora on NVIDIA RTX PRO 4500 Blackwell Server Edition

    The corresponding non-LoRA profiles are unaffected on these GPUs, and these LoRA profiles remain supported on the higher-memory GPUs listed in the Support Matrix for NIMs.

  • llama-3.1-8b-instruct adds throughput profiles for NVFP4 at TP1 on NVIDIA B200 and FP8 at TP1 on NVIDIA H200. These profiles use FP8 KV cache and prefix caching for chat workloads with repeated prefixes and do not support LoRA. Existing generic profiles remain available.

    For the 2.0.13 profile IDs and supported GPUs, refer to llama-3.1-8b-instruct. For selection and deployment instructions, refer to Model Profiles and Selection.

  • llama-3.1-70b-instruct adds throughput profiles for NVFP4 at TP2 on NVIDIA B200 and FP8 at TP2 on NVIDIA H200. These profiles use FP8 KV cache and prefix caching for chat workloads with repeated prefixes and do not support LoRA. Existing generic profiles remain available, with the following BF16 GPU coverage updates:

    • TP2 with and without LoRA is supported on NVIDIA RTX PRO 6000 Blackwell Server Edition.

    • TP8 without LoRA is supported on NVIDIA B200 and NVIDIA A100 SXM4 40GB.

    • TP2 with LoRA is supported on NVIDIA B300.

    BF16 TP2 with LoRA is not supported on NVIDIA H100 NVL because the profile requires more GPU memory than the device provides. BF16 TP1 with LoRA is not supported on NVIDIA GB300-WS because the model weights leave insufficient memory for the KV cache. These exclusions apply only to the specified LoRA profiles. For other supported combinations, refer to llama-3.1-70b-instruct.

    For the 2.0.13 profile IDs and supported GPUs, refer to llama-3.1-70b-instruct. For selection and deployment instructions, refer to Model Profiles and Selection.

  • This release adds a throughput-optimized MXFP4 TP1 profile for gpt-oss-120b. It enables an FP8 KV cache, which requires compute capability 8.9 or newer, and is not supported on NVIDIA A100-SXM4-80GB (compute capability 8.0). The default MXFP4 TP1 profile does not use an FP8 KV cache and remains supported on NVIDIA A100-SXM4-80GB. Refer to the Support Matrix for NIMs for the exact profile ID and supported GPUs.

    The following requirements apply to gpt-oss-120b in this release:

    • MXFP4 TP8 on NVIDIA A10G and NVIDIA A100-SXM4-40GB requires NIM_KVCACHE_PERCENT=0.80.

    • MXFP4 LoRA TP4 and TP8 on NVIDIA L40S, and MXFP4 LoRA TP8 on NVIDIA RTX PRO 4500 Blackwell Server Edition, require piecewise CUDA graphs.

    For configuration mechanisms and precedence, refer to Advanced Configuration.

  • The MXFP4 LoRA TP4 profile (vllm-mxfp4-tp4-pp1-lora) of gpt-oss-120b is not supported on NVIDIA RTX PRO 4500 Blackwell Server Edition in NIM 2.0.13. The released profile requires at least 33 GB of memory per GPU, so the 32 GB GPU is excluded from this profile in the support matrix. This changes the support coverage from NIM 2.0.12. For LoRA on this GPU, use the TP8 profile (vllm-mxfp4-tp8-pp1-lora). The non-LoRA MXFP4 TP4 and TP8 profiles also remain supported on this GPU.

  • Docker-based deployment tests for mistral-small-4-119b-2603 were excluded from release-candidate qualification for NIM 2.0.13. The container was validated to start and serve through automated API tests and Kubernetes-based deployment. This is a test-coverage exception; no model-specific Docker deployment defect was identified, and the supported GPU/profile combinations are unchanged.

Known Issues#

This release includes the following known issues and limitations. For shared diagnostic steps, refer to Troubleshooting GPU Memory Out-of-Memory Errors and Troubleshooting Request Timeouts and Responses. Those steps do not establish workarounds for the model-specific failures listed here.

Show 21 Known Issues and Limitations

  • nemotron-3-nano supports LoRA adapters with rank 128 or lower because of a fused mixture-of-experts kernel limit. Adapters with greater ranks remain unsupported. Workaround: Use adapters with rank 128 or lower.

  • nemotron-3-nano NVFP4 profiles, including the NVIDIA B200 throughput profile, can produce unrelated output or reach the generation limit without a final answer. Workaround: Use a BF16 or FP8 profile and validate its output for your application.

  • Some nemotron-3-nano /v1/responses requests exceed 60 seconds: empty input on NVFP4 TP1/NVIDIA B200 or FP8 TP1 LoRA/NVIDIA H100-NVL, and a short nonempty prompt on NVFP4 TP1/NVIDIA GB300-WS. Completion with a longer timeout is unverified. Refer to Requests Exceed the Client Timeout for diagnosis.

  • nemotron-3-ultra-550b-a55b defaults to the built-in nemotron_v3 reasoning parser instead of the checkpoint’s ultra_v3 parser. Workaround: Set the following variable at startup. It preserves the default tool configuration; preserve any additional arguments required by your deployment.

    NIM_PASSTHROUGH_ARGS="--reasoning-parser-plugin ultra_v3_reasoning_parser.py --reasoning-parser ultra_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder"

  • nemotron-3-ultra-550b-a55b text completions at temperature=0 can repeat until max_tokens without emitting the requested stop string, in streaming and non-streaming modes. Affected GPUs are NVIDIA B200, B300-SXM6-AC, GB200, GB300, H100 80GB HBM3, H100-NVL, and H200, including optimized NVFP4 TP4 profiles. Workaround: Set max_tokens to bound generated output; this does not correct the repetitive response.

  • gpt-oss-120b LoRA adapters above rank 128 fail at startup with a fused mixture-of-experts kernel rank-limit assertion. Workaround: Use adapters with rank 128 or lower. Refer to Out of Memory During LoRA Adapter Allocation for the distinction between rank limits and memory failures.

  • gpt-oss-120b MXFP4 LoRA TP4 (vllm-mxfp4-tp4-pp1-lora) exhausts adapter memory on NVIDIA L40S with --max-lora-rank 64 --max-loras 8. Workaround: Leave --max-loras at its default. This does not establish limits for other ranks or TP sizes; refer to Out of Memory During LoRA Adapter Allocation.

  • These qwen3.5-122b-a10b LoRA profiles fail at startup with default settings because a 2.05 GiB multimodal GPU memory reservation leaves insufficient KV cache memory. Text and image serving are unavailable even without video input.

    Affected Configurations

    Profile ID prefix

    Precision

    Tensor parallel size

    GPU

    c06f2ba3

    BF16

    4

    NVIDIA A100-SXM4-80GB

    c06f2ba3

    BF16

    4

    NVIDIA H100-80GB-HBM3

    4d73f737

    BF16

    2

    NVIDIA GH200 144GB HBM3e

    4d73f737

    BF16

    2

    NVIDIA H200

    4d73f737

    BF16

    2

    NVIDIA H200-NVL

    7897bbe8

    FP8

    4

    NVIDIA L40S

    e1274533

    BF16

    8

    NVIDIA L40S

    4ce2ff55

    FP8

    8

    NVIDIA RTX PRO 4500 Blackwell Server Edition

  • qwen3.5-122b-a10b FP8 TP2 LoRA (profile prefix 00282e75) runs out of memory during CUDA graph/KV cache profiling on NVIDIA H100-80GB-HBM3 with default settings, preventing text and image serving. Non-LoRA profiles are unaffected. Refer to Troubleshooting GPU Memory Out-of-Memory Errors for diagnosis.

  • These qwen3.5-122b-a10b BF16 configurations fail during FlashInfer trtllm_bf16_moe warm-up with a CUDA illegal-memory-access error or worker exit, preventing text and image serving. Other profiles serve normally on these GPUs.

    Affected Configurations

    Profile ID prefix

    Precision

    Tensor parallel size

    LoRA

    GPU

    4d73f737

    BF16

    2

    Enabled

    NVIDIA B200

    cddf999c

    BF16

    4

    Disabled

    NVIDIA B200

    b1d3696f

    BF16

    8

    Disabled

    NVIDIA B200

    4d73f737

    BF16

    2

    Enabled

    NVIDIA GB200

    cddf999c

    BF16

    4

    Disabled

    NVIDIA GB200

    cddf999c

    BF16

    4

    Disabled

    NVIDIA GB300

  • qwen3.5-397b-a17b chat requests with tool_choice=required or a named function can return HTTP 200 with empty tool_calls across multiple profile and GPU combinations. Workaround: Use tool_choice=auto if optional tool selection is acceptable. Refer to A Forced Tool Call Is Missing.

  • qwen3.5-122b-a10b forced-tool requests can return HTTP 200 with empty tool_calls. Failures increased from 2.0.12 on all shared tested GPUs: NVIDIA B200, GB200, H200, and H100 80GB HBM3. Most forced requests succeed, and unforced selection is unaffected. Workaround: Use tool_choice=auto if optional tool selection is acceptable. Refer to A Forced Tool Call Is Missing.

  • qwen3.5-397b-a17b NVFP4 TP4 on NVIDIA B200 returned reasoning without a final answer for an image chat request with temperature=0.7, frequency_penalty=0.5, presence_penalty=0.5, and max_tokens=2000. The HTTP 200 response had content=null and finish_reason=stop.

  • kimi-k2.6 text completions can ignore stop strings on NVIDIA B200 and B300-SXM6-AC in streaming and non-streaming modes. Chat is unaffected on all supported GPUs. Workaround: Use NVIDIA GB200, GB300, or H200, where completions are unaffected, when completion stop strings are required. Refer to Completion Stop Strings Are Not Honored.

  • kimi-k2.6 /v1/responses requests with empty input on NVIDIA GB300 can exceed 60 seconds, compared with under a second in 2.0.12. Nonempty requests and NVIDIA B200, B300-SXM6-AC, GB200, and H200 are unaffected. Use nonempty input or an unaffected GPU if prompt completion is required; no fix is available for the affected case. Refer to Requests Exceed the Client Timeout.

  • kimi-k2.6 supports text and image input only. Video input to /v1/chat/completions returns HTTP 500 on all supported GPUs with both NVFP4 TP4 and INT4 TP8 profiles. Workaround: Use text or image input; video input has no workaround in 2.0.13.

  • Explicit NIM_MODEL_PROFILE selection for mistral-small-4-119b-2603 bypasses GPU compatibility checks, so an unsupported or incorrect SKU profile can start and report healthy. Automatic selection still enforces the checks. Workaround: Use automatic selection or a profile that list-model-profiles reports as compatible. Refer to the Support Matrix for NIMs.

  • qwen3.5-397b-a17b FP8 TP8 LoRA (vllm-fp8-tp8-pp1-lora) exceeds GPU memory at startup on NVIDIA H100 80GB HBM3 and is unsupported there in 2.0.13. The non-LoRA FP8 TP8 profile is unaffected. Workaround: Use a higher-memory GPU listed for the LoRA profile in the Support Matrix for NIMs. Generic tuning does not establish support for the excluded GPU.

  • llama-3.3-nemotron-super-49b-v1.5 NVFP4 TP2 text completions can ignore stop strings on NVIDIA B200, B300, and GB300 in streaming and non-streaming modes. Chat, BF16, and other quantized profiles are unaffected. Refer to Completion Stop Strings Are Not Honored for diagnosis.

  • llama-3.3-nemotron-super-49b-v1.5 NVFP4 TP1 on NVIDIA GB10 can exceed client timeouts for large generations while the server remains healthy. Chat and smaller or capped completions are unaffected. Workaround: Cap output, increase the client timeout, or use a faster GPU as described in Requests Exceed the Client Timeout. There is no 2.0.12 API test baseline for this profile.

  • gpt-oss-20b text completions can ignore stop strings on all supported GPUs, in streaming and non-streaming modes. Chat is unaffected. Refer to Completion Stop Strings Are Not Honored for diagnosis.

For information about past updates and older versions, refer to the previous release notes.