Is this page helpful?

Release Notes#

This page lists changes, fixes, and known issues for each NIM LLM release.

Release 3.0.0#

Highlights#

NIM LLM 3.0.0 is the first release to introduce Dynamo mode: an opt-in path that runs the model on NVIDIA Dynamo, separating an OpenAI-compatible frontend and router from independently scaled vLLM workers. The same NIM LLM 3.0 container can be configured in standard mode instead, which matches the single-container behavior of NIM LLM 2.0. This release uses vLLM 0.26.0 and Dynamo recipes compatible with 1.4.0. Refer to Dynamo Mode Support for the full component pin list.

Version Series#

The NIM LLM 3.0 series runs alongside the NIM LLM 2.0 series, and neither supersedes the other. The 2.0 series remains the default image on NGC. Pin the 3.0.0 tag explicitly to deploy this release. Refer to Version Series for the architecture of each series, feature parity between them, and how to choose.

To move a NIM 2.0 deployment to Dynamo mode on NIM 3.0.0, refer to the Dynamo Mode Migration Guide. If you are on a 1.x NIM, consider upgrading to the more mature NIM 2.0 before upgrading to NIM 3.0. Refer to the 1.x to 2.0 Migration Guide for details.

Warning

The NIM LLM 3.0.0 release is not NIM Certified. For NIM Certified containers, use the NIM 2.0 series. Refer to NIM Offerings for the difference between NIM and NIM Certified.

Dynamo Mode#

Dynamo mode is delivered on the released NIM image, not as a separate set of containers, which keeps profile behavior, security patching, and validation consistent across both modes. The Dynamo runtime and Dynamo Python package ship inside the image and stay inactive until you set a mode selector, so you can adopt Dynamo incrementally and revert by removing it. With no selector set, the container starts in standard mode. Dynamo mode requires a vLLM image. SGLang images serve standard mode only.

Deploy the image with the recipe published for your model, GPU SKU, and workload, then override only what you need to tune. Refer to Dynamo Deployment to get started.

Dynamo Deployment Roles#

Set exactly one of the following environment variables to run the container as a single-container frontend and worker, a frontend only, or a worker only. Refer to Environment Variables for the complete variable definitions.

Single-Container Deployment#

With NIM_DYNAMO_SINGLE=1, the container runs the frontend and one worker together, using file-based discovery and a ZMQ event plane instead of etcd and NATS. Use this role for evaluation, single-GPU serving, and managed endpoints that expose one container and one port. Refer to Run NIM in Dynamo Mode.

Frontend and Worker Deployment#

With NIM_DYNAMO_FRONTEND=1, the container runs only the frontend and router and needs no GPU. With NIM_DYNAMO_WORKER=1, the container runs only a worker, needs a GPU, and registers with a frontend running elsewhere. Use these roles together for multi-pod deployments. Refer to Dynamo Graph Deployment on Kubernetes for the NVIDIA Dynamo Operator path.

KV Cache-Aware Routing#

The Dynamo frontend routes each request across interchangeable worker replicas. The default router mode, DYN_ROUTER_MODE=kv, evaluates key-value (KV) cache overlap and decode load for every worker and selects the lowest-cost replica, which raises cache hit rates for workloads that share prompt prefixes. This is not a prefill and decode split: every worker runs the full forward pass. Refer to Dynamo Performance Tuning for worker scaling and to Routing and Ports for routing configuration.

Validated Per-Model Deployment Recipes#

Each Dynamo deployment starts from a recipe published in the model’s collection in the NGC Catalog. A recipe pins the container image, model profile, deployment mode, and workload-specific worker arguments that were validated together. Each model provides a chat variant and an agentic variant.

Refer to Dynamo Mode Support for recipe configurations and the full support scope for this release.

Kubernetes Deployment with the NVIDIA Dynamo Operator#

The supported Kubernetes path is a DynamoGraphDeployment (DGD) custom resource reconciled by the NVIDIA Dynamo Operator, which produces a frontend deployment and a separate GPU worker deployment that you scale independently. Recipes in this release are validated with Dynamo Operator 1.2.0, installed from the dynamo-platform chart that also provides the required custom resource definitions. Refer to Dynamo Graph Deployment on Kubernetes.

NIM Management Endpoints on the Dynamo Frontend#

The Dynamo frontend serves the NIM management endpoints (/v1/health/*, /v1/version, /v1/metadata, /v1/manifest, /v1/license) and /v1/models through the bundled nim-management route extension, so they behave as they do in standard mode. Refer to API Reference for detailed route behavior and readiness semantics.

LoRA Adapter Serving in Dynamo Mode#

Single-container Dynamo deployments support static and dynamic LoRA with the same NIM_PEFT_SOURCE experience as standard mode. The management plane differs: the worker’s Dynamo system server, not the frontend, serves POST, GET, and DELETE on /v1/loras, on the port set by NIM_HEALTH_PORT, which defaults to 9090. Refer to Fine-Tuning with LoRA.

Dynamo Configuration Through Environment Variables#

Dynamo routing, ports, and parser selection are configured with environment variables rather than command-line flags. Refer to Environment Variables for the Dynamo variables, including DYN_ROUTER_MODE, DYN_HTTP_PORT, DYN_FRONTEND_EXTRA_ARGS, DYN_TOOL_CALL_PARSER, DYN_REASONING_PARSER, and the NIM_HEALTH_PORT / DYN_SYSTEM_PORT relationship.

Model-Specific NIM and Model-Free NIM Updates#

This release refreshes the 3.0.0 containers used by the published Dynamo recipes:

  • nemotron-3-super-120b-a12b (recipe with model-specific NIM)

  • nemotron-3-ultra-550b-a55b (recipe with model-free NIM)

Model-specific NIM containers ship with curated model weights, validated quantization profiles, and optimal runtime configurations. Refer to the Support Matrix for NIMs for supported profiles and verified GPUs, and to Model-Free NIM for the model-free container.

Behavior Changes in Dynamo Mode#

Dynamo mode starts without nginx and without the request middleware that standard mode installs, so TLS, CORS, request shaping, metrics scrape targets, parser selection, logging, and the temporary-directory layout differ from standard mode. Standard-mode deployments are unaffected. Refer to Dynamo environment variables for variables that do not apply and for Dynamo-specific parser, logging, and TMPDIR behavior.

Known Issues#

Known issues for this release are listed by deployment mode.

Dynamo Mode#

The following known issues and limitations apply to Dynamo mode:

  • The entrypoint does not reject more than one enabled mode selector. It evaluates NIM_DYNAMO_SINGLE, then NIM_DYNAMO_WORKER, then NIM_DYNAMO_FRONTEND, uses the first enabled selector, and ignores the rest without a warning. Remove the selectors you are not using.

  • Features implemented by the NIM request middleware are unavailable, and the container does not warn. Payload capture (NIM_CAPTURE_ENABLE and NIM_CAPTURE_ARGS) installs nothing. The SageMaker compatibility routes /ping and /invocations are not installed, so NIM_SAGEMAKER_MODE cannot serve a SageMaker endpoint, although its port side effect still applies and the frontend binds 8080 when a SageMaker signal is present.

Standard Mode#

Standard mode known issues for this release match those documented for the NIM 2.0 series. Refer to the NIM 2.0.10 release notes for details.