Is this page helpful?

Migrate to Dynamo Mode#

This guide identifies the configuration and operational changes required to move a standard NIM 2.x vLLM deployment to Dynamo mode in NIM 3.0.0, validate the result, and preserve a rollback path. For end-to-end single-container and Kubernetes procedures, use the corresponding deployment guides.

Note

Dynamo mode is generally available with NIM 3.0.0. Existing 2.x containers remain on their current release line. You cannot enable Dynamo mode in an existing 2.x container.

Plan the Migration#

Before changing a deployment:

  1. Confirm that the model and GPU configuration appear in Support Matrix, and obtain the model-specific recipe from the model’s NGC collection. Use the image and profile pinned by that recipe together; substituting either one independently is not a validated migration. SGLang is not supported by this deployment path.

  2. Choose a topology: a single container or Kubernetes with a DynamoGraphDeployment. Start with Quickstart for a single-container deployment and Kubernetes Deployment for a cluster deployment.

  3. Record the current image, environment, profile, ports, probes, ingress settings, metrics scrape target, and any explicit TMPDIR value.

  4. Do not carry a NIM 2.x profile ID into the new deployment. For a split deployment, retain the NIM_MODEL_PROFILE pinned by the NIM 3.0 recipe on the frontend and every worker. Refer to Model Profiles and Selection.

For complete mode-selector definitions, refer to Environment Variables.

Configuration Changes#

The following table compares standard NIM 2.0 configuration and behavior with Dynamo mode in NIM 3.0.0:

Area

Standard NIM 2.0

Dynamo Mode in NIM 3.0.0

Process topology

nginx fronts one vLLM server

A Dynamo frontend routes requests to one or more Dynamo-wrapped vLLM workers

Mode selection

No Dynamo selector

Set exactly one of NIM_DYNAMO_SINGLE, NIM_DYNAMO_FRONTEND, or NIM_DYNAMO_WORKER on each container

Backend

vLLM or SGLang, depending on the image

vLLM only

Public HTTP port

NIM_SERVER_PORT, normally 8000

DYN_HTTP_PORT, falling back to NIM_SERVER_PORT and then 8000

Worker system port

Not used for the standard request path

Configure NIM_HEALTH_PORT; the entrypoint derives DYN_SYSTEM_PORT; default 9090

Readiness

/v1/health/ready through nginx

Frontend: /v1/health/ready on the public port; workers: /v1/health/ready on the system port

Metrics

/v1/metrics represents the selected backend

/v1/metrics exposes frontend metrics; /metrics on each worker system port exposes vLLM metrics

TLS, CORS, and proxy policy

Implemented by NIM nginx settings

nginx is not started; provide and validate these controls at another layer

OpenAPI

NIM middleware merges NIM management paths into /openapi.json

The Dynamo frontend schema does not include that NIM middleware augmentation

LoRA management

vLLM load and unload endpoints

Dynamo worker adapter API; refer to Fine-Tuning with LoRA

Management Routes#

The Dynamo frontend receives NIM management routes through a frontend route extension, which the entrypoint enables automatically for the released Dynamo runtime. Use /v1/health/ready for frontend readiness. After readiness succeeds, query /v1/models to verify that the expected model registered. Refer to API Reference for the current route behavior.

Tool Calling and Reasoning Parsers#

Dynamo parser selection is worker-side. Set DYN_TOOL_CALL_PARSER and DYN_REASONING_PARSER consistently on every worker. Do not reuse the standard vLLM API-server options --enable-auto-tool-choice, --tool-call-parser, --tool-parser-plugin, or --chat-template: NIM retains them on the converted command, but the strict dynamo.vllm worker parser rejects them and startup fails.

Reasoning-parser options are different: --reasoning-parser and --reasoning-parser-plugin are engine arguments accepted by dynamo.vllm. NIM resolves a checkpoint-relative reasoning plugin path before launching the worker. The current Dynamo path does not accept a custom tool-parser plugin or worker-side chat template.

The Dynamo frontend does not receive the standard NIM ASGI middleware or its merged OpenAPI document. Validate tool calls and reasoning output end to end through the frontend after migration. For parser configuration, refer to Custom Parsers and Chat Templates.

Temporary and Workspace State#

When TMPDIR is not set explicitly, the entrypoint uses /tmp for Dynamo roles if the directory is writable. For standard mode, or when /tmp is read-only, the entrypoint uses a temporary directory on the writable root filesystem. An explicit TMPDIR setting takes precedence. In split deployments, ensure that the directory allows execution and is local to each pod.

Worker replicas use a deterministic workspace identity so that their model-card metadata matches. The frontend creates its own metadata-only workspace. Do not share a manually chosen temporary workspace between unrelated deployments.

Apply the Migration#

  1. Deploy the image pinned by the model-specific NIM 3.0 recipe. Use Quickstart for a container deployment or Kubernetes Deployment for a cluster deployment.

  2. Preserve the required credentials, but use the profile and worker arguments pinned by the NIM 3.0 recipe. In a split deployment, keep the same NIM_MODEL_PROFILE on the frontend and workers.

  3. Configure the role-specific probes documented by the recipe. Frontend readiness uses /v1/health/ready on the public port. Worker readiness uses /v1/health/ready on the worker system port, which defaults to 9090.

  4. Move TLS, CORS, request limits, timeout policy, and forwarded-header policy to an ingress or proxy that fronts the Dynamo frontend.

  5. Scrape frontend metrics at /v1/metrics. If dashboards or alerts require vLLM metrics, also scrape /metrics on every worker system port. Refer to Logging and Observability.

  6. Verify the worker log contains Starting Dynamo worker: python -m dynamo.vllm, wait for the frontend /v1/health/ready endpoint to succeed, and confirm that /v1/models contains the expected model.

  7. Send representative chat, tool-calling, reasoning, and LoRA requests before directing production traffic to the deployment.

For a single-container Docker command, refer to Quickstart. For multi-pod Kubernetes, refer to Kubernetes Deployment.

Revert to Standard Mode#

Redeploy the saved standard image and configuration with all three NIM_DYNAMO_* selectors absent. Restore:

  • Standard health probes and port mappings.

  • nginx TLS, CORS, proxy, timeout, and header settings.

  • Backend metrics expectations on /v1/metrics.

  • The previous model profile and any explicit TMPDIR value.

  • Standard middleware, parser, and LoRA configuration.

Afterward, verify /v1/health/ready, /openapi.json, metrics, and representative inference requests. Treat this as a configuration reversal; validate application and external state separately for your deployment.