Is this page helpful?

Dynamo Deployment#

Dynamo mode runs NIM LLM on NVIDIA Dynamo, a distributed inference runtime. A Dynamo deployment separates the HTTP frontend and router from one or more vLLM workers.

Image Requirements#

Use the NIM 3.0 vLLM image specified by the recipe published for your model. SGLang with Dynamo is not supported by the deployment paths documented here.

A worker can fall back to direct vLLM serving when the Dynamo package is absent. After deployment, verify that its logs contain Starting Dynamo worker: python -m dynamo.vllm.

Deployment Modes#

Set exactly one mode selector. For the complete variable definitions, refer to Environment Variables.

The following table summarizes the available deployment modes:

Dynamo deployment modes#

Mode

Selector

Processes

GPU Required

Single container

NIM_DYNAMO_SINGLE=1

One frontend and one worker

Yes

Frontend only

NIM_DYNAMO_FRONTEND=1

One frontend/router

No

Worker only

NIM_DYNAMO_WORKER=1

One worker registered with a separate frontend

Yes

Warning

The entrypoint does not reject multiple enabled selectors. It evaluates NIM_DYNAMO_SINGLE, then NIM_DYNAMO_WORKER, then NIM_DYNAMO_FRONTEND, and uses the first enabled selector. Remove unused selectors so the container does not start in an unintended mode.

Single-container mode uses file discovery and ZMQ without etcd or NATS. Kubernetes deployments use the Dynamo Operator.

Ports and Endpoints#

Dynamo mode bypasses nginx. The frontend owns DYN_HTTP_PORT, which defaults to NIM_SERVER_PORT and then 8000. Configure the worker system port with NIM_HEALTH_PORT, which defaults to 9090. The entrypoint derives DYN_SYSTEM_PORT from that value and overwrites any value set directly for DYN_SYSTEM_PORT.

For the complete endpoint contract, including NIM management-route availability, refer to API Reference. Use /v1/models to verify that a worker registered and its model is available. Workers expose /v1/health/ready on their system port.

The frontend and workers expose different Prometheus surfaces: the frontend serves Dynamo metrics at /v1/metrics, while each worker serves vLLM metrics at /metrics on its system port. Refer to Logging and Observability for scrape configuration.

Compatibility Boundaries#

Dynamo starts without nginx and the standard NIM ASGI middleware. As a result:

  • SageMaker compatibility routes /ping and /invocations are not installed.

  • NIM_SSL_*, NIM_CORS_*, NIM_PROXY_*, NIM_NGINX_*, NIM_CAPTURE_ENABLE, and NIM_CAPTURE_ARGS do not configure the Dynamo frontend. Apply the equivalent policy at an ingress or gateway.

  • NIM_LOG_LEVEL and NIM_JSONL_LOGGING configure the worker, not the Dynamo frontend.

  • Do not pass API-server-only options such as --enable-auto-tool-choice, --tool-call-parser, --tool-parser-plugin, or --chat-template to the Dynamo worker. Its strict argument parser rejects them. Configure Dynamo parser names with DYN_TOOL_CALL_PARSER and DYN_REASONING_PARSER instead. The engine-side --reasoning-parser and --reasoning-parser-plugin options remain valid worker arguments.

For the complete variable definitions, refer to Environment Variables.

Deployment Paths#

Single-Container Quickstart

Start one frontend and one worker with docker run.

how-to

Run NIM in Dynamo Mode
Kubernetes

Deploy a DynamoGraphDeployment with the Dynamo Operator.

how-to

Dynamo Graph Deployment on Kubernetes
Troubleshooting

Diagnose Dynamo-specific startup and registration failures.

reference

Troubleshooting Dynamo Mode
Performance Tuning

Review router, worker-scaling, and measurement controls.

reference

Dynamo Performance Tuning