Is this page helpful?

Dynamo Performance Tuning#

Dynamo deployments provide routing and worker-engine controls for performance tuning. Start from the recipe published for your model and measure every change on the target hardware and request mix.

Important

The published recipe pins a validated image, model profile, deployment shape, and workload-specific engine settings. Changing them produces a configuration that requires separate validation.

Frontend Router Configuration#

The Frontend uses DYN_ROUTER_MODE, which defaults to kv. DYN_FRONTEND_EXTRA_ARGS forwards additional arguments to the installed Dynamo frontend. NIM does not validate the accepted frontend values or their semantics, so confirm them against the Dynamo runtime version in the recipe’s image.

Do not place frontend or --dyn-* arguments in NIM_PASSTHROUGH_ARGS. That variable is parsed as worker-side vLLM arguments.

Worker Engine Configuration#

Set worker-side vLLM tuning through NIM_PASSTHROUGH_ARGS on the Worker component. The published recipes use this variable for workload-specific settings such as expert parallelism, cache behavior, batching, compilation, speculative decoding, and KV-cache events.

Preserve profile-owned arguments and the NIM_PASSTHROUGH_ARGS value from the published recipe unless you intend to revalidate the model configuration. In a split deployment, do not set worker engine arguments on the Frontend.

For the complete NIM variable definitions, refer to Environment Variables.

Worker Scaling#

The Worker.replicas field in the published DGD recipe determines the number of worker pods. Each recipe also pins the GPU request for one worker. Capacity is therefore:

worker replicas x GPUs requested per worker

Before changing the replica count, confirm that the cluster has enough matching GPU capacity. Do not reduce the GPU count per worker or replace the selected model profile independently. Those values are coupled to the recipe’s tensor parallel configuration.

Measure the Effect#

The Dynamo Frontend serves Prometheus metrics at /v1/metrics. Each Worker serves vLLM metrics at /metrics on its system port, which is configured with NIM_HEALTH_PORT and defaults to 9090.

curl -s http://localhost:8000/v1/metrics \
  | grep dynamo_frontend_cached_tokens

Interpret metric names and values using the documentation for the Dynamo runtime installed in the recipe’s image. For Frontend and Worker scrape targets, refer to Logging and Observability.