Profiler Configuration Reference

Field reference for the Dynamo profiler CLI arguments — the SLA profiler entry point and the low-level per-endpoint profiler.

View as Markdown

The Dynamo profiler has two entry points, each with its own arguments:

  • python -m dynamo.profiler — the SLA profiler. Sweeps a model across tensor-parallel sizes and interpolates prefill/decode performance to recommend a deployment that meets your SLA. Takes a DynamoGraphDeploymentRequest (DGDR) spec plus a handful of operational flags.
  • python -m dynamo.profiler.profile_endpoint — the low-level per-endpoint profiler. Profiles a single already-running prefill or decode endpoint.

For the profiling workflow and DGDR spec structure, see the Profiler Guide. Field names, types, and defaults on this page come from __main__.py and profile_endpoint.py.

SLA profiler (dynamo.profiler)

--config
stringRequired

DynamoGraphDeploymentRequestSpec as an inline JSON string or a path to a JSON/YAML file. This defines the model, backend, and SLA targets to profile against. Startup fails if it is neither a valid file path nor valid JSON.

--output-dir
stringDefaults to profiling_results

Path to the output results directory. The profiler writes plots, the interpolation data, profile_sla.log, and a profiler_status.yaml status file here.

--deployment-timeout
integerDefaults to 3600

Maximum seconds to wait for each trial deployment to become ready before failing.

--prefill-interpolation-granularity
integerDefaults to 16

Number of input-sequence-length (ISL) samples used to interpolate prefill performance. Higher values give a finer prefill curve at the cost of more profiling runs.

--decode-interpolation-granularity
integerDefaults to 6

Number of samples used to interpolate decode performance across the KV-cache load range. Higher values give a finer decode curve at the cost of more profiling runs.

--dry-run
booleanDefaults to false

Skip deployments and benchmarking. Use this to validate the config and preview the planned sweep without provisioning GPUs.

Per-endpoint profiler (dynamo.profiler.profile_endpoint)

This entry point profiles a single running endpoint. It is normally invoked by the SLA profiler, but you can run it directly against an existing deployment.

--mode
stringRequired

Which phase to profile.

Allowed values: prefill decode
--model_name
stringRequired

Model name to send with requests. Must match the model the endpoint serves.

--url
stringRequired

Base URL of the endpoint to profile, for example http://localhost:8000.

--num_gpus
integerRequired

Number of GPUs backing the endpoint. Used to normalize the reported per-GPU performance.

--tokenizer_path
string

Path to the tokenizer. When empty, the tokenizer is resolved from --model_name.

--max_kv_tokens
integerDefaults to 0

Maximum KV-cache tokens for the endpoint. Used only in decode mode to bound the KV-load sweep. 0 lets the profiler detect the limit.

--work_dir
stringDefaults to endpoint_profiling_results/

Directory to save this endpoint’s profiling results.

--max_context_length
integerDefaults to 16384

Maximum context length to probe when generating request payloads.

--interpolation_granularity
integerDefaults to 8

Number of samples used to interpolate this endpoint’s results.

--attention_dp_size
integerDefaults to 1

Attention data-parallel size of the endpoint. Set this for MoE models that run attention with data parallelism.

Profiler Guide

Profiling workflow, DGDR spec structure, and how the SLA recommendation is produced.

Profiler Examples

End-to-end profiling runs and how to read the output.

Planner Configuration

Field reference for the Planner that consumes profiler output.

Runtime Configuration

Shared Dynamo runtime flags parsed by every component.