Dynamo Profiler

The profiler is the engine behind Auto Deploy with DGDR. When you submit a DynamoGraphDeploymentRequest (DGDR), the profiler analyzes your model, hardware, and SLA targets, chooses a parallelization strategy for the prefill and decode engines, and generates the DynamoGraphDeployment (DGD) that serves traffic.

The DGDR walkthrough is enough to deploy a model. This page is for the next level: understanding what the profiler does with your request, choosing between the two search modes deliberately, and reading what the profiler decided. It assumes you have already authored a DGDR — it does not re-cover spec fields, autoApply, model caching, or monitoring, which the walkthrough and the DGDR Reference already document.

The profiling pipeline

The profiler runs as a Kubernetes Job while the DGDR sits in the Profiling phase. The stages below map to the sub-phases you can watch with kubectl get dgdr <name> -o jsonpath='{.status.profilingPhase}'.

agg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+----------+--------------+-------------+----------+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | replicas | gpus/replica | gpus/worker | parallel |
+------+--------------+---------------+--------+----------+--------------+-------------+----------+
|  1   |    410.22    |     108.48    | 251.10 |    8     |      4       | 4 (=4x1x1)  |  tp4pp1  |
|  2   |    361.33    |     107.43    | 224.48 |    4     |      8       | 8 (=8x1x1)  |  tp8pp1  |
+------+--------------+---------------+--------+----------+--------------+-------------+----------+
disagg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+----------+------------+-------------+------------+-------------+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | replicas | (p)workers | (p)parallel | (d)workers | (d)parallel |
+------+--------------+---------------+--------+----------+------------+-------------+------------+-------------+
|  1   |    684.79    |     100.31    | 295.71 |    4     |     2      |    tp2pp1   |     1      |    tp4pp1   |
|  2   |    684.79    |     100.16    | 295.71 |    2     |     4      |    tp2pp1   |     1      |    tp8pp1   |
+------+--------------+---------------+--------+----------+------------+-------------+------------+-------------+

Search Strategy

The searchStrategy field controls how the profiler scores candidates during the search stage — simulated versus measured. It trades profiling time against accuracy.

Rapid

Rapid is the default. It scores candidate configurations with AIC simulation instead of deploying engines, so it finishes in about 30 seconds and consumes no GPUs during profiling.

1
spec:
2
  searchStrategy: rapid

Reach for rapid when you are getting started, iterating on SLA targets, or running in CI. It supports all three backends and both aggregated and disaggregated topologies. The tradeoff is accuracy: results are estimated, and unusual configurations can carry error.

If AIC does not support your model, GPU SKU, and backend combination, the profiler falls back to a naive memory-fit calculation and logs AIC does not support this combo — falling back to naive config generation. The fallback may not be optimal and skips the ranked tables above, so confirm your SKU is in the AIC support matrix first.

Thorough

Thorough enumerates candidate parallelization mappings, deploys each as a real Kubernetes workload, and benchmarks it with AIPerf. It produces measured rather than estimated data, at the cost of 2–4 hours and real GPU capacity during profiling.

1
spec:
2
  searchStrategy: thorough
3
  backend: vllm      # a concrete backend is required

Use thorough when tuning for production, when your hardware is outside the AIC support matrix, or when you need measured numbers. It carries two hard constraints that the profiler rejects at validation:

• Disaggregated only — thorough does not evaluate aggregated topologies.
• backend: auto is rejected — name vllm, sglang, or trtllm.

It also consumes real GPUs during profiling, since it deploys and benchmarks an engine for each candidate. After picking, thorough mode runs in-depth profiling on the selected prefill and decode engines to produce the interpolation curves — sweeping input lengths for prefill, and KV-cache load and context length for decode.

How the profiler picks a configuration

You do not choose a picking mode. The profiler derives it from two things you already set — whether the Planner is enabled, and whether you gave it a target load — and optimizes accordingly.

The Planner takes precedence: if you set both a Planner and a target load, the profiler optimizes for autoscaling. To size for a fixed load instead, leave the Planner out and set workload.requestRate or workload.concurrency.

Model and parallelization support

The profiler supports dense and MoE models, with MoE coverage depending on the backend:

Which parallelization mappings it sweeps depends on the model architecture:

Constraints

Beyond the strategy-specific rules covered under Thorough, the profiler enforces these at validation, before any GPUs are used:

• sla.e2eLatency cannot be combined with an explicit sla.ttft or sla.itl — provide only one form. The request is rejected otherwise.
• An unachievable SLA is not fatal. The profiler logs a warning, relaxes the SLA to the best achievable value, and continues.
• A target load that exceeds the GPU budget is not fatal. The profiler logs a warning and returns its best effort within budget.
• A model that spans more GPUs than one node provides needs a gang scheduler (Grove or LWS); see Multinode Orchestration.

Simulate without GPUs

To validate a configuration or exercise Planner behavior at scale without real engines, enable the mocker backend. The profiler swaps the real-engine base for a mocker template that registers workers and replays modeled performance, so no GPUs are needed to run the deployment.

1
spec:
2
  model: Qwen/Qwen3-0.6B
3
  backend: vllm
4
  features:
5
    mocker:
6
      enabled: true

Mocker is a testing and experimentation path, not a serving deployment. It is independent of searchStrategy — enabling it does not change or override rapid versus thorough, and the profiler still runs the search strategy you set. The one interaction is with the Planner: when you enable mocker alongside the Planner, the Planner’s pre-deployment sweeping cannot be none, because the mocker needs the simulated performance data that sweeping produces. Without a Planner, mocker carries no such requirement. For how the simulated backend works and how it models performance, see Live Simulation with Mocker.

Accessing profiling artifacts

By default the profiler writes only its output — the generated DGD, and interpolation data when the Planner or mocker needs it — to ConfigMaps. For the full record of a run (performance plots, per-candidate configs, AIPerf results, raw .npz data, and logs), attach a PVC to the profiling Job through overrides.profilingJob:

1
spec:
2
  overrides:
3
    profilingJob:
4
      template:
5
        spec:
6
          containers: []    # required placeholder; leave empty to inherit defaults
7
          volumes:
8
            - name: profiling-output
9
              persistentVolumeClaim:
10
                claimName: dynamo-pvc

Copy the results off the PVC with a temporary access pod:

$
kubectl apply -f deploy/utils/manifests/pvc-access-pod.yaml -n <namespace>
$
kubectl wait --for=condition=Ready pod/pvc-access-pod -n <namespace> --timeout=60s
$
kubectl cp <namespace>/pvc-access-pod:/data ./profiling-results
$
kubectl delete pod pvc-access-pod -n <namespace>

The interpolation .npz files are the same data the Planner consumes for autoscaling decisions. For their array schema and how the Planner uses them, see the Planner Guide.

Related pages