DynamoGraphDeploymentRequest (DGDR) Reference
Field reference and lifecycle for the DynamoGraphDeploymentRequest custom resource — Dynamo’s deploy-by-intent generator.
A DynamoGraphDeploymentRequest (DGDR) is Dynamo’s deploy-by-intent generator for DynamoGraphDeployment (DGD) resources. You describe what you want to run and your performance targets; the profiler determines a configuration and produces the DGD that serves traffic.
For the task-oriented walkthrough — authoring a request step by step — see Auto Deploy with DGDR. For ready-to-copy manifests, see DGDR Templates.
This page documents the nvidia.com/v1beta1 API — the storage version.
Conversion views
When the nvidia.com/v1beta2 API is served, the API server represents a native
v1beta2 request through v1alpha1 or v1beta1 by populating spec.v1beta2.
Conversely, it represents a native v1beta1 request through v1beta2 by
populating spec.v1beta1. These fields are read-only conversion views. Create
and modify a request through the API version that owns its native fields; do not
set or edit the conversion field directly. An unchanged conversion view can be
written back without changing the stored request.
v1beta2 hardware SKU
spec.hardware.gpu.sku in nvidia.com/v1beta2 is an open string containing an
AISimulate system-catalog identifier. The AISimulate release bundled with this
Dynamo release includes a100_pcie, a100_sxm, a30, b200_sxm, b300_sxm,
b60, gb200, gb300, h100_pcie, h100_sxm, h200_sxm, l4, l40s, and
rtx_pro_6000_server.
Admission passes an explicit SKU through unchanged. It warns when the bundled catalog does not contain that identifier, but does not reject it: an external AISimulate systems catalog may provide additional identifiers.
Omit hardware to let admission discover and persist sku, budget, and
numGpusPerNode. An exactly empty hardware: {} object opts out of all operator
cluster and DCGM discovery and typed hardware defaults. Any nonempty object,
including hardware: {gpu: {}}, is partial hardware intent: admission discovers
and persists each missing field while preserving every explicit value. The Run
controller never performs hardware discovery.
Set all three values to bypass discovery with fully explicit hardware. An unknown
SKU must provide both budget and numGpusPerNode, because admission cannot map
that identifier to cluster capacity or topology.
To leave the complete hardware input to AISimulate and inputOverrides, use the
empty-object opt-out and supply both the upstream hardware selector and candidate
GPU budget yourself. Request and Run admission validate the merged input against
the bundled AISimulate input schema. Run specs remain immutable; unchanged specs
remain metadata-updatable after a bundled schema change.
v1beta2 DGD Overrides
Experimental; for development only. Enable dynamo-operator.dgdrV1Beta2.enabled
in the platform Helm chart to use spec.overrides.dgd. It accepts one partial
nvidia.com/v1beta1 DGD, applied to every selected Candidate across backends and topologies.
Other DGD API versions are not accepted here.
Overrides follow the DGD schema’s Server-Side Apply merge rules.
For example, components and named containers merge by name; container command and
args are atomic lists and replace the entire generated list. These are examples:
the schema’s map, set, and atomic markers determine how all fields merge. Omitted
fields retain their generated values.
The existing DGD $patch: {args: append} customization is also supported on a named
container. It appends the supplied arguments to an explicit generated args list and
removes the directive before validation. Both the target container and its args
must already exist. This is the only supported $patch operation.
The override is frozen into the immutable Run spec and applied atomically: if a target
component is absent or any part cannot be applied, the Candidate retains the unchanged
generated DGD spec and reports Valid=False. A new named container can be added to an
existing component through the normal merge; $patch: {args: append} requires an
existing container instead.
For example, a decode override can fail for an aggregated Candidate whose topology
contains only worker. No subset of a failed override is applied.
Choose target names that exist in the results you want to keep. Restricting backends
can avoid engine-specific mismatches, but does not guarantee a particular topology.
A shared decode override can invalidate every aggregated result, including the entire
result set. Invalid Candidates still count toward recommendation.maxCandidates and
do not trigger replacement searches.
After a successful merge, a copy of the DGD undergoes strict server-side dry-run creation,
including DGD defaults, schema, CEL, and validating webhooks. The dry-run response is
discarded; the stored payload remains the one from before the check.
Unknown fields and incorrect field types remain visible on invalid Candidates;
Valid=False records the failure. Infrastructure failures are retried instead of being
recorded as an invalid deployment.
Retryable Candidate preparation errors are visible on the Run as Published=False,
reason PublicationBlocked, with the error in the condition’s message. This includes
HTTP 403 from a DGD dry-run: RBAC changes propagate asynchronously, and a permission or
policy configuration can be repaired. The publisher keeps retrying and preserves the
already published Candidate references and progress. Once publication succeeds, it sets
Published=True with reason PublicationSucceeded. Published is owned by the publisher;
the Run controller independently owns Completed. Changing the request’s spec stops the
old Run and its publisher before starting the replacement Run.
For example, the Run can report this status while retrying a denied DGD dry-run:
Candidate.spec contains exactly the DGD spec, including backendFramework.
DGD labels and annotations are copied to Candidate metadata. If an override fails to
apply, only the original generated labels and annotations are copied; metadata from
the failed override is discarded. If the merge succeeds but dry-run validation fails,
the override’s labels and annotations remain on the invalid Candidate.
An override that makes the complete Candidate metadata invalid is rejected in full.
The total annotation-size limit includes generated annotations and the reserved
annotations added to the Candidate.
Candidate identity, including name, namespace, and owner references, cannot be overridden. The following metadata is also reserved; attempts to override it fail atomically:
- Labels
nvidia.com/dgdr-run-uidandnvidia.com/dgdr-candidate-id. - Annotation
dgdr.nvidia.com/additional-resources. - Annotations with the
nvidia.com/dgd-conversion-bookkeeping prefix.
Evaluation parameters, metrics, and the final Valid condition belong to Candidate status.
The condition’s statuses and reasons are documented on
DynamoGraphDeploymentCandidateStatus.conditions.
A Candidate can retain an invalid DGD, including unknown fields or incorrect field types.
Check for status.conditions with type: Valid and status: "True" before using its
spec as a DGD. A missing condition does not mean validation succeeded.
The Valid condition reports whether the Candidate’s DGD passed override application
and strict API admission. Its absence does not indicate success.
Valid=True means the API object is structurally valid and passed strict DGD admission
at checking time. It does not indicate a running or Ready deployment, or prove image
availability, GPU capacity, or successful workload scheduling.
Overrides may change model, backend, GPU allocation, parallelism, images, or arguments. The original simulation parameters and metrics are not a performance prediction for the overridden DGD.
List only Candidates with Valid=True in the current namespace (requires jq):
Invalid Candidates count toward recommendation.maxCandidates and retain their original
search order in Run references. They do not trigger replacement searches. A successful
Run requires a successful search and at least one Valid=True Candidate; when all
selected Candidates are invalid, the Run fails and keeps them for diagnosis.
Spec reference
Only model is unconditionally required by the schema. Admission rejects a new DGDR whose supplied spec.image has no parseable semantic-version tag unless spec.runtimeVersionOverride is set. When spec.image is omitted, the operator supplies a versioned default as described below.
Model to deploy — a HuggingFace ID (for example Qwen/Qwen3-0.6B, meta-llama/Llama-3-70b) or a private model name. Must be non-empty.
Container image for the profiling job (the planner image), for example nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.4.0. For Dynamo < 1.1.0, use dynamo-frontend.
Required when spec.image is set and has no parseable semantic-version tag, such as latest, a custom build tag, or a digest-only reference. Also supplies the default Dynamo runtime version for generated DGD components that do not set their own override. An explicit component value in overrides.dgd takes precedence. Set this when an effective generated component image has no parseable semantic-version tag or its tag does not identify the intended Dynamo runtime.
Use canonical MAJOR.MINOR.PATCH, for example "1.4.0", with each part between 0 and 9999 and no leading zeros, v prefix, prerelease suffix, or build metadata. This field does not change any image. See Profiler image version compatibility.
Inference backend used for profiling and deployment.
Allowed values: auto vllm sglang trtllmProfiling depth. rapid performs a fast, AIC-backed sweep (~30s); thorough explores more configurations on real GPUs (2–4h).
Automatically create a DGD after profiling completes. When false, the generated spec is stored in status for manual review instead of being applied.
Profiler image version compatibility
When spec.image is omitted, the operator defaults it on creation to
nvcr.io/nvidia/ai-dynamo/dynamo-planner:<operatorVersion>. The operator
requires operatorVersion to be valid semantic versioning, so the default
image has a parseable version tag.
The DGDR-level spec.runtimeVersionOverride supplies a default for generated
DGD components. After the profiler materializes the DGD and applies
spec.overrides.dgd, the operator copies the DGDR value only to components that
do not already have an explicit override. This behavior applies across profiler
versions. Profilers through Dynamo 1.3.0 may discard the field while parsing the
DGDR, but the operator reapplies the default from the stored DGDR.
Set spec.runtimeVersionOverride when an effective generated component runtime image
uses a non-semantic-version tag or digest, or when its tag does not identify the
intended Dynamo runtime version. The DGDR value also covers images replaced
through spec.overrides.dgd when that component does not set its own
runtimeVersionOverride. An explicit component-level value takes precedence.
While a DGDR has autoApply: false, you may set or change
runtimeVersionOverride during Profiling or Ready. After it reaches
Ready, you may enable autoApply, either separately or while changing the
override. The operator applies the current value while creating the new DGD.
These updates do not modify status.profilingResults.selectedConfig or the
nvidia.com/generated-dgd-spec annotation. Other spec updates remain
forbidden after profiling starts.
When the DGDR-level field is unset, each effective generated DGD component
follows the component runtime version rules independently. An overridden runtime image without a
parseable semantic-version tag requires runtimeVersionOverride on that
component.
Profilers through Dynamo 1.3.0 discard the embedded DGD override’s apiVersion
and kind, then directly merge the remaining dictionary into a generated
v1alpha1 DGD. Raw storage preserves the override fields but does not convert
between DGD schemas. Use a v1alpha1-shaped override under spec.services; a
v1beta1 spec.components override is not translated into the generated
service.
Expected workload characteristics for SLA-based profiling.
Input Sequence Length — expected average input tokens.
Output Sequence Length — expected average output tokens.
Optional target concurrency level. Mutually exclusive with requestRate. When neither field is set and the Planner is disabled, the profiler uses its default maximum-throughput selection.
Optional target request rate in requests per second. Mutually exclusive with concurrency. When neither field is set and the Planner is disabled, the profiler uses its default maximum-throughput selection.
Service-level agreement targets that drive profiling optimization.
Time To First Token target, in milliseconds.
Inter-Token Latency target, in milliseconds.
End-to-end request latency target, in milliseconds. An alternative to specifying ttft + itl; cannot be combined with them.
Optimization target for SLA profiling.
Allowed values: latency throughputGPU hardware for profiling and deployment.
GPU type to target (see SKU format). When omitted, auto-detected as the GPU type of the node with the most GPUs (ties broken by highest VRAM per GPU). In mixed-GPU clusters, set this to choose the GPU type; discovery and totalGpus are then restricted to matching nodes.
VRAM per GPU in MiB. Auto-detected from cluster GPU nodes when omitted. An explicit value within 10% of the auto-selected shape’s measured VRAM permits automatic GPU-budget discovery; for example, vramMb: 81920 matches a measured 81079 MiB. This tolerance applies only to the requested value; aggregating discovered nodes still requires equal measured VRAM. If this value differs by more than 10% and totalGpus is omitted, automatic GPU-budget discovery fails and the DGDR moves to phase: Failed before a profiling Job is created; set totalGpus explicitly to use that override.
GPU budget for profiling and deployment; the profiler uses it to determine parallelism and replica count. When omitted, computed by summing GPUs on nodes matching the auto-selected shape: GPU type, GPUs per node, and VRAM per GPU. The result is temporarily capped at 32 to limit the profiler search space. This reflects discovered GPUs, not currently free capacity. For example, same-type nodes with 8 and 4 GPUs represent 12 GPUs of inventory, while the auto-selected 8-GPU shape yields an automatic budget of 8. Set explicitly to override the derived budget, use a different numGpusPerNode, or use a vramMb value more than 10% from the auto-selected shape’s measured VRAM; these shape overrides require an explicit totalGpus.
GPUs per node. When omitted, auto-detected as the highest GPU count on any discovered node of the selected GPU type. If this value differs from the auto-selected shape and totalGpus is omitted, automatic GPU-budget discovery fails and the DGDR moves to phase: Failed before a profiling Job is created; set totalGpus explicitly to use that override.
Primary GPU-to-GPU interconnect within a node, for example nvlink or pcie. Capability metadata for profiling and planning only — it does not enable any interconnect. Best-effort discovery distinguishes nvlink from pcie when omitted. An over-optimistic value can lead the planner to overestimate intra-node bandwidth and pick overly aggressive parallelism.
Whether the cluster has RDMA-capable networking for Dynamo data movement. Capability metadata only — it does not install or configure RDMA. Best-effort discovery via node labels when omitted. A false positive can cause transports to assume RDMA is usable and fail to connect; a false negative falls back to non-RDMA transports.
Optional PVC configuration for pre-downloaded model weights. When set, weights are loaded from the PVC instead of downloading from HuggingFace.
Name of a ReadWriteMany PVC containing cached model weights. The PVC must exist in the same namespace as the DGDR.
Path to the model checkpoint directory within the PVC, for example deepseek-r1 or models/Llama-3.1-405B-FP8.
Mount path for the PVC inside the container.
Optional Dynamo platform features in the generated deployment.
A PlannerConfig object passed to the Planner service. The presence of this field (non-null) enables the planner in the generated DGD. DGDR passes it through without field-level validation; the Planner service validates it at startup. See Planner for the full field list.
Configures KV-cache-aware routing for the generated deployment. When enabled, DGDR sets
DYN_ROUTER_MODE=kv on the generated Frontend.
Enables KV-cache-aware routing.
Configures the simulated (mocker) backend for testing without GPUs.
Deploy mocker workers instead of real inference workers.
spec.overrides.dgd takes precedence over features.kvRouter. A DGD override can replace
DYN_ROUTER_MODE on the Frontend container or provide an explicit --router-mode argument. The
command-line flag takes precedence over the environment variable when both are present. See
Generated DGD overrides.
Overrides
Customizes the profiling job and the generated DGD.
Overrides merged into the controller-generated profiling Job spec (for example, tolerations or node selectors). Only customizes the profiling Job — not the deployed worker pods.
batch/v1.JobSpecA partial, versioned DGD merged into the generated deployment. Use nvidia.com/v1beta1 for new overrides; nvidia.com/v1alpha1 remains supported for compatibility. See Generated DGD overrides.
Planner
spec.features.planner is a PlannerConfig object, passed through to the Planner service without field-level validation; the Planner validates it at startup. Setting it (non-null) enables the planner in the generated DGD. The same object is documented standalone at Planner Configuration; its fields are reproduced here. Nested list-and-object types are expanded under Additional Types. All fields are optional.
Core settings
Planner operating mode, matching the deployment topology.
Allowed values: disagg prefill decode aggInference backend the Planner scales.
Allowed values: vllm sglang trtllm mockerRuntime environment that determines how the Planner applies scaling actions.
Allowed values: kubernetes virtual global-plannerDynamo namespace of the deployment the Planner manages. Defaults from DYN_NAMESPACE, which the operator injects as {k8s_namespace}-{dgd_name}. Excluded from serialized output.
Optional model name override. Auto-detected from the deployment when unset.
Namespace where the GlobalPlanner runs. Required when environment is global-planner.
Optional directory for Planner log output.
Optimization target
Scaling strategy. throughput and latency use static thresholds on queue depth and KV cache utilization — no SLA targets or profiling required. load uses user-defined prefill queue-token and decode KV-utilization thresholds. sla uses the AISimulate RustForwardPassPerfModel.best_available(config) performance model to target specific ttft_ms / itl_ms values. Under any target other than sla, the Planner forces load scaling on, throughput scaling off, and ignores ttft_ms / itl_ms.
Enable predictive, traffic-based scaling. Only honored when optimization_target is sla.
Enable reactive, load-based scaling. Only honored when optimization_target is sla. At least one scaling mode must be enabled.
Time To First Token SLA target, in milliseconds. Also accepts the alias ttft. Must be greater than 0. Used only under optimization_target: sla.
Inter-Token Latency SLA target, in milliseconds. Also accepts the alias itl. Used only under optimization_target: sla.
Performance model and pre-deployment sweeping
How to generate optional bootstrap performance data. none skips bootstrap data; rapid uses AIConfigurator to simulate engine performance (~30s); thorough measures on real GPUs (several hours).
Directory holding profiler-generated performance data (npz or JSON), used to bootstrap or tune the performance model when the get_perf_metrics endpoint is unavailable.
Role-indexed AISimulate forward-pass configurations. Constructs each model through
aisimulate_core.sdk.RustForwardPassPerfModel.best_available(config) without an
interpolation sweep. Configure model identity and estimator controls through
ais_perf_model.roles; the retired aic_perf_model and hf_id / pick shape
are rejected.
Maps deployment roles to complete upstream ForwardPassPerfModelConfig objects.
Supply prefill and decode for disagg, the matching single role for prefill
or decode, and aggregated for agg. Each key must match worker_type.
Model identifier, AISimulate system identifier, and inference backend
(vllm, sglang, or trtllm). Optional backend_version, tp, pp,
attention_dp, MoE axes, quantization, KV block size, and speculation fields
use the upstream schema. Published worker capabilities must agree with the
configured identity.
auto searches op_level, fpm_interpolation, then fpm_regression.
Explicit modes select the named estimator.
deny rejects failure of an explicitly selected mode. allow tries the other modes in
op_level, fpm_interpolation, fpm_regression priority order; regression permits a regression fallback.
Auto selection searches the full priority order even with deny.
Ordered performance-data roots. The SDK resolves its packaged root when unset.
database_mode and transfer_policy are also passed through unchanged.
Complete upstream estimator controls, including independent regression and
correction sampling, fit settings, feature weights, and correction bounds.
Explicit settings override Planner sampling defaults. A legacy total bucket
count of 16 maps to bins_per_axis: [4, 4]. Unknown fields are rejected by
AISimulate. See the AISimulate core API.
AIConfigurator interpolation spec. Populated by the profiler in rapid mode and written onto the Planner ConfigMap; you do not normally set this by hand. When present, the Planner runs the AIC sweep in-process at bootstrap to seed the performance model. See AICInterpolationSpec.
GPU budget
Maximum total GPUs the Planner may allocate across worker types.
Per-DGD GPU floor enforced by the local Planner. -1 disables it. When set with max_gpu_budget such that min == max, the Planner pins the per-DGD total and only redistributes replicas between prefill and decode.
Minimum engine endpoints for aggregated deployments. In disaggregated deployments, this value applies to both prefill and decode unless a role-specific value is set. In prefill-only or decode-only deployments, it supplies the active role when the corresponding role-specific value is unset. Must be nonnegative; use 0 for scale-to-zero compatibility.
Minimum prefill endpoints in disaggregated and prefill-only modes. When set, replaces the prefill value supplied by min_endpoint. Must be at least 1.
Minimum decode endpoints in disaggregated and decode-only modes. When set, replaces the decode value supplied by min_endpoint. Aggregated deployments use min_endpoint. Must be at least 1.
GPUs per decode engine replica. Auto-detected from the deployment when unset.
GPUs per prefill engine replica. Auto-detected from the deployment when unset.
Throughput-based scaling
Seconds between throughput-based scaling decisions. Also accepts the alias throughput_adjustment_interval.
Maximum replica-count change per component from one throughput observation. The Planner caps and stores the throughput lower bound, so load-scaling ticks cannot repeatedly apply the same uncapped prediction. GPU and power budgets can reduce the change further. At the nominal minimum GPU budget, scale-down-only requests hold. At a full GPU budget, two scale-up requests hold; an opposing prefill/decode proposal applies only the largest atomic pair that fits the budget. Endpoint and out-of-band budget recovery take precedence over this limit. Must be greater than 0.
Prometheus traffic source. frontend reads dynamo_frontend_* metrics from the public Frontend; router reads dynamo_component_router_* from a LocalRouter (use for a pool-local Planner in GlobalPlanner deployments).
Load-based scaling
Seconds between FPM tuning updates and load-based scaling decisions. Even when only throughput scaling is enabled, live FPM observations feed the performance model at this interval. Also accepts the alias load_adjustment_interval. Must be greater than 0 and, when both scaling modes are on, shorter than throughput_adjustment_interval_seconds.
Maximum retained ForwardPassMetrics observations for online tuning and regression fallback.
Number of buckets for observation retirement. Must be a perfect square.
Scale-down sensitivity from 0 to 100 for SLA-mode load scaling (100 = most aggressive). It sets how much
latency headroom the performance model must predict before load-based scaling removes a replica; 0 blocks
these model-based scale-downs. It does not apply when every fresh FPM observation for a pool is idle: the
Planner then removes one replica per decision, down to the larger of 1 and the endpoint floor. Set
min_endpoint (or the per-component floors) to keep idle capacity. Ignored unless optimization_target
is sla.
Minimum observations before load-based scaling decisions begin (cold-start threshold).
Prefill queue-token count that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes prefill. Must be greater than 0 and greater than prefill_scale_down_queue_tokens.
Prefill queue-token count that allows scale-down. Required alongside prefill_scale_up_queue_tokens.
Decode KV-utilization percentage (0–100) that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes decode. Must be greater than decode_scale_down_kv_rate.
Decode KV-utilization percentage (0–100) that allows scale-down. Required alongside decode_scale_up_kv_rate.
Manual fallback speculative-decoding depth. A worker’s published runtime_config.runtime_data.spec_decode.nextn takes precedence when present. Must be 0 or greater.
Load prediction
Prediction method for request count, ISL, and OSL.
Allowed values: constant arima kalman prophetApply a log1p transform to predicted request count, ISL, and OSL.
Window size (seconds) for the Prophet predictor.
Path to a warmup trace file for bootstrapping predictions.
Kalman process noise for the level component.
Kalman process noise for the trend component.
Kalman measurement noise.
Minimum data points before Kalman predictions activate.
Prometheus metrics
These fields default from environment variables and are excluded from serialized output.
Prometheus endpoint the Planner queries for traffic metrics. Defaults to PROMETHEUS_ENDPOINT, else the in-cluster kube-prometheus service URL.
Optional bearer token sent as Authorization: Bearer <token> on every PromQL request. Read once at startup.
Path to a file containing a bearer token, re-read before every request so rotated tokens are picked up without a restart.
Verify the upstream Prometheus TLS certificate. Defaults from PROMETHEUS_SSL_VERIFY (1/true/yes enable it). Pair with a CA bundle for a private CA.
Fixed key/value pairs appended as URL query parameters on every PromQL request. Set PROMETHEUS_EXTRA_QUERY_PARAMS as a URL query string, for example namespace=my-ns&tenant=foo.
Path to a CA bundle for verifying the upstream Prometheus TLS certificate. When set, the bundle is used for verification regardless of metric_pulling_prometheus_ssl_verify. Must point to an existing file.
Connection and read inactivity timeout in seconds for each Prometheus API request. Defaults from DYN_PLANNER_PROMETHEUS_REQUEST_TIMEOUT_SECONDS and must be greater than 0. This is not a total wall-clock deadline: a response that continues delivering data can run longer. The Planner does not retry failed requests within the same collection cycle.
Port on which the Planner exposes its own dynamo_planner_* metrics. Defaults from PLANNER_PROMETHEUS_PORT. 0 disables.
Advisory and diagnostics
Suggestion-only mode. The Planner computes, logs, exports, and reports recommended replica counts without executing scaling actions or changing the deployment. Use it to evaluate a new configuration or validate SLA targets against production traffic. A planner-in-the-loop replay of the same configuration also keeps the simulated deployment fixed.
Generate an HTML diagnostics report every N hours (simulated time). Set to null to disable. Must be a positive finite number or null.
Directory for HTML diagnostics reports.
Fixed report filename written under report_output_dir. When unset, a timestamped name is used.
Write a compressed JSONL diagnostics log (.log.jsonl.gz) next to each HTML report.
Port for the live diagnostics dashboard HTTP server. 0 disables. When enabled, visit http://host:port/ for a real-time Plotly report.
Port for the loopback-only runtime minimum-endpoint API. 0 disables the API. The operator does not expose this port through a Kubernetes Service.
Scheduling and plugin pipeline
Plugin-pipeline scheduling. Rejects unknown fields.
Base pipeline cadence. The pipeline wakes once per interval; each plugin’s execution_interval_seconds decides whether it fires. When unset, computed as the gcd of load_adjustment_interval_seconds and, when throughput scaling is enabled, throughput_adjustment_interval_seconds. Must be greater than 0 and evenly divide both adjustment intervals.
Outer deadline wrapping the full 4-stage pipeline. Exceeding it aborts the tick; the next tick runs from a clean state. Must be greater than 0.
Static external-plugin registrations applied at startup. Per-entry failures are logged but do not crash the Planner. See ExternalPluginEntry.
gRPC registration gateway for self-registering plugins. Disabled by default.
Open the gRPC gateway at listen. Not needed for static external_plugins.
Bind address. A unix: socket path for in-Pod registration, or host:port for TCP.
Permit binding a plaintext (no-TLS) gateway on a TCP listen. Fails closed by default because the gateway receives plugins’ shared-secret tokens. unix: listens are always allowed.
Plugin registry configuration — auth, transport, heartbeat, and in-process plugins. Rejects unknown fields.
Registry authentication. In the Planner process, an empty trusted_sources activates a legacy compatibility fallback to unauthenticated access and logs a warning. This fallback is DEV ONLY; do not rely on it as an intentional configuration. Select allow_unauthenticated explicitly for development. Production deployments must configure static_secret.
Accepted auth sources. Use ['allow_unauthenticated'] for development, or ['static_secret'] with static_secrets for production.
Map of secret_value to subject_label. Populate from a mounted Kubernetes Secret rather than hard-coding in the ConfigMap.
Outbound gRPC transport settings for calling plugins.
Allow plaintext grpc:// plugin endpoints. Required to use any grpc:// endpoint; mTLS support is not yet shipped.
Per-RPC timeout applied to every plugin call. Must be greater than 0.
gRPC keepalive time, in milliseconds.
Maximum gRPC message size (10 MB).
Minimum accepted plugin protocol version.
Maximum accepted plugin protocol version.
Reserved liveness setting. Heartbeats update registry timestamps, but no monitor currently applies this timeout or evicts plugins automatically.
Reserved liveness setting. Automatic missed-heartbeat eviction is not currently wired.
In-process plugins loaded from Python modules at startup. See InProcessPluginSpec.
Admin (ListPlugins) RBAC config. Parsed but currently inert — the gateway default-denies ListPlugins regardless of mode.
spec.overrides.dgd is not required to enable the Planner. Use spec.features.planner for Planner enablement and configuration; use spec.overrides.dgd to customize profiler-generated benchmark deployments and the final DGD. For the deployment workflow, scaling-mode concepts, and cross-field validation rules, see the Planner Configuration reference and Planner Guide.
Generated DGD overrides
spec.overrides.dgd is a partial, versioned DGD. It does not define or extend
the profiler’s candidate topology. Thorough profiling merges the override into
each generated benchmark candidate before measurement and into the
interpolation deployment. Every search strategy merges it into the final
generated DGD. Use apiVersion: nvidia.com/v1beta1 and
kind: DynamoGraphDeployment for new overrides. Components, pod-template
containers, and container environment variables merge by name; atomic lists
such as graph-level spec.env and container args replace their generated
values.
To permit custom model repository code without depending on the selected topology, set
spec.overrides.trustRemoteCode: true. The profiler applies --trust-remote-code to every generated
vLLM or SGLang worker after the component topology is known. Enable this setting only for model
repositories you trust.
Generated component names
An entry in spec.overrides.dgd.spec.components must use the exact, case-sensitive
component name from the generated DGD. These are DGD component names, not Kubernetes
Service resource names. The current profiler templates generate these components:
Setting spec.features.mocker.enabled: true replaces the inference workers with decode and
prefill mocker components. Setting spec.features.planner adds a Planner component.
Add overrides before profiling starts. The DGDR spec becomes immutable during profiling, so you
cannot inspect selectedConfig and then add overrides to the same resource. To inspect the names
generated by a specific Dynamo release, follow the
discovery workflow,
then recreate the DGDR with the required overrides.
For example, set an environment variable on one generated component:
Overrides can modify only components present in the generated DGD. An unresolved spec.components or
spec.services name causes DGD materialization to fail. A recognized deprecated name may be
translated to the corresponding component in the selected topology with a warning. An override
cannot add deployment topology. Labels and annotations merge, metadata.name selects the final DGD
name, and other identity metadata is ignored. status, null values for typed fields, and field
deletion are not supported.
To add flags to a container already present in the generated DGD, set $patch.args to append on
that named container:
The only supported modifier is args: append, and its args list must contain at least one
non-empty string. The target container and its args list must already be present in the generated
blueprint. The profiler blueprints define the complete default CLI for Frontend/main, so the
example preserves python3 -m dynamo.frontend and appends the KV router flags. The override helper
combines the argument lists and removes $patch before returning the final DGD. Without $patch,
specifying args retains the standard replacement behavior and requires the complete desired list.
Existing nvidia.com/v1alpha1 overrides remain supported. They use the older spec.services
shape, merge services by name, and append extraPodSpec.mainContainer.args to generated worker
arguments. In v1beta1, container args is atomic and replaces the generated list. Use
v1beta1 for new overrides and provide the complete desired argument list when overriding
args.
overrides.profilingJob customizes only the profiling Job. Use overrides.dgd for settings that
must appear on deployed components.
For worked examples, see DGDR Templates. For router configuration and EPP topologies, see the Router Guide and Gateway API Inference Extension.
SKU format
hardware.gpuSku uses lowercase underscore format (h100_sxm, not H100-SXM5-80GB). When
gpuSku is omitted, the operator normalizes the discovered GPU product name (for example,
NVIDIA B300 SXM6 AC) to the matching value (b300_sxm). The observed Vera Rubin product name
NVIDIA VR NVL72 ES and its GPU Feature Discovery form NVIDIA-VR-NVL72-ES map to the canonical
AISimulate identifier vr_nvl72. GB200 discovery uses the canonical AISimulate identifier gb200:
gb200_sxm is deprecated because GB200 systems use NVL rather than SXM. Existing manifests remain
admitted with their existing behavior, but admission emits a warning. Update manifests to use
gpuSku: gb200; newly discovered GB200 hardware is stored as gb200.
Not all SKUs are supported by the AIC profiler for rapid mode.
PCIe variants not yet supported by the profiler. The CRD admits PCIe SKUs (h100_pcie, a100_pcie, v100_pcie), but the profiler does not currently ship training data for them. You can submit a DGDR with a PCIe value; the operator accepts it but profiler-assisted sizing falls back to defaults. Profiler support for PCIe SKUs is tracked as an engineering follow-up.
Status
The operator maintains observed state under status. See Lifecycle for how phase and profilingPhase progress, and for the full condition-type list.
High-level lifecycle phase of the deployment request.
Allowed values: Pending Profiling Ready Deploying Deployed FailedCurrent sub-phase of the profiling pipeline. Only meaningful while phase is Profiling; cleared when profiling completes or fails.
Name of the generated or created DynamoGraphDeployment.
Name of the Kubernetes Job running the profiler.
Latest observed conditions. Standard types include Succeeded, Validation, Profiling, SpecGenerated, and DeploymentReady — see Conditions.
Output of the profiling process, including the selected deployment configuration.
Deprecated compatibility field retained for status objects written by older releases. The current controller does not populate it.
Recommended configuration chosen by the profiler based on the SLA targets. This is the configuration used for deployment when autoApply is true.
State of the deployed DynamoGraphDeployment. Populated once a DGD has been created (via autoApply or manually).
Desired number of replicas.
Number of replicas that are available and ready.
Most recent metadata.generation observed by the controller.
Additional Types
Nested struct types referenced by the fields above, broken out here to keep the field lists shallow. Types prefixed batch/v1., metav1., or runtime. are standard Kubernetes types and link to their Go package documentation instead of being expanded.
ParetoConfig
Deprecated compatibility type for status.profilingResults.pareto. Current
profiling runs do not populate this type.
Full deployment configuration for this Pareto point.
runtime.RawExtensionPickedParallelConfig
A parallelism pick emitted by AIConfigurator. Referenced by AICInterpolationSpec.
Tensor-parallel size.
Pipeline-parallel size.
Data-parallel (attention-DP) size.
Mixture-of-Experts tensor-parallel size.
Mixture-of-Experts expert-parallel size.
AICInterpolationSpec
Everything the Planner needs to reproduce a rapid-mode AIC sweep. Written by the profiler onto the Planner ConfigMap; not normally hand-authored. Referenced by features.planner.aic_interpolation.
HuggingFace model id.
AIC system identifier.
Input sequence length for the sweep. Must be greater than 0.
Output sequence length for the sweep. Must be greater than 0.
Maximum context length swept. Must be greater than 0.
Number of prefill interpolation points. Must be greater than 0.
Number of decode interpolation points. Must be greater than 0.
Prefill parallelism pick. See PickedParallelConfig.
Decode parallelism pick. See PickedParallelConfig.
ExternalPluginEntry
One static external-plugin registration under features.planner.scheduling.external_plugins. Rejects unknown fields.
Unique identifier. Must not collide with a builtin plugin id.
Pipeline stage the plugin participates in.
Allowed values: predict propose reconcile constrainStage priority. Smaller number is more authoritative.
Wire endpoint. Must start with grpc://host:port; inproc:// is rejected here.
Bearer token validated by the registry. Populate from a mounted Secret.
Plugin protocol version. Must fall within the registry’s supported range.
Plugin’s own version string, surfaced in ListPlugins.
0.0 runs every tick; a positive value throttles to every N seconds. Must be 0 or greater.
Behavior when throttled. HOLD_LAST reuses the cached result; ACCEPT_WHEN_IDLE treats the plugin as no-opinion when not due. Accepts the name (case-insensitive) or its integer value.
Capability list consumed by the type-aware merge.
Dot-paths into the pipeline context (for example predictions, observations.traffic) that must be set for the plugin to fire on a tick. Empty means no gating.
Aggregation window for windowed observation types in needs. 0.0 uses scale_interval freshness; a positive value aggregates over the last N seconds and must be an integer multiple of scale_interval_seconds.
InProcessPluginSpec
One in-process plugin under features.planner.plugin_registration.in_process_plugins. Rejects unknown fields.
Python module path containing the plugin class.
Plugin class name within module.
Unique plugin identifier.
Stage priority. Smaller number is more authoritative.
0.0 runs every tick; a positive value throttles to every N seconds.
Behavior when throttled.
Allowed values: ACCEPT_WHEN_IDLE HOLD_LASTCapability list consumed by the type-aware merge.
Dot-paths into the pipeline context that must be set for the plugin to fire on a tick.
Aggregation window for windowed observation types in needs. Must be 0.0 or a positive integer multiple of scale_interval_seconds.
Keyword arguments passed to the plugin class constructor.
Lifecycle
When you create a DGDR, it progresses through these phases (status.phase):
Conditions
The operator maintains these conditions on status.conditions:
Monitoring
Resource ownership
- The DGDR does not set an owner reference on the DGD it creates. Deleting a DGDR does not delete the DGD — it persists independently so it can continue serving traffic.
- The relationship is tracked via labels:
dgdr.nvidia.com/nameanddgdr.nvidia.com/namespace. - Additional resources (planner ConfigMaps) are created in the same namespace and labeled with
dgdr.nvidia.com/name.
Replay a recorded trace (v1beta2)
The nvidia.com/v1beta2 API is experimental and disabled by default. See the
installation guide before you enable it.
A native v1beta2 request can evaluate every candidate against a recorded request trace instead
of synthetic traffic. Set spec.workload.trace instead of spec.workload.static; the two are
mutually exclusive, and admission adds no static default when trace is set. To switch an existing
request from static traffic to a trace, set spec.workload.static to null in the same update.
Before you create the request:
- Create the PersistentVolumeClaim in the request’s namespace. Use a filesystem volume, not
volumeMode: Block. When the process that writes the trace keeps running, useReadWriteMany. WithReadWriteOnce, a Job scheduled to another node than the writer waits inContainerCreating, and withReadWriteOncePod, a Job waits inPendinguntil the writer’s pod is gone. The Run reportsRunningwhile its Job waits. - Finish writing the file.
subPathnames one file relative to the claim root; directories and globs are not expanded. Every Run of the request reads the file when its Job starts, so do not rewrite it afterwards. To replay several rolledjsonl_gzsegments, concatenate them into one.jsonl.gzfile. A relative symlink to another file in the claim works. An absolute symlink does not, because the kubelet resolves it on the node rather than inside the claim, and neither does a symlink that leaves the claim. - Make the file readable by others (for example, mode
0644in0755directories) or by group65532, which the Job’s pod has as a supplemental group. The pod sets nofsGroup, so neither the kubelet nor a CSI driver changes the ownership or permissions of anything on the claim.
The operator mounts only this file, read-only and only into the AISimulate container, through a
subPath of the claim at /var/run/aisimulate/trace/trace.jsonl.gz, or at
/var/run/aisimulate/trace/trace.jsonl when subPath does not end in .gz. The kubelet resolves
subPath inside the claim and refuses to mount a file outside it. If the file does not exist when
the Job starts, the kubelet cannot mount it either, and the AISimulate container waits; see
Trace claim lifecycle. On a volume type whose node directory the kubelet
can write, such as hostPath, the kubelet instead creates an empty directory at subPath on the
claim, and the Run fails when AISimulate cannot read it.
format defaults to Dynamo, the only supported format. It reads the Dynamo request trace
(dynamo.request.trace.v1) written by the frontend, as JSON Lines or as gzip-compressed JSON Lines
when the file name ends in .gz. See the Request Trace Reference.
The operator lowers the trace to the AISimulate input traffic.source (type: trace,
format: dynamo, and paths with the mounted file) and traffic.load.type: trace_timestamps,
with traffic.load.speedup when arrivalSpeedupRatio is set. It adds no traffic.stop.requests.
These values are an underlay, like every value lowered from a typed field:
spec.aisim.inputOverrides is applied last as a JSON Merge Patch and can replace any of them,
including traffic.source.type, traffic.source.format, and traffic.source.paths. A null
override deletes the named field or parent object, and the operator does not restore it. For
example, this override deletes the lowered source:
The typed persistentVolumeClaim source alone decides which claim the Run’s Job mounts and which
claim the Run protects. Overriding traffic.source.paths adds no volume mount. AISimulate validates
the resulting input when the Run’s Job starts, and an input it rejects fails the Run; admission does
not validate it against the AISimulate schema yet.
Trace claim lifecycle
The operator ties the claim’s lifetime to the Runs that read it. It decides from structured state
only: the claim’s deletionTimestamp, volumeMode, and phase, the Job’s conditions, and the
request’s spec. It copies container status messages into the Run but never interprets them.
- Protection. Before a Run creates its Job, the operator adds the finalizer
nvidia.com/dgdr-run-<run-uid>to the claim, where<run-uid>is the Run’smetadata.uid. Every Run that reads the claim adds its own finalizer, so Runs of several requests can share a claim. The operator removes a Run’s finalizer once the Run is complete and none of its Jobs is still running, or once the Run and its Jobs are deleted, for example when the request retires old Runs or is deleted. Kubernetes garbage collection deletes the Job of a deleted Run before the Job’s pod has stopped, so the finalizer of a Run deleted before it completed can be removed while that pod still mounts the file;kubernetes.io/pvc-protectionkeeps the claim until the pod is gone. - Deleting the claim cancels its Runs. When the claim has a
deletionTimestamp, the operator stops the Job of every Run that reads it through foreground deletion, whether or not the Job’s pod has started. Once the Job and its pod are gone, the Run fails with reasonTraceSourceDeleted, and the operator then removes the Run’s finalizer. A Job that already completed or failed before the claim’s deletion, and is not itself being deleted, is not stopped; its own outcome decides the Run. Once the operator has started stopping a Job, the Run fails withTraceSourceDeletedeven if the Job is marked failed while its pod is removed. A Run whose claim is already being deleted before it creates its Job fails withTraceSourceDeletedwithout creating anything. - A missing claim keeps the Run open. When the claim does not exist, the Run stays
PendingwithCompleted=Falseand reasonTraceSourceMissing, and starts once you create the claim. - A claim no Job can mount fails the Run before it starts. Before the Run creates its Job, a
volumeMode: Blockclaim or a claim in phaseLostfails the Run with reasonTraceSourceUnavailable. After the Job exists, the operator does not check the claim’s volume mode or phase again; the Job’s pod waits until the volume is usable, the Job fails, or you cancel the Run. - A waiting container is a diagnostic, not a failure. While a container of the Run’s Job waits
on an error, such as a trace file the kubelet cannot mount (a missing file, an absolute symlink,
or a symlink that leaves the claim) or an image it cannot pull, the Run stays open with
Completed=False, reasonContainerWaiting, and the messagecontainer "<name>" is waiting: <reason>: <message>, where<reason>and<message>are copied from the container’s status. The operator does not delete the Job, and there is no timeout. The kubelet keeps retrying the container, so fixing the cause, for example writing the missing file, lets the same Run continue. Waiting inContainerCreatingorPodInitializingis not reported: volume errors that keep a pod there, such as aReadWriteOnceclaim attached to another node, appear only in the pod’s events.
To stop a Run that cannot continue, cancel it explicitly: change the request’s spec (the Run
fails with reason Superseded), delete the request, or delete the claim (TraceSourceDeleted). A
Pending claim, such as one that waits for its first consumer, is accepted. The request starts one
Run per metadata.generation, so after a Run failed, change the request’s spec to start a new
Run.
Kubernetes keeps a deleted claim Terminating while any pod that mounts it exists
(kubernetes.io/pvc-protection), including the completed pods of finished Jobs. The operator keeps
the Runs, Jobs, and pods of a request’s five most recent finished Runs, so a deleted trace claim can
stay Terminating after every Run released it, until those Runs are retired or the request is
deleted.
If the operator is uninstalled, or the v1beta2 API is disabled, while Runs are in flight, their
finalizers stay on the claim and keep it from being deleted. List them, then remove only the
nvidia.com/dgdr-run- finalizers (requires jq):
Known issues
- PCIe profiler data not yet available. See the PCIe callout under SKU format.