Skip to navigation

DynamoGraphDeploymentRequest (DGDR) Reference

Field reference and lifecycle for the DynamoGraphDeploymentRequest custom resource — Dynamo’s deploy-by-intent generator.

View as Markdown

A DynamoGraphDeploymentRequest (DGDR) is Dynamo’s deploy-by-intent generator for DynamoGraphDeployment (DGD) resources. You describe what you want to run and your performance targets; the profiler determines a configuration and produces the DGD that serves traffic.

For the task-oriented walkthrough — authoring a request step by step — see Auto Deploy with DGDR. For ready-to-copy manifests, see DGDR Templates.

This page documents the nvidia.com/v1beta1 API — the storage version.

Conversion views

When the nvidia.com/v1beta2 API is served, the API server represents a native v1beta2 request through v1alpha1 or v1beta1 by populating spec.v1beta2. Conversely, it represents a native v1beta1 request through v1beta2 by populating spec.v1beta1. These fields are read-only conversion views. Create and modify a request through the API version that owns its native fields; do not set or edit the conversion field directly. An unchanged conversion view can be written back without changing the stored request.

v1beta2 hardware SKU

spec.hardware.gpu.sku in nvidia.com/v1beta2 is an open string containing an AISimulate system-catalog identifier. The AISimulate release bundled with this Dynamo release includes a100_pcie, a100_sxm, a30, b200_sxm, b300_sxm, b60, gb200, gb300, h100_pcie, h100_sxm, h200_sxm, l4, l40s, and rtx_pro_6000_server.

Admission passes an explicit SKU through unchanged. It warns when the bundled catalog does not contain that identifier, but does not reject it: an external AISimulate systems catalog may provide additional identifiers.

Omit hardware to let admission discover and persist sku, budget, and numGpusPerNode. An exactly empty hardware: {} object opts out of all operator cluster and DCGM discovery and typed hardware defaults. Any nonempty object, including hardware: {gpu: {}}, is partial hardware intent: admission discovers and persists each missing field while preserving every explicit value. The Run controller never performs hardware discovery.

Set all three values to bypass discovery with fully explicit hardware. An unknown SKU must provide both budget and numGpusPerNode, because admission cannot map that identifier to cluster capacity or topology.

apiVersion: nvidia.com/v1beta2
kind: DynamoGraphDeploymentRequest
metadata:
name: future-accelerator
spec:
model:
name: Qwen/Qwen3-0.6B
hardware:
gpu:
sku: future_accelerator # Passed to AISimulate unchanged; admission warns if unknown.
budget: 8
numGpusPerNode: 8 # Completes explicit hardware; admission performs no discovery.

To leave the complete hardware input to AISimulate and inputOverrides, use the empty-object opt-out and supply both the upstream hardware selector and candidate GPU budget yourself. Request and Run admission validate the merged input against the bundled AISimulate input schema. Run specs remain immutable; unchanged specs remain metadata-updatable after a bundled schema change.

spec:
hardware: {}
aisim:
inputOverrides:
engine:
hardware: h200_sxm
optimization:
constraints:
max_candidate_gpus: 8

v1beta2 DGD Overrides

Experimental; for development only. Enable dynamo-operator.dgdrV1Beta2.enabled in the platform Helm chart to use spec.overrides.dgd. It accepts one partial nvidia.com/v1beta1 DGD, applied to every selected Candidate across backends and topologies. Other DGD API versions are not accepted here.

Overrides follow the DGD schema’s Server-Side Apply merge rules. For example, components and named containers merge by name; container command and args are atomic lists and replace the entire generated list. These are examples: the schema’s map, set, and atomic markers determine how all fields merge. Omitted fields retain their generated values.

The existing DGD $patch: {args: append} customization is also supported on a named container. It appends the supplied arguments to an explicit generated args list and removes the directive before validation. Both the target container and its args must already exist. This is the only supported $patch operation.

apiVersion: nvidia.com/v1beta2
kind: DynamoGraphDeploymentRequest
metadata:
name: custom-worker-image
spec:
model:
name: Qwen/Qwen3-0.6B
backends: [vllm] # OPTIONAL; restricts results to this backend.
overrides: # OPTIONAL; omitted means no DGD override.
dgd: # OPTIONAL; partial DGD resource.
apiVersion: nvidia.com/v1beta1 # REQUIRED when dgd is set.
kind: DynamoGraphDeployment # REQUIRED when dgd is set.
metadata:
labels:
workload: customized
spec:
components:
- name: worker
podTemplate:
spec:
containers:
- name: main
image: registry.example/runtime:1.7.0

The override is frozen into the immutable Run spec and applied atomically: if a target component is absent or any part cannot be applied, the Candidate retains the unchanged generated DGD spec and reports Valid=False. A new named container can be added to an existing component through the normal merge; $patch: {args: append} requires an existing container instead. For example, a decode override can fail for an aggregated Candidate whose topology contains only worker. No subset of a failed override is applied.

Choose target names that exist in the results you want to keep. Restricting backends can avoid engine-specific mismatches, but does not guarantee a particular topology. A shared decode override can invalidate every aggregated result, including the entire result set. Invalid Candidates still count toward recommendation.maxCandidates and do not trigger replacement searches.

After a successful merge, a copy of the DGD undergoes strict server-side dry-run creation, including DGD defaults, schema, CEL, and validating webhooks. The dry-run response is discarded; the stored payload remains the one from before the check. Unknown fields and incorrect field types remain visible on invalid Candidates; Valid=False records the failure. Infrastructure failures are retried instead of being recorded as an invalid deployment.

Retryable Candidate preparation errors are visible on the Run as Published=False, reason PublicationBlocked, with the error in the condition’s message. This includes HTTP 403 from a DGD dry-run: RBAC changes propagate asynchronously, and a permission or policy configuration can be repaired. The publisher keeps retrying and preserves the already published Candidate references and progress. Once publication succeeds, it sets Published=True with reason PublicationSucceeded. Published is owned by the publisher; the Run controller independently owns Completed. Changing the request’s spec stops the old Run and its publisher before starting the replacement Run.

For example, the Run can report this status while retrying a denied DGD dry-run:

status:
conditions:
- type: Published
status: "False"
reason: PublicationBlocked
message: "DGD dry-run forbidden: exceeded namespace quota"
observedGeneration: 1
lastTransitionTime: "2026-11-04T18:00:00Z"

Candidate.spec contains exactly the DGD spec, including backendFramework. DGD labels and annotations are copied to Candidate metadata. If an override fails to apply, only the original generated labels and annotations are copied; metadata from the failed override is discarded. If the merge succeeds but dry-run validation fails, the override’s labels and annotations remain on the invalid Candidate. An override that makes the complete Candidate metadata invalid is rejected in full. The total annotation-size limit includes generated annotations and the reserved annotations added to the Candidate.

Candidate identity, including name, namespace, and owner references, cannot be overridden. The following metadata is also reserved; attempts to override it fail atomically:

  • Labels nvidia.com/dgdr-run-uid and nvidia.com/dgdr-candidate-id.
  • Annotation dgdr.nvidia.com/additional-resources.
  • Annotations with the nvidia.com/dgd- conversion-bookkeeping prefix.

Evaluation parameters, metrics, and the final Valid condition belong to Candidate status. The condition’s statuses and reasons are documented on DynamoGraphDeploymentCandidateStatus.conditions.

A Candidate can retain an invalid DGD, including unknown fields or incorrect field types. Check for status.conditions with type: Valid and status: "True" before using its spec as a DGD. A missing condition does not mean validation succeeded.

apiVersion: nvidia.com/v1beta2
kind: DynamoGraphDeploymentCandidate
metadata:
name: example-candidate
spec: # Immutable, opaque DGD spec; preserved even when validation fails.
backendFramework: vllm
components:
- name: frontend
type: frontend
replicas: 1
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.7.0
- name: worker
type: worker
replicas: 1
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.7.0
args: [--model, Qwen/Qwen3-0.6B]
status:
parameters: # Original resolved simulation inputs.
tp: 2
metrics: # Original simulation results; not recomputed after overrides.
throughput: 12.5
conditions:
- type: Valid
status: "True"
reason: ValidationSucceeded
message: Strict DGD dry-run succeeded
observedGeneration: 1
lastTransitionTime: "2026-10-30T12:00:00Z"
status.conditions
Condition[]

The Valid condition reports whether the Candidate’s DGD passed override application and strict API admission. Its absence does not indicate success.

TypeStatusReasonMeaning
ValidTrueValidationSucceededThe override was applied in full and the DGD passed strict dry-run admission.
ValidFalseOverrideTargetNotFoundAn override names a component absent from the generated DGD; the original DGD is retained.
ValidFalseOverrideFailedThe override could not be applied in full, for example because it changes reserved metadata or has an invalid merge directive; the original DGD is retained.
ValidFalseValidationFailedThe resulting DGD was rejected by strict dry-run admission.

Valid=True means the API object is structurally valid and passed strict DGD admission at checking time. It does not indicate a running or Ready deployment, or prove image availability, GPU capacity, or successful workload scheduling.

Overrides may change model, backend, GPU allocation, parallelism, images, or arguments. The original simulation parameters and metrics are not a performance prediction for the overridden DGD.

List only Candidates with Valid=True in the current namespace (requires jq):

kubectl get dynamographdeploymentcandidates -o json \
| jq -r '.items[] | select(any(.status.conditions[]?; .type == "Valid" and .status == "True")) | .metadata.name'

Invalid Candidates count toward recommendation.maxCandidates and retain their original search order in Run references. They do not trigger replacement searches. A successful Run requires a successful search and at least one Valid=True Candidate; when all selected Candidates are invalid, the Run fails and keeps them for diagnosis.

Spec reference

Only model is unconditionally required by the schema. Admission rejects a new DGDR whose supplied spec.image has no parseable semantic-version tag unless spec.runtimeVersionOverride is set. When spec.image is omitted, the operator supplies a versioned default as described below.

model
stringRequired

Model to deploy — a HuggingFace ID (for example Qwen/Qwen3-0.6B, meta-llama/Llama-3-70b) or a private model name. Must be non-empty.

image
string

Container image for the profiling job (the planner image), for example nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.4.0. For Dynamo < 1.1.0, use dynamo-frontend.

runtimeVersionOverride
string

Required when spec.image is set and has no parseable semantic-version tag, such as latest, a custom build tag, or a digest-only reference. Also supplies the default Dynamo runtime version for generated DGD components that do not set their own override. An explicit component value in overrides.dgd takes precedence. Set this when an effective generated component image has no parseable semantic-version tag or its tag does not identify the intended Dynamo runtime.

Use canonical MAJOR.MINOR.PATCH, for example "1.4.0", with each part between 0 and 9999 and no leading zeros, v prefix, prerelease suffix, or build metadata. This field does not change any image. See Profiler image version compatibility.

backend
stringDefaults to auto

Inference backend used for profiling and deployment.

Allowed values: auto vllm sglang trtllm
searchStrategy
stringDefaults to rapid

Profiling depth. rapid performs a fast, AIC-backed sweep (~30s); thorough explores more configurations on real GPUs (2–4h).

Allowed values: rapid thorough
autoApply
booleanDefaults to true

Automatically create a DGD after profiling completes. When false, the generated spec is stored in status for manual review instead of being applied.

Profiler image version compatibility

When spec.image is omitted, the operator defaults it on creation to nvcr.io/nvidia/ai-dynamo/dynamo-planner:<operatorVersion>. The operator requires operatorVersion to be valid semantic versioning, so the default image has a parseable version tag.

The DGDR-level spec.runtimeVersionOverride supplies a default for generated DGD components. After the profiler materializes the DGD and applies spec.overrides.dgd, the operator copies the DGDR value only to components that do not already have an explicit override. This behavior applies across profiler versions. Profilers through Dynamo 1.3.0 may discard the field while parsing the DGDR, but the operator reapplies the default from the stored DGDR.

Set spec.runtimeVersionOverride when an effective generated component runtime image uses a non-semantic-version tag or digest, or when its tag does not identify the intended Dynamo runtime version. The DGDR value also covers images replaced through spec.overrides.dgd when that component does not set its own runtimeVersionOverride. An explicit component-level value takes precedence.

While a DGDR has autoApply: false, you may set or change runtimeVersionOverride during Profiling or Ready. After it reaches Ready, you may enable autoApply, either separately or while changing the override. The operator applies the current value while creating the new DGD. These updates do not modify status.profilingResults.selectedConfig or the nvidia.com/generated-dgd-spec annotation. Other spec updates remain forbidden after profiling starts.

When the DGDR-level field is unset, each effective generated DGD component follows the component runtime version rules independently. An overridden runtime image without a parseable semantic-version tag requires runtimeVersionOverride on that component.

Profilers through Dynamo 1.3.0 discard the embedded DGD override’s apiVersion and kind, then directly merge the remaining dictionary into a generated v1alpha1 DGD. Raw storage preserves the override fields but does not convert between DGD schemas. Use a v1alpha1-shaped override under spec.services; a v1beta1 spec.components override is not translated into the generated service.

workload
WorkloadSpec

Expected workload characteristics for SLA-based profiling.

isl
integerDefaults to 4000

Input Sequence Length — expected average input tokens.

osl
integerDefaults to 1000

Output Sequence Length — expected average output tokens.

concurrency
number

Optional target concurrency level. Mutually exclusive with requestRate. When neither field is set and the Planner is disabled, the profiler uses its default maximum-throughput selection.

requestRate
number

Optional target request rate in requests per second. Mutually exclusive with concurrency. When neither field is set and the Planner is disabled, the profiler uses its default maximum-throughput selection.

sla
SLASpec

Service-level agreement targets that drive profiling optimization.

ttft
numberDefaults to 2000

Time To First Token target, in milliseconds.

itl
numberDefaults to 30

Inter-Token Latency target, in milliseconds.

e2eLatency
number

End-to-end request latency target, in milliseconds. An alternative to specifying ttft + itl; cannot be combined with them.

optimizationType
OptimizationType

Optimization target for SLA profiling.

Allowed values: latency throughput
hardware
HardwareSpec

GPU hardware for profiling and deployment.

gpuSku
GPUSKUType

GPU type to target (see SKU format). When omitted, auto-detected as the GPU type of the node with the most GPUs (ties broken by highest VRAM per GPU). In mixed-GPU clusters, set this to choose the GPU type; discovery and totalGpus are then restricted to matching nodes.

vramMb
number

VRAM per GPU in MiB. Auto-detected from cluster GPU nodes when omitted. An explicit value within 10% of the auto-selected shape’s measured VRAM permits automatic GPU-budget discovery; for example, vramMb: 81920 matches a measured 81079 MiB. This tolerance applies only to the requested value; aggregating discovered nodes still requires equal measured VRAM. If this value differs by more than 10% and totalGpus is omitted, automatic GPU-budget discovery fails and the DGDR moves to phase: Failed before a profiling Job is created; set totalGpus explicitly to use that override.

totalGpus
integer

GPU budget for profiling and deployment; the profiler uses it to determine parallelism and replica count. When omitted, computed by summing GPUs on nodes matching the auto-selected shape: GPU type, GPUs per node, and VRAM per GPU. The result is temporarily capped at 32 to limit the profiler search space. This reflects discovered GPUs, not currently free capacity. For example, same-type nodes with 8 and 4 GPUs represent 12 GPUs of inventory, while the auto-selected 8-GPU shape yields an automatic budget of 8. Set explicitly to override the derived budget, use a different numGpusPerNode, or use a vramMb value more than 10% from the auto-selected shape’s measured VRAM; these shape overrides require an explicit totalGpus.

numGpusPerNode
integer

GPUs per node. When omitted, auto-detected as the highest GPU count on any discovered node of the selected GPU type. If this value differs from the auto-selected shape and totalGpus is omitted, automatic GPU-budget discovery fails and the DGDR moves to phase: Failed before a profiling Job is created; set totalGpus explicitly to use that override.

interconnect
string

Primary GPU-to-GPU interconnect within a node, for example nvlink or pcie. Capability metadata for profiling and planning only — it does not enable any interconnect. Best-effort discovery distinguishes nvlink from pcie when omitted. An over-optimistic value can lead the planner to overestimate intra-node bandwidth and pick overly aggressive parallelism.

rdma
boolean

Whether the cluster has RDMA-capable networking for Dynamo data movement. Capability metadata only — it does not install or configure RDMA. Best-effort discovery via node labels when omitted. A false positive can cause transports to assume RDMA is usable and fail to connect; a false negative falls back to non-RDMA transports.

modelCache
ModelCacheSpec

Optional PVC configuration for pre-downloaded model weights. When set, weights are loaded from the PVC instead of downloading from HuggingFace.

pvcName
string

Name of a ReadWriteMany PVC containing cached model weights. The PVC must exist in the same namespace as the DGDR.

pvcModelPath
string

Path to the model checkpoint directory within the PVC, for example deepseek-r1 or models/Llama-3.1-405B-FP8.

pvcMountPath
stringDefaults to /opt/model-cache

Mount path for the PVC inside the container.

features
FeaturesSpec

Optional Dynamo platform features in the generated deployment.

planner
runtime.RawExtension

A PlannerConfig object passed to the Planner service. The presence of this field (non-null) enables the planner in the generated DGD. DGDR passes it through without field-level validation; the Planner service validates it at startup. See Planner for the full field list.

runtime.RawExtension
kvRouter
KVRouterSpec

Configures KV-cache-aware routing for the generated deployment. When enabled, DGDR sets DYN_ROUTER_MODE=kv on the generated Frontend.

enabled
boolean

Enables KV-cache-aware routing.

mocker
MockerSpec

Configures the simulated (mocker) backend for testing without GPUs.

enabled
boolean

Deploy mocker workers instead of real inference workers.

spec.overrides.dgd takes precedence over features.kvRouter. A DGD override can replace DYN_ROUTER_MODE on the Frontend container or provide an explicit --router-mode argument. The command-line flag takes precedence over the environment variable when both are present. See Generated DGD overrides.

Overrides

overrides
OverridesSpec

Customizes the profiling job and the generated DGD.

profilingJob
batch/v1.JobSpec

Overrides merged into the controller-generated profiling Job spec (for example, tolerations or node selectors). Only customizes the profiling Job — not the deployed worker pods.

batch/v1.JobSpec
dgd
runtime.RawExtension

A partial, versioned DGD merged into the generated deployment. Use nvidia.com/v1beta1 for new overrides; nvidia.com/v1alpha1 remains supported for compatibility. See Generated DGD overrides.

runtime.RawExtension

Planner

spec.features.planner is a PlannerConfig object, passed through to the Planner service without field-level validation; the Planner validates it at startup. Setting it (non-null) enables the planner in the generated DGD. The same object is documented standalone at Planner Configuration; its fields are reproduced here. Nested list-and-object types are expanded under Additional Types. All fields are optional.

Core settings

mode
stringDefaults to disagg

Planner operating mode, matching the deployment topology.

Allowed values: disagg prefill decode agg
backend
stringDefaults to vllm

Inference backend the Planner scales.

Allowed values: vllm sglang trtllm mocker
environment
stringDefaults to kubernetes

Runtime environment that determines how the Planner applies scaling actions.

Allowed values: kubernetes virtual global-planner
namespace
stringDefaults to env DYN_NAMESPACE, else dynamo

Dynamo namespace of the deployment the Planner manages. Defaults from DYN_NAMESPACE, which the operator injects as {k8s_namespace}-{dgd_name}. Excluded from serialized output.

model_name
stringDefaults to null

Optional model name override. Auto-detected from the deployment when unset.

global_planner_namespace
stringDefaults to null

Namespace where the GlobalPlanner runs. Required when environment is global-planner.

log_dir
stringDefaults to null

Optional directory for Planner log output.

Optimization target

optimization_target
stringDefaults to throughput

Scaling strategy. throughput and latency use static thresholds on queue depth and KV cache utilization — no SLA targets or profiling required. load uses user-defined prefill queue-token and decode KV-utilization thresholds. sla uses the AISimulate RustForwardPassPerfModel.best_available(config) performance model to target specific ttft_ms / itl_ms values. Under any target other than sla, the Planner forces load scaling on, throughput scaling off, and ignores ttft_ms / itl_ms.

Allowed values: throughput latency load sla
enable_throughput_scaling
booleanDefaults to true

Enable predictive, traffic-based scaling. Only honored when optimization_target is sla.

enable_load_scaling
booleanDefaults to false

Enable reactive, load-based scaling. Only honored when optimization_target is sla. At least one scaling mode must be enabled.

ttft_ms
numberDefaults to 500.0

Time To First Token SLA target, in milliseconds. Also accepts the alias ttft. Must be greater than 0. Used only under optimization_target: sla.

itl_ms
numberDefaults to 50.0

Inter-Token Latency SLA target, in milliseconds. Also accepts the alias itl. Used only under optimization_target: sla.

Performance model and pre-deployment sweeping

pre_deployment_sweeping_mode
stringDefaults to rapid

How to generate optional bootstrap performance data. none skips bootstrap data; rapid uses AIConfigurator to simulate engine performance (~30s); thorough measures on real GPUs (several hours).

Allowed values: none rapid thorough
profile_results_dir
stringDefaults to profiling_results

Directory holding profiler-generated performance data (npz or JSON), used to bootstrap or tune the performance model when the get_perf_metrics endpoint is unavailable.

ais_perf_model
AISPerfModelSpecDefaults to null

Role-indexed AISimulate forward-pass configurations. Constructs each model through aisimulate_core.sdk.RustForwardPassPerfModel.best_available(config) without an interpolation sweep. Configure model identity and estimator controls through ais_perf_model.roles; the retired aic_perf_model and hf_id / pick shape are rejected.

roles
objectRequired

Maps deployment roles to complete upstream ForwardPassPerfModelConfig objects. Supply prefill and decode for disagg, the matching single role for prefill or decode, and aggregated for agg. Each key must match worker_type.

roles.<role>.model / system / backend
stringRequired

Model identifier, AISimulate system identifier, and inference backend (vllm, sglang, or trtllm). Optional backend_version, tp, pp, attention_dp, MoE axes, quantization, KV block size, and speculation fields use the upstream schema. Published worker capabilities must agree with the configured identity.

roles.<role>.estimation_mode
stringDefaults to auto

auto searches op_level, fpm_interpolation, then fpm_regression. Explicit modes select the named estimator.

roles.<role>.fallback_policy
stringDefaults to deny

deny rejects failure of an explicitly selected mode. allow tries the other modes in op_level, fpm_interpolation, fpm_regression priority order; regression permits a regression fallback. Auto selection searches the full priority order even with deny.

roles.<role>.systems_paths
list[string]Defaults to []

Ordered performance-data roots. The SDK resolves its packaged root when unset. database_mode and transfer_policy are also passed through unchanged.

roles.<role>.estimator_config
objectDefaults to {}

Complete upstream estimator controls, including independent regression and correction sampling, fit settings, feature weights, and correction bounds. Explicit settings override Planner sampling defaults. A legacy total bucket count of 16 maps to bins_per_axis: [4, 4]. Unknown fields are rejected by AISimulate. See the AISimulate core API.

aic_interpolation
AICInterpolationSpecDefaults to null

AIConfigurator interpolation spec. Populated by the profiler in rapid mode and written onto the Planner ConfigMap; you do not normally set this by hand. When present, the Planner runs the AIC sweep in-process at bootstrap to seed the performance model. See AICInterpolationSpec.

GPU budget

max_gpu_budget
integerDefaults to 8

Maximum total GPUs the Planner may allocate across worker types.

min_gpu_budget
integerDefaults to -1

Per-DGD GPU floor enforced by the local Planner. -1 disables it. When set with max_gpu_budget such that min == max, the Planner pins the per-DGD total and only redistributes replicas between prefill and decode.

min_endpoint
integerDefaults to 1

Minimum engine endpoints for aggregated deployments. In disaggregated deployments, this value applies to both prefill and decode unless a role-specific value is set. In prefill-only or decode-only deployments, it supplies the active role when the corresponding role-specific value is unset. Must be nonnegative; use 0 for scale-to-zero compatibility.

prefill_min_endpoint
integer | nullDefaults to null

Minimum prefill endpoints in disaggregated and prefill-only modes. When set, replaces the prefill value supplied by min_endpoint. Must be at least 1.

decode_min_endpoint
integer | nullDefaults to null

Minimum decode endpoints in disaggregated and decode-only modes. When set, replaces the decode value supplied by min_endpoint. Aggregated deployments use min_endpoint. Must be at least 1.

decode_engine_num_gpu
integerDefaults to null

GPUs per decode engine replica. Auto-detected from the deployment when unset.

prefill_engine_num_gpu
integerDefaults to null

GPUs per prefill engine replica. Auto-detected from the deployment when unset.

Throughput-based scaling

throughput_adjustment_interval_seconds
integerDefaults to 180

Seconds between throughput-based scaling decisions. Also accepts the alias throughput_adjustment_interval.

max_throughput_scaling_replicas
integerDefaults to 8

Maximum replica-count change per component from one throughput observation. The Planner caps and stores the throughput lower bound, so load-scaling ticks cannot repeatedly apply the same uncapped prediction. GPU and power budgets can reduce the change further. At the nominal minimum GPU budget, scale-down-only requests hold. At a full GPU budget, two scale-up requests hold; an opposing prefill/decode proposal applies only the largest atomic pair that fits the budget. Endpoint and out-of-band budget recovery take precedence over this limit. Must be greater than 0.

throughput_metrics_source
stringDefaults to frontend

Prometheus traffic source. frontend reads dynamo_frontend_* metrics from the public Frontend; router reads dynamo_component_router_* from a LocalRouter (use for a pool-local Planner in GlobalPlanner deployments).

Allowed values: frontend router

Load-based scaling

load_adjustment_interval_seconds
integerDefaults to 5

Seconds between FPM tuning updates and load-based scaling decisions. Even when only throughput scaling is enabled, live FPM observations feed the performance model at this interval. Also accepts the alias load_adjustment_interval. Must be greater than 0 and, when both scaling modes are on, shorter than throughput_adjustment_interval_seconds.

max_num_fpm_samples
integerDefaults to 64

Maximum retained ForwardPassMetrics observations for online tuning and regression fallback.

fpm_sample_bucket_size
integerDefaults to 16

Number of buckets for observation retirement. Must be a perfect square.

load_scaling_down_sensitivity
integerDefaults to 80

Scale-down sensitivity from 0 to 100 for SLA-mode load scaling (100 = most aggressive). It sets how much latency headroom the performance model must predict before load-based scaling removes a replica; 0 blocks these model-based scale-downs. It does not apply when every fresh FPM observation for a pool is idle: the Planner then removes one replica per decision, down to the larger of 1 and the endpoint floor. Set min_endpoint (or the per-component floors) to keep idle capacity. Ignored unless optimization_target is sla.

load_min_observations
integerDefaults to 5

Minimum observations before load-based scaling decisions begin (cold-start threshold).

prefill_scale_up_queue_tokens
integerDefaults to null

Prefill queue-token count that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes prefill. Must be greater than 0 and greater than prefill_scale_down_queue_tokens.

prefill_scale_down_queue_tokens
integerDefaults to null

Prefill queue-token count that allows scale-down. Required alongside prefill_scale_up_queue_tokens.

decode_scale_up_kv_rate
numberDefaults to null

Decode KV-utilization percentage (0–100) that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes decode. Must be greater than decode_scale_down_kv_rate.

decode_scale_down_kv_rate
numberDefaults to null

Decode KV-utilization percentage (0–100) that allows scale-down. Required alongside decode_scale_up_kv_rate.

speculative_nextn
integerDefaults to 0

Manual fallback speculative-decoding depth. A worker’s published runtime_config.runtime_data.spec_decode.nextn takes precedence when present. Must be 0 or greater.

Load prediction

load_predictor
stringDefaults to arima

Prediction method for request count, ISL, and OSL.

Allowed values: constant arima kalman prophet
load_predictor_log1p
booleanDefaults to false

Apply a log1p transform to predicted request count, ISL, and OSL.

prophet_window_size
integerDefaults to 50

Window size (seconds) for the Prophet predictor.

load_predictor_warmup_trace
stringDefaults to null

Path to a warmup trace file for bootstrapping predictions.

kalman_q_level
numberDefaults to 1.0

Kalman process noise for the level component.

kalman_q_trend
numberDefaults to 0.1

Kalman process noise for the trend component.

kalman_r
numberDefaults to 10.0

Kalman measurement noise.

kalman_min_points
integerDefaults to 5

Minimum data points before Kalman predictions activate.

Prometheus metrics

These fields default from environment variables and are excluded from serialized output.

metric_pulling_prometheus_endpoint
stringDefaults to env PROMETHEUS_ENDPOINT

Prometheus endpoint the Planner queries for traffic metrics. Defaults to PROMETHEUS_ENDPOINT, else the in-cluster kube-prometheus service URL.

metric_pulling_prometheus_token
stringDefaults to env PROMETHEUS_TOKEN

Optional bearer token sent as Authorization: Bearer <token> on every PromQL request. Read once at startup.

metric_pulling_prometheus_token_file
stringDefaults to env PROMETHEUS_TOKEN_FILE

Path to a file containing a bearer token, re-read before every request so rotated tokens are picked up without a restart.

metric_pulling_prometheus_ssl_verify
booleanDefaults to false

Verify the upstream Prometheus TLS certificate. Defaults from PROMETHEUS_SSL_VERIFY (1/true/yes enable it). Pair with a CA bundle for a private CA.

metric_pulling_prometheus_extra_query_params
objectDefaults to env PROMETHEUS_EXTRA_QUERY_PARAMS

Fixed key/value pairs appended as URL query parameters on every PromQL request. Set PROMETHEUS_EXTRA_QUERY_PARAMS as a URL query string, for example namespace=my-ns&tenant=foo.

metric_pulling_prometheus_ca_bundle
stringDefaults to env PROMETHEUS_CA_BUNDLE

Path to a CA bundle for verifying the upstream Prometheus TLS certificate. When set, the bundle is used for verification regardless of metric_pulling_prometheus_ssl_verify. Must point to an existing file.

metric_pulling_prometheus_request_timeout_seconds
numberDefaults to 10.0

Connection and read inactivity timeout in seconds for each Prometheus API request. Defaults from DYN_PLANNER_PROMETHEUS_REQUEST_TIMEOUT_SECONDS and must be greater than 0. This is not a total wall-clock deadline: a response that continues delivering data can run longer. The Planner does not retry failed requests within the same collection cycle.

metric_reporting_prometheus_port
integerDefaults to 0

Port on which the Planner exposes its own dynamo_planner_* metrics. Defaults from PLANNER_PROMETHEUS_PORT. 0 disables.

Advisory and diagnostics

advisory
booleanDefaults to false

Suggestion-only mode. The Planner computes, logs, exports, and reports recommended replica counts without executing scaling actions or changing the deployment. Use it to evaluate a new configuration or validate SLA targets against production traffic. A planner-in-the-loop replay of the same configuration also keeps the simulated deployment fixed.

report_interval_hours
numberDefaults to 24.0

Generate an HTML diagnostics report every N hours (simulated time). Set to null to disable. Must be a positive finite number or null.

report_output_dir
stringDefaults to ./planner_reports

Directory for HTML diagnostics reports.

report_filename
stringDefaults to null

Fixed report filename written under report_output_dir. When unset, a timestamped name is used.

report_write_gzip_log
booleanDefaults to true

Write a compressed JSONL diagnostics log (.log.jsonl.gz) next to each HTML report.

live_dashboard_port
integerDefaults to 8080

Port for the live diagnostics dashboard HTTP server. 0 disables. When enabled, visit http://host:port/ for a real-time Plotly report.

control_api_port
integerDefaults to 9086

Port for the loopback-only runtime minimum-endpoint API. 0 disables the API. The operator does not expose this port through a Kubernetes Service.

Scheduling and plugin pipeline

scheduling
SchedulingConfig

Plugin-pipeline scheduling. Rejects unknown fields.

scale_interval_seconds
numberDefaults to gcd of enabled builtin intervals

Base pipeline cadence. The pipeline wakes once per interval; each plugin’s execution_interval_seconds decides whether it fires. When unset, computed as the gcd of load_adjustment_interval_seconds and, when throughput scaling is enabled, throughput_adjustment_interval_seconds. Must be greater than 0 and evenly divide both adjustment intervals.

tick_max_duration_seconds
numberDefaults to 30.0

Outer deadline wrapping the full 4-stage pipeline. Exceeding it aborts the tick; the next tick runs from a clean state. Must be greater than 0.

external_plugins
[]ExternalPluginEntryDefaults to []

Static external-plugin registrations applied at startup. Per-entry failures are logged but do not crash the Planner. See ExternalPluginEntry.

gateway
GatewayConfig

gRPC registration gateway for self-registering plugins. Disabled by default.

enabled
booleanDefaults to false

Open the gRPC gateway at listen. Not needed for static external_plugins.

listen
stringDefaults to unix:///var/run/dynamo/planner/registry.sock

Bind address. A unix: socket path for in-Pod registration, or host:port for TCP.

allow_insecure
booleanDefaults to false

Permit binding a plaintext (no-TLS) gateway on a TCP listen. Fails closed by default because the gateway receives plugins’ shared-secret tokens. unix: listens are always allowed.

plugin_registration
PluginRegistrationConfig

Plugin registry configuration — auth, transport, heartbeat, and in-process plugins. Rejects unknown fields.

auth
AuthConfig

Registry authentication. In the Planner process, an empty trusted_sources activates a legacy compatibility fallback to unauthenticated access and logs a warning. This fallback is DEV ONLY; do not rely on it as an intentional configuration. Select allow_unauthenticated explicitly for development. Production deployments must configure static_secret.

trusted_sources
[]stringDefaults to []

Accepted auth sources. Use ['allow_unauthenticated'] for development, or ['static_secret'] with static_secrets for production.

Allowed values: static_secret allow_unauthenticated
static_secrets
objectDefaults to {}

Map of secret_value to subject_label. Populate from a mounted Kubernetes Secret rather than hard-coding in the ConfigMap.

transport
TransportConfig

Outbound gRPC transport settings for calling plugins.

allow_insecure_grpc
booleanDefaults to false

Allow plaintext grpc:// plugin endpoints. Required to use any grpc:// endpoint; mTLS support is not yet shipped.

request_timeout_seconds
numberDefaults to 5.0

Per-RPC timeout applied to every plugin call. Must be greater than 0.

keepalive_time_ms
integerDefaults to 30000

gRPC keepalive time, in milliseconds.

max_message_size_bytes
integerDefaults to 10485760

Maximum gRPC message size (10 MB).

protocol_version_min
stringDefaults to 1.0

Minimum accepted plugin protocol version.

protocol_version_max
stringDefaults to 1.0

Maximum accepted plugin protocol version.

heartbeat_timeout_seconds
numberDefaults to 15.0

Reserved liveness setting. Heartbeats update registry timestamps, but no monitor currently applies this timeout or evicts plugins automatically.

heartbeat_missed_threshold
integerDefaults to 2

Reserved liveness setting. Automatic missed-heartbeat eviction is not currently wired.

in_process_plugins
[]InProcessPluginSpecDefaults to []

In-process plugins loaded from Python modules at startup. See InProcessPluginSpec.

admin
AdminAuthConfig

Admin (ListPlugins) RBAC config. Parsed but currently inert — the gateway default-denies ListPlugins regardless of mode.

mode
stringDefaults to allow_all
Allowed values: allow_all k8s_rbac

spec.overrides.dgd is not required to enable the Planner. Use spec.features.planner for Planner enablement and configuration; use spec.overrides.dgd to customize profiler-generated benchmark deployments and the final DGD. For the deployment workflow, scaling-mode concepts, and cross-field validation rules, see the Planner Configuration reference and Planner Guide.

Generated DGD overrides

spec.overrides.dgd is a partial, versioned DGD. It does not define or extend the profiler’s candidate topology. Thorough profiling merges the override into each generated benchmark candidate before measurement and into the interpolation deployment. Every search strategy merges it into the final generated DGD. Use apiVersion: nvidia.com/v1beta1 and kind: DynamoGraphDeployment for new overrides. Components, pod-template containers, and container environment variables merge by name; atomic lists such as graph-level spec.env and container args replace their generated values.

To permit custom model repository code without depending on the selected topology, set spec.overrides.trustRemoteCode: true. The profiler applies --trust-remote-code to every generated vLLM or SGLang worker after the component topology is known. Enable this setting only for model repositories you trust.

Generated component names

An entry in spec.overrides.dgd.spec.components must use the exact, case-sensitive component name from the generated DGD. These are DGD component names, not Kubernetes Service resource names. The current profiler templates generate these components:

BackendAggregatedDisaggregated
vLLMFrontend, workerFrontend, decode, prefill
SGLangFrontend, decodeFrontend, decode, prefill
TensorRT-LLMFrontend, TRTLLMWorkerFrontend, decode, prefill

Setting spec.features.mocker.enabled: true replaces the inference workers with decode and prefill mocker components. Setting spec.features.planner adds a Planner component.

Add overrides before profiling starts. The DGDR spec becomes immutable during profiling, so you cannot inspect selectedConfig and then add overrides to the same resource. To inspect the names generated by a specific Dynamo release, follow the discovery workflow, then recreate the DGDR with the required overrides.

For example, set an environment variable on one generated component:

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
name: qwen3-sglang
spec:
model: Qwen/Qwen3-30B-A3B
backend: sglang
overrides:
dgd:
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
spec:
components:
- name: decode
podTemplate:
spec:
containers:
- name: main
env:
- name: TRITON_PTXAS_PATH
value: /usr/local/cuda/bin/ptxas

Overrides can modify only components present in the generated DGD. An unresolved spec.components or spec.services name causes DGD materialization to fail. A recognized deprecated name may be translated to the corresponding component in the selected topology with a warning. An override cannot add deployment topology. Labels and annotations merge, metadata.name selects the final DGD name, and other identity metadata is ignored. status, null values for typed fields, and field deletion are not supported.

To add flags to a container already present in the generated DGD, set $patch.args to append on that named container:

spec:
overrides:
dgd:
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
spec:
components:
- name: Frontend
podTemplate:
spec:
containers:
- name: main
$patch:
args: append
args:
- --router-mode
- kv

The only supported modifier is args: append, and its args list must contain at least one non-empty string. The target container and its args list must already be present in the generated blueprint. The profiler blueprints define the complete default CLI for Frontend/main, so the example preserves python3 -m dynamo.frontend and appends the KV router flags. The override helper combines the argument lists and removes $patch before returning the final DGD. Without $patch, specifying args retains the standard replacement behavior and requires the complete desired list.

Existing nvidia.com/v1alpha1 overrides remain supported. They use the older spec.services shape, merge services by name, and append extraPodSpec.mainContainer.args to generated worker arguments. In v1beta1, container args is atomic and replaces the generated list. Use v1beta1 for new overrides and provide the complete desired argument list when overriding args.

overrides.profilingJob customizes only the profiling Job. Use overrides.dgd for settings that must appear on deployed components.

For worked examples, see DGDR Templates. For router configuration and EPP topologies, see the Router Guide and Gateway API Inference Extension.

SKU format

hardware.gpuSku uses lowercase underscore format (h100_sxm, not H100-SXM5-80GB). When gpuSku is omitted, the operator normalizes the discovered GPU product name (for example, NVIDIA B300 SXM6 AC) to the matching value (b300_sxm). The observed Vera Rubin product name NVIDIA VR NVL72 ES and its GPU Feature Discovery form NVIDIA-VR-NVL72-ES map to the canonical AISimulate identifier vr_nvl72. GB200 discovery uses the canonical AISimulate identifier gb200:

spec:
hardware:
gpuSku: gb200
Allowed values: vr_nvl72 gb200 gb200_sxm gb10 b300_sxm b200_sxm h200_sxm h100_sxm h100_pcie a100_sxm a100_pcie a30 l40s l40 l4 v100_sxm v100_pcie t4 mi200 mi300

gb200_sxm is deprecated because GB200 systems use NVL rather than SXM. Existing manifests remain admitted with their existing behavior, but admission emits a warning. Update manifests to use gpuSku: gb200; newly discovered GB200 hardware is stored as gb200.

Not all SKUs are supported by the AIC profiler for rapid mode.

PCIe variants not yet supported by the profiler. The CRD admits PCIe SKUs (h100_pcie, a100_pcie, v100_pcie), but the profiler does not currently ship training data for them. You can submit a DGDR with a PCIe value; the operator accepts it but profiler-assisted sizing falls back to defaults. Profiler support for PCIe SKUs is tracked as an engineering follow-up.

Status

The operator maintains observed state under status. See Lifecycle for how phase and profilingPhase progress, and for the full condition-type list.

status.phase
DGDRPhase

High-level lifecycle phase of the deployment request.

Allowed values: Pending Profiling Ready Deploying Deployed Failed
status.profilingPhase
ProfilingPhase

Current sub-phase of the profiling pipeline. Only meaningful while phase is Profiling; cleared when profiling completes or fails.

Allowed values: Initializing SweepingPrefill SweepingDecode SelectingConfig BuildingCurves GeneratingDGD Done
status.dgdName
string

Name of the generated or created DynamoGraphDeployment.

status.profilingJobName
string

Name of the Kubernetes Job running the profiler.

status.conditions
[]metav1.Condition

Latest observed conditions. Standard types include Succeeded, Validation, Profiling, SpecGenerated, and DeploymentReady — see Conditions.

metav1.Condition
status.profilingResults
ProfilingResultsStatus

Output of the profiling process, including the selected deployment configuration.

pareto
[]ParetoConfig

Deprecated compatibility field retained for status objects written by older releases. The current controller does not populate it.

selectedConfig
runtime.RawExtension

Recommended configuration chosen by the profiler based on the SLA targets. This is the configuration used for deployment when autoApply is true.

runtime.RawExtension
status.deploymentInfo
DeploymentInfoStatus

State of the deployed DynamoGraphDeployment. Populated once a DGD has been created (via autoApply or manually).

replicas
integer

Desired number of replicas.

availableReplicas
integer

Number of replicas that are available and ready.

status.observedGeneration
integer

Most recent metadata.generation observed by the controller.

Additional Types

Nested struct types referenced by the fields above, broken out here to keep the field lists shallow. Types prefixed batch/v1., metav1., or runtime. are standard Kubernetes types and link to their Go package documentation instead of being expanded.

ParetoConfig

Deprecated compatibility type for status.profilingResults.pareto. Current profiling runs do not populate this type.

config
runtime.RawExtension

Full deployment configuration for this Pareto point.

runtime.RawExtension

PickedParallelConfig

A parallelism pick emitted by AIConfigurator. Referenced by AICInterpolationSpec.

tp
integerDefaults to 1

Tensor-parallel size.

pp
integerDefaults to 1

Pipeline-parallel size.

dp
integerDefaults to 1

Data-parallel (attention-DP) size.

moe_tp
integerDefaults to 1

Mixture-of-Experts tensor-parallel size.

moe_ep
integerDefaults to 1

Mixture-of-Experts expert-parallel size.

AICInterpolationSpec

Everything the Planner needs to reproduce a rapid-mode AIC sweep. Written by the profiler onto the Planner ConfigMap; not normally hand-authored. Referenced by features.planner.aic_interpolation.

hf_id
stringRequired

HuggingFace model id.

system
stringRequired

AIC system identifier.

backend
stringRequired
Allowed values: trtllm vllm sglang
isl
integerRequired

Input sequence length for the sweep. Must be greater than 0.

osl
integerRequired

Output sequence length for the sweep. Must be greater than 0.

sweep_max_context_length
integerRequired

Maximum context length swept. Must be greater than 0.

prefill_interpolation_granularity
integerRequired

Number of prefill interpolation points. Must be greater than 0.

decode_interpolation_granularity
integerRequired

Number of decode interpolation points. Must be greater than 0.

prefill_pick
PickedParallelConfigRequired

Prefill parallelism pick. See PickedParallelConfig.

decode_pick
PickedParallelConfigRequired

Decode parallelism pick. See PickedParallelConfig.

ExternalPluginEntry

One static external-plugin registration under features.planner.scheduling.external_plugins. Rejects unknown fields.

plugin_id
stringRequired

Unique identifier. Must not collide with a builtin plugin id.

plugin_type
stringRequired

Pipeline stage the plugin participates in.

Allowed values: predict propose reconcile constrain
priority
integerRequired

Stage priority. Smaller number is more authoritative.

endpoint
stringRequired

Wire endpoint. Must start with grpc://host:port; inproc:// is rejected here.

auth_token
string

Bearer token validated by the registry. Populate from a mounted Secret.

protocol_version
stringDefaults to 1.0

Plugin protocol version. Must fall within the registry’s supported range.

version
stringDefaults to v1

Plugin’s own version string, surfaced in ListPlugins.

execution_interval_seconds
numberDefaults to 0.0

0.0 runs every tick; a positive value throttles to every N seconds. Must be 0 or greater.

hold_policy
stringDefaults to HOLD_LAST

Behavior when throttled. HOLD_LAST reuses the cached result; ACCEPT_WHEN_IDLE treats the plugin as no-opinion when not due. Accepts the name (case-insensitive) or its integer value.

Allowed values: HOLD_LAST ACCEPT_WHEN_IDLE
needs
[]stringDefaults to []

Capability list consumed by the type-aware merge.

requires_produced_fields
[]stringDefaults to []

Dot-paths into the pipeline context (for example predictions, observations.traffic) that must be set for the plugin to fire on a tick. Empty means no gating.

observation_window_seconds
numberDefaults to 0.0

Aggregation window for windowed observation types in needs. 0.0 uses scale_interval freshness; a positive value aggregates over the last N seconds and must be an integer multiple of scale_interval_seconds.

InProcessPluginSpec

One in-process plugin under features.planner.plugin_registration.in_process_plugins. Rejects unknown fields.

module
stringRequired

Python module path containing the plugin class.

class
stringRequired

Plugin class name within module.

plugin_id
stringRequired

Unique plugin identifier.

plugin_type
stringRequired
Allowed values: predict propose reconcile constrain
priority
integerRequired

Stage priority. Smaller number is more authoritative.

execution_interval_seconds
numberDefaults to 0.0

0.0 runs every tick; a positive value throttles to every N seconds.

hold_policy
stringDefaults to ACCEPT_WHEN_IDLE

Behavior when throttled.

Allowed values: ACCEPT_WHEN_IDLE HOLD_LAST
needs
[]stringDefaults to []

Capability list consumed by the type-aware merge.

requires_produced_fields
[]stringDefaults to []

Dot-paths into the pipeline context that must be set for the plugin to fire on a tick.

observation_window_seconds
numberDefaults to 0.0

Aggregation window for windowed observation types in needs. Must be 0.0 or a positive integer multiple of scale_interval_seconds.

kwargs
objectDefaults to {}

Keyword arguments passed to the plugin class constructor.

Lifecycle

When you create a DGDR, it progresses through these phases (status.phase):

PhaseWhat is happening
PendingSpec validated; operator is discovering GPU hardware and preparing the profiling job
ProfilingProfiling job running — sub-phases in status.profilingPhase: Initializing, SweepingPrefill, SweepingDecode, SelectingConfig, BuildingCurves, GeneratingDGD, Done
ReadyProfiling complete; optimal config stored in status.profilingResults.selectedConfig. Waits for manual application or for autoApply to be enabled.
DeployingCreating the DGD (only when autoApply: true)
DeployedDGD is running and healthy
FailedUnrecoverable error — profiling failures are not retried (backoffLimit: 0); check events and conditions for details

Conditions

The operator maintains these conditions on status.conditions:

ConditionMeaning
ValidationSpec validation passed or failed
ProfilingProfiling job is running, succeeded, or failed
SpecGeneratedGenerated DGD spec is available
DeploymentReadyDGD is deployed and healthy
SucceededAggregate condition — true when the DGDR has reached its target state

Monitoring

# Watch phase transitions
kubectl get dgdr my-model -n $NAMESPACE -w
# Detailed status, conditions, and events
kubectl describe dgdr my-model -n $NAMESPACE
# Profiling sub-phase
kubectl get dgdr my-model -n $NAMESPACE -o jsonpath='{.status.profilingPhase}'
# Profiling job logs
PROFILING_JOB=$(kubectl get dgdr my-model -n "$NAMESPACE" -o jsonpath='{.status.profilingJobName}')
kubectl logs -f "job/${PROFILING_JOB}" -c profiler -n "$NAMESPACE"
# View generated DGD spec (when autoApply: false)
kubectl get dgdr my-model -n $NAMESPACE \
-o jsonpath='{.status.profilingResults.selectedConfig}' | python3 -m json.tool

Resource ownership

  • The DGDR does not set an owner reference on the DGD it creates. Deleting a DGDR does not delete the DGD — it persists independently so it can continue serving traffic.
  • The relationship is tracked via labels: dgdr.nvidia.com/name and dgdr.nvidia.com/namespace.
  • Additional resources (planner ConfigMaps) are created in the same namespace and labeled with dgdr.nvidia.com/name.

Replay a recorded trace (v1beta2)

The nvidia.com/v1beta2 API is experimental and disabled by default. See the installation guide before you enable it.

A native v1beta2 request can evaluate every candidate against a recorded request trace instead of synthetic traffic. Set spec.workload.trace instead of spec.workload.static; the two are mutually exclusive, and admission adds no static default when trace is set. To switch an existing request from static traffic to a trace, set spec.workload.static to null in the same update.

apiVersion: nvidia.com/v1beta2
kind: DynamoGraphDeploymentRequest
metadata:
name: qwen3-trace-replay
spec:
model:
name: Qwen/Qwen3-0.6B
workload:
trace:
# format defaults to Dynamo
source:
persistentVolumeClaim:
claimName: request-traces # PersistentVolumeClaim in this namespace
subPath: windows/2026-10-27T10-00.jsonl.gz
replay:
arrivalSpeedupRatio: 1.0 # optional; 2.0 replays arrivals twice as fast

Before you create the request:

  • Create the PersistentVolumeClaim in the request’s namespace. Use a filesystem volume, not volumeMode: Block. When the process that writes the trace keeps running, use ReadWriteMany. With ReadWriteOnce, a Job scheduled to another node than the writer waits in ContainerCreating, and with ReadWriteOncePod, a Job waits in Pending until the writer’s pod is gone. The Run reports Running while its Job waits.
  • Finish writing the file. subPath names one file relative to the claim root; directories and globs are not expanded. Every Run of the request reads the file when its Job starts, so do not rewrite it afterwards. To replay several rolled jsonl_gz segments, concatenate them into one .jsonl.gz file. A relative symlink to another file in the claim works. An absolute symlink does not, because the kubelet resolves it on the node rather than inside the claim, and neither does a symlink that leaves the claim.
  • Make the file readable by others (for example, mode 0644 in 0755 directories) or by group 65532, which the Job’s pod has as a supplemental group. The pod sets no fsGroup, so neither the kubelet nor a CSI driver changes the ownership or permissions of anything on the claim.

The operator mounts only this file, read-only and only into the AISimulate container, through a subPath of the claim at /var/run/aisimulate/trace/trace.jsonl.gz, or at /var/run/aisimulate/trace/trace.jsonl when subPath does not end in .gz. The kubelet resolves subPath inside the claim and refuses to mount a file outside it. If the file does not exist when the Job starts, the kubelet cannot mount it either, and the AISimulate container waits; see Trace claim lifecycle. On a volume type whose node directory the kubelet can write, such as hostPath, the kubelet instead creates an empty directory at subPath on the claim, and the Run fails when AISimulate cannot read it.

format defaults to Dynamo, the only supported format. It reads the Dynamo request trace (dynamo.request.trace.v1) written by the frontend, as JSON Lines or as gzip-compressed JSON Lines when the file name ends in .gz. See the Request Trace Reference.

The operator lowers the trace to the AISimulate input traffic.source (type: trace, format: dynamo, and paths with the mounted file) and traffic.load.type: trace_timestamps, with traffic.load.speedup when arrivalSpeedupRatio is set. It adds no traffic.stop.requests. These values are an underlay, like every value lowered from a typed field: spec.aisim.inputOverrides is applied last as a JSON Merge Patch and can replace any of them, including traffic.source.type, traffic.source.format, and traffic.source.paths. A null override deletes the named field or parent object, and the operator does not restore it. For example, this override deletes the lowered source:

spec:
aisim:
inputOverrides:
traffic:
source: null

The typed persistentVolumeClaim source alone decides which claim the Run’s Job mounts and which claim the Run protects. Overriding traffic.source.paths adds no volume mount. AISimulate validates the resulting input when the Run’s Job starts, and an input it rejects fails the Run; admission does not validate it against the AISimulate schema yet.

Trace claim lifecycle

The operator ties the claim’s lifetime to the Runs that read it. It decides from structured state only: the claim’s deletionTimestamp, volumeMode, and phase, the Job’s conditions, and the request’s spec. It copies container status messages into the Run but never interprets them.

  • Protection. Before a Run creates its Job, the operator adds the finalizer nvidia.com/dgdr-run-<run-uid> to the claim, where <run-uid> is the Run’s metadata.uid. Every Run that reads the claim adds its own finalizer, so Runs of several requests can share a claim. The operator removes a Run’s finalizer once the Run is complete and none of its Jobs is still running, or once the Run and its Jobs are deleted, for example when the request retires old Runs or is deleted. Kubernetes garbage collection deletes the Job of a deleted Run before the Job’s pod has stopped, so the finalizer of a Run deleted before it completed can be removed while that pod still mounts the file; kubernetes.io/pvc-protection keeps the claim until the pod is gone.
  • Deleting the claim cancels its Runs. When the claim has a deletionTimestamp, the operator stops the Job of every Run that reads it through foreground deletion, whether or not the Job’s pod has started. Once the Job and its pod are gone, the Run fails with reason TraceSourceDeleted, and the operator then removes the Run’s finalizer. A Job that already completed or failed before the claim’s deletion, and is not itself being deleted, is not stopped; its own outcome decides the Run. Once the operator has started stopping a Job, the Run fails with TraceSourceDeleted even if the Job is marked failed while its pod is removed. A Run whose claim is already being deleted before it creates its Job fails with TraceSourceDeleted without creating anything.
  • A missing claim keeps the Run open. When the claim does not exist, the Run stays Pending with Completed=False and reason TraceSourceMissing, and starts once you create the claim.
  • A claim no Job can mount fails the Run before it starts. Before the Run creates its Job, a volumeMode: Block claim or a claim in phase Lost fails the Run with reason TraceSourceUnavailable. After the Job exists, the operator does not check the claim’s volume mode or phase again; the Job’s pod waits until the volume is usable, the Job fails, or you cancel the Run.
  • A waiting container is a diagnostic, not a failure. While a container of the Run’s Job waits on an error, such as a trace file the kubelet cannot mount (a missing file, an absolute symlink, or a symlink that leaves the claim) or an image it cannot pull, the Run stays open with Completed=False, reason ContainerWaiting, and the message container "<name>" is waiting: <reason>: <message>, where <reason> and <message> are copied from the container’s status. The operator does not delete the Job, and there is no timeout. The kubelet keeps retrying the container, so fixing the cause, for example writing the missing file, lets the same Run continue. Waiting in ContainerCreating or PodInitializing is not reported: volume errors that keep a pod there, such as a ReadWriteOnce claim attached to another node, appear only in the pod’s events.

To stop a Run that cannot continue, cancel it explicitly: change the request’s spec (the Run fails with reason Superseded), delete the request, or delete the claim (TraceSourceDeleted). A Pending claim, such as one that waits for its first consumer, is accepted. The request starts one Run per metadata.generation, so after a Run failed, change the request’s spec to start a new Run.

# Active and recent Runs of the request
kubectl get dynamographdeploymentrequests.v1beta2.nvidia.com qwen3-trace-replay -n "$NAMESPACE" \
-o jsonpath='{.status.activeRunRef}{"\n"}{.status.recentRunRefs}{"\n"}'
# Why a Run failed or waits
kubectl get dynamographdeploymentrun qwen3-trace-replay-1 -n "$NAMESPACE" \
-o jsonpath='{.status.conditions}'

Kubernetes keeps a deleted claim Terminating while any pod that mounts it exists (kubernetes.io/pvc-protection), including the completed pods of finished Jobs. The operator keeps the Runs, Jobs, and pods of a request’s five most recent finished Runs, so a deleted trace claim can stay Terminating after every Run released it, until those Runs are retired or the request is deleted.

If the operator is uninstalled, or the v1beta2 API is disabled, while Runs are in flight, their finalizers stay on the claim and keep it from being deleted. List them, then remove only the nvidia.com/dgdr-run- finalizers (requires jq):

kubectl get pvc request-traces -n "$NAMESPACE" -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl get pvc request-traces -n "$NAMESPACE" -o json \
| jq '.metadata.finalizers |= map(select(startswith("nvidia.com/dgdr-run-") | not))' \
| kubectl replace -f -

Known issues

  • PCIe profiler data not yet available. See the PCIe callout under SKU format.