⚠️ Important: This documentation is automatically generated from source code. Do not edit this file directly.

API Reference#

Packages#

nvidia.com/v1alpha1#

Package v1alpha1 contains API Schema definitions for the nvidia.com v1alpha1 API group.

Resource Types#

Autoscaling#

Appears in:

Field

Description

Default

Validation

enabled boolean

minReplicas integer

maxReplicas integer

behavior HorizontalPodAutoscalerBehavior

metrics MetricSpec array

DynamoComponentDeployment#

DynamoComponentDeployment is the Schema for the dynamocomponentdeployments API

Field

Description

Default

Validation

apiVersion string

nvidia.com/v1alpha1

kind string

DynamoComponentDeployment

metadata ObjectMeta

Refer to Kubernetes API documentation for fields of metadata.

spec DynamoComponentDeploymentSpec

Spec defines the desired state for this Dynamo component deployment.

DynamoComponentDeploymentSharedSpec#

Appears in:

Field

Description

Default

Validation

annotations object (keys:string, values:string)

Annotations to add to generated Kubernetes resources for this component
(such as Pod, Service, and Ingress when applicable).

labels object (keys:string, values:string)

Labels to add to generated Kubernetes resources for this component.

serviceName string

The name of the component

componentType string

ComponentType indicates the role of this component (for example, “main”).

subComponentType string

SubComponentType indicates the sub-role of this component (for example, “prefill”).

dynamoNamespace string

Dynamo namespace of the service (allows to override the Dynamo namespace of the service defined in annotations inside the Dynamo archive)

resources Resources

Resources requested and limits for this component, including CPU, memory,
GPUs/devices, and any runtime-specific resources.

autoscaling Autoscaling

Autoscaling config for this component (replica range, target utilization, etc.).

envs EnvVar array

Envs defines additional environment variables to inject into the component containers.

envFromSecret string

EnvFromSecret references a Secret whose key/value pairs will be exposed as
environment variables in the component containers.

volumeMounts VolumeMount array

VolumeMounts references PVCs defined at the top level for volumes to be mounted by the component.

ingress IngressSpec

Ingress config to expose the component outside the cluster (or through a service mesh).

sharedMemory SharedMemorySpec

SharedMemory controls the tmpfs mounted at /dev/shm (enable/disable and size).

extraPodMetadata ExtraPodMetadata

ExtraPodMetadata adds labels/annotations to the created Pods.

extraPodSpec ExtraPodSpec

ExtraPodSpec allows to override the main pod spec configuration.
It is a k8s standard PodSpec. It also contains a MainContainer (standard k8s Container) field
that allows overriding the main container configuration.

livenessProbe Probe

LivenessProbe to detect and restart unhealthy containers.

readinessProbe Probe

ReadinessProbe to signal when the container is ready to receive traffic.

replicas integer

Replicas is the desired number of Pods for this component when autoscaling is not used.

multinode MultinodeSpec

Multinode is the configuration for multinode components.

DynamoComponentDeploymentSpec#

DynamoComponentDeploymentSpec defines the desired state of DynamoComponentDeployment

Appears in:

Field

Description

Default

Validation

backendFramework string

BackendFramework specifies the backend framework (e.g., “sglang”, “vllm”, “trtllm”)

Enum: [sglang vllm trtllm]

annotations object (keys:string, values:string)

Annotations to add to generated Kubernetes resources for this component
(such as Pod, Service, and Ingress when applicable).

labels object (keys:string, values:string)

Labels to add to generated Kubernetes resources for this component.

serviceName string

The name of the component

componentType string

ComponentType indicates the role of this component (for example, “main”).

subComponentType string

SubComponentType indicates the sub-role of this component (for example, “prefill”).

dynamoNamespace string

Dynamo namespace of the service (allows to override the Dynamo namespace of the service defined in annotations inside the Dynamo archive)

resources Resources

Resources requested and limits for this component, including CPU, memory,
GPUs/devices, and any runtime-specific resources.

autoscaling Autoscaling

Autoscaling config for this component (replica range, target utilization, etc.).

envs EnvVar array

Envs defines additional environment variables to inject into the component containers.

envFromSecret string

EnvFromSecret references a Secret whose key/value pairs will be exposed as
environment variables in the component containers.

volumeMounts VolumeMount array

VolumeMounts references PVCs defined at the top level for volumes to be mounted by the component.

ingress IngressSpec

Ingress config to expose the component outside the cluster (or through a service mesh).

sharedMemory SharedMemorySpec

SharedMemory controls the tmpfs mounted at /dev/shm (enable/disable and size).

extraPodMetadata ExtraPodMetadata

ExtraPodMetadata adds labels/annotations to the created Pods.

extraPodSpec ExtraPodSpec

ExtraPodSpec allows to override the main pod spec configuration.
It is a k8s standard PodSpec. It also contains a MainContainer (standard k8s Container) field
that allows overriding the main container configuration.

livenessProbe Probe

LivenessProbe to detect and restart unhealthy containers.

readinessProbe Probe

ReadinessProbe to signal when the container is ready to receive traffic.

replicas integer

Replicas is the desired number of Pods for this component when autoscaling is not used.

multinode MultinodeSpec

Multinode is the configuration for multinode components.

DynamoGraphDeployment#

DynamoGraphDeployment is the Schema for the dynamographdeployments API.

Field

Description

Default

Validation

apiVersion string

nvidia.com/v1alpha1

kind string

DynamoGraphDeployment

metadata ObjectMeta

Refer to Kubernetes API documentation for fields of metadata.

spec DynamoGraphDeploymentSpec

Spec defines the desired state for this graph deployment.

status DynamoGraphDeploymentStatus

Status reflects the current observed state of this graph deployment.

DynamoGraphDeploymentSpec#

DynamoGraphDeploymentSpec defines the desired state of DynamoGraphDeployment.

Appears in:

Field

Description

Default

Validation

pvcs PVC array

PVCs defines a list of persistent volume claims that can be referenced by components.
Each PVC must have a unique name that can be referenced in component specifications.

Optional: {}

services object (keys:string, values:DynamoComponentDeploymentSharedSpec)

Services are the services to deploy as part of this deployment.

Optional: {}

envs EnvVar array

Envs are environment variables applied to all services in the deployment unless
overridden by service-specific configuration.

Optional: {}

backendFramework string

BackendFramework specifies the backend framework (e.g., “sglang”, “vllm”, “trtllm”).

Enum: [sglang vllm trtllm]

DynamoGraphDeploymentStatus#

DynamoGraphDeploymentStatus defines the observed state of DynamoGraphDeployment.

Appears in:

Field

Description

Default

Validation

state string

State is a high-level textual status of the graph deployment lifecycle.

conditions Condition array

Conditions contains the latest observed conditions of the graph deployment.
The slice is merged by type on patch updates.

IngressSpec#

Appears in:

Field

Description

Default

Validation

enabled boolean

Enabled exposes the component through an ingress or virtual service when true.

host string

Host is the base host name to route external traffic to this component.

useVirtualService boolean

UseVirtualService indicates whether to configure a service-mesh VirtualService instead of a standard Ingress.

virtualServiceGateway string

VirtualServiceGateway optionally specifies the gateway name to attach the VirtualService to.

hostPrefix string

HostPrefix is an optional prefix added before the host.

annotations object (keys:string, values:string)

Annotations to set on the generated Ingress/VirtualService resources.

labels object (keys:string, values:string)

Labels to set on the generated Ingress/VirtualService resources.

tls IngressTLSSpec

TLS holds the TLS configuration used by the Ingress/VirtualService.

hostSuffix string

HostSuffix is an optional suffix appended after the host.

ingressControllerClassName string

IngressControllerClassName selects the ingress controller class (e.g., “nginx”).

IngressTLSSpec#

Appears in:

Field

Description

Default

Validation

secretName string

SecretName is the name of a Kubernetes Secret containing the TLS certificate and key.

MultinodeSpec#

Appears in:

Field

Description

Default

Validation

nodeCount integer

Indicates the number of nodes to deploy for multinode components.
Total number of GPUs is NumberOfNodes * GPU limit.
Must be greater than 1.

2

Minimum: 2

PVC#

Appears in:

Field

Description

Default

Validation

create boolean

Create indicates to create a new PVC

name string

Name is the name of the PVC

Required: {}

storageClass string

StorageClass to be used for PVC creation. Required when create is true.

size Quantity

Size of the volume in Gi, used during PVC creation. Required when create is true.

volumeAccessMode PersistentVolumeAccessMode

VolumeAccessMode is the volume access mode of the PVC. Required when create is true.

SharedMemorySpec#

Appears in:

Field

Description

Default

Validation

disabled boolean

size Quantity

VolumeMount#

VolumeMount references a PVC defined at the top level for volumes to be mounted by the component

Appears in:

Field

Description

Default

Validation

name string

Name references a PVC name defined in the top-level PVCs map

Required: {}

mountPoint string

MountPoint specifies where to mount the volume.
If useAsCompilationCache is true and mountPoint is not specified,
a backend-specific default will be used.

useAsCompilationCache boolean

UseAsCompilationCache indicates this volume should be used as a compilation cache.
When true, backend-specific environment variables will be set and default mount points may be used.

false

Operator Default Values Injection#

The Dynamo operator automatically applies default values to various fields when they are not explicitly specified in your deployments. These defaults include:

  • Health Probes: Startup, liveness, and readiness probes are configured differently for frontend, worker, and planner components. For example, worker components receive a startup probe with a 2-hour timeout (720 failures × 10 seconds) to accommodate long model loading times.

  • Shared Memory: All components receive an 8Gi shared memory volume mounted at /dev/shm by default (can be disabled or resized via the sharedMemory field).

  • Environment Variables: Components automatically receive environment variables like DYN_NAMESPACE, DYN_PARENT_DGD_K8S_NAME, DYNAMO_PORT, and backend-specific variables.

  • Pod Configuration: Default terminationGracePeriodSeconds of 60 seconds and restartPolicy: Always.

  • Autoscaling: When enabled without explicit metrics, defaults to CPU-based autoscaling with 80% target utilization.

  • Backend-Specific Behavior: For multinode deployments, probes are automatically modified or removed for worker nodes depending on the backend framework (VLLM, SGLang, or TensorRT-LLM).

Pod Specification Defaults#

All components receive the following pod-level defaults unless overridden:

  • terminationGracePeriodSeconds: 60 seconds

  • restartPolicy: Always

Shared Memory Configuration#

Shared memory is enabled by default for all components:

  • Enabled: true (unless explicitly disabled via sharedMemory.disabled)

  • Size: 8Gi

  • Mount Path: /dev/shm

  • Volume Type: emptyDir with memory medium

To disable shared memory or customize the size, use the sharedMemory field in your component specification.

Health Probes by Component Type#

The operator applies different default health probes based on the component type.

Frontend Components#

Frontend components receive the following probe configurations:

Liveness Probe:

  • Type: HTTP GET

  • Path: /health

  • Port: http (8000)

  • Initial Delay: 60 seconds

  • Period: 60 seconds

  • Timeout: 30 seconds

  • Failure Threshold: 10

Readiness Probe:

  • Type: Exec command

  • Command: curl -s http://localhost:${DYNAMO_PORT}/health | jq -e ".status == \"healthy\""

  • Initial Delay: 60 seconds

  • Period: 60 seconds

  • Timeout: 30 seconds

  • Failure Threshold: 10

Worker Components#

Worker components receive the following probe configurations:

Liveness Probe:

  • Type: HTTP GET

  • Path: /live

  • Port: system (9090)

  • Period: 5 seconds

  • Timeout: 30 seconds

  • Failure Threshold: 1

Readiness Probe:

  • Type: HTTP GET

  • Path: /health

  • Port: system (9090)

  • Period: 10 seconds

  • Timeout: 30 seconds

  • Failure Threshold: 60

Startup Probe:

  • Type: HTTP GET

  • Path: /live

  • Port: system (9090)

  • Period: 10 seconds

  • Timeout: 5 seconds

  • Failure Threshold: 720 (allows up to 2 hours for startup: 10s × 720 = 7200s)

Note

For larger models (typically >70B parameters) or slower storage systems, you may need to increase the failureThreshold to allow more time for model loading. Calculate the required threshold based on your expected startup time: failureThreshold = (expected_startup_seconds / period). Override the startup probe in your component specification if the default 2-hour window is insufficient.

Multinode Deployment Probe Modifications#

For multinode deployments, the operator modifies probes based on the backend framework and node role:

VLLM Backend#

  • Worker nodes: All probes (liveness, readiness, startup) are removed

SGLang Backend#

  • Worker nodes: All probes (liveness, readiness, startup) are removed

TensorRT-LLM Backend#

  • Leader nodes: All probes remain unchanged

  • Worker nodes:

    • Liveness and startup probes are removed

    • Readiness probe is replaced with a TCP socket check on SSH port (2222):

      • Initial Delay: 20 seconds

      • Period: 20 seconds

      • Timeout: 5 seconds

      • Failure Threshold: 10

Environment Variables#

The operator automatically injects environment variables based on component type and configuration:

All Components#

  • DYN_NAMESPACE: The Dynamo namespace for the component

  • DYN_PARENT_DGD_K8S_NAME: The parent DynamoGraphDeployment Kubernetes resource name

  • DYN_PARENT_DGD_K8S_NAMESPACE: The parent DynamoGraphDeployment Kubernetes namespace

Frontend Components#

  • DYNAMO_PORT: 8000

  • DYN_HTTP_PORT: 8000

Worker Components#

  • DYN_SYSTEM_ENABLED: true

  • DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS: ["generate"]

  • DYN_SYSTEM_PORT: 9090

Planner Components#

  • PLANNER_PROMETHEUS_PORT: 9085

VLLM Backend (with compilation cache)#

When a volume mount is configured with useAsCompilationCache: true:

  • VLLM_CACHE_ROOT: Set to the mount point of the cache volume

Service Account#

Planner components automatically receive the following service account:

  • serviceAccountName: planner-serviceaccount

Image Pull Secrets#

The operator automatically discovers and injects image pull secrets for container images. When a component specifies a container image, the operator:

  1. Scans all Kubernetes secrets of type kubernetes.io/dockerconfigjson in the component’s namespace

  2. Extracts the docker registry server URLs from each secret’s authentication configuration

  3. Matches the container image’s registry host against the discovered registry URLs

  4. Automatically injects matching secrets as imagePullSecrets in the pod specification

This eliminates the need to manually specify image pull secrets for each component. The operator maintains an internal index of docker secrets and their associated registries, refreshing this index periodically.

To disable automatic image pull secret discovery for a specific component, add the following annotation:

annotations:
  nvidia.com/disable-image-pull-secret-discovery: "true"

Autoscaling Defaults#

When autoscaling is enabled but no metrics are specified, the operator applies:

  • Default Metric: CPU utilization

  • Target Average Utilization: 80%

Port Configurations#

Default container ports are configured based on component type:

Frontend Components#

  • Port: 8000

  • Protocol: TCP

  • Name: http

Worker Components#

  • Port: 9090

  • Protocol: TCP

  • Name: system

Planner Components#

  • Port: 9085

  • Protocol: TCP

  • Name: metrics

Backend-Specific Configurations#

VLLM#

  • Ray Head Port: 6379 (for multinode deployments)

SGLang#

  • Distribution Init Port: 29500 (for multinode deployments)

TensorRT-LLM#

  • SSH Port: 2222 (for multinode MPI communication)

  • OpenMPI Environment: OMPI_MCA_orte_keep_fqdn_hostnames=1

Implementation Reference#

For users who want to understand the implementation details or contribute to the operator, the default values described in this document are set in the following source files:

Notes#

  • All these defaults can be overridden by explicitly specifying values in your DynamoComponentDeployment or DynamoGraphDeployment resources

  • User-specified probes (via livenessProbe, readinessProbe, or startupProbe fields) take precedence over operator defaults

  • For multinode deployments, some defaults are modified or removed as described above to accommodate distributed execution patterns

  • The extraPodSpec.mainContainer field can be used to override probe configurations set by the operator