Operator Metrics

Kubernetes-only dynamo_operator_* metrics and the annotations and Helm values that enable metrics collection.

View as Markdown

The Dynamo Operator exposes its own Prometheus metrics for controller reconciliation, webhook validation, and resource inventory. These are Kubernetes-scoped — they exist only where the operator runs, and are separate from the application metrics emitted by frontends and workers (see Metrics Catalog). For enablement and dashboards (including the operator dashboard), see the Kubernetes observability guide.

All operator metrics use the dynamo_operator prefix. Unlike application metrics (which use a PodMonitor), the operator is scraped via a ServiceMonitor created by the Helm chart.

Reconciliation metrics

Track how efficiently controllers process DynamoGraphDeployment, DynamoComponentDeployment, DynamoModel, DynamoGraphDeploymentRequest, and DynamoGraphDeploymentScalingAdapter resources.

dynamo_operator_reconcile_duration_seconds
histogram

Duration of reconciliation loops. Labeled by resource_type, namespace, and result.

dynamo_operator_reconcile_total
counter

Total number of reconciliations. Labeled by resource_type, namespace, and result.

dynamo_operator_reconcile_errors_total
counter

Total reconciliation errors by type. Labeled by resource_type, namespace, and error_type.

Webhook metrics

Track performance and outcomes of admission webhook requests.

dynamo_operator_webhook_duration_seconds
histogram

Duration of webhook validation requests. Labeled by resource_type and operation.

dynamo_operator_webhook_requests_total
counter

Total webhook admission requests. Labeled by resource_type, operation, and result.

dynamo_operator_webhook_denials_total
counter

Total webhook denials with reasons. Labeled by resource_type, operation, and reason.

Resource inventory metrics

dynamo_operator_resources_total
gauge

Current count of managed resources by state. Labeled by resource_type, namespace, and status.

Operator metric labels

These labels are specific to operator metrics and are distinct from the application Metric Labels.

resource_type
string

The managed CRD: DynamoGraphDeployment, DynamoComponentDeployment, DynamoModel, DynamoGraphDeploymentRequest, or DynamoGraphDeploymentScalingAdapter.

namespace
string

Target Kubernetes namespace of the resource.

result
string

Outcome of the operation. On reconcile metrics: success, error, requeue. On webhook metrics: allowed, denied.

error_type
string

On dynamo_operator_reconcile_errors_total — the failure category.

Allowed values: not_found already_exists conflict validation bad_request unauthorized forbidden timeout server_timeout unavailable rate_limited internal
operation
string

On webhook metrics — the admission operation.

Allowed values: CREATE UPDATE DELETE
reason
string

On dynamo_operator_webhook_denials_total — the validation failure reason, e.g. immutable_field_changed, invalid_config.

status
string

On dynamo_operator_resources_total — resource state derived from each CRD’s status. Common values: ready, not_ready, unknown (DCD, DM, DGDSA). DGD uses pending, successful, failed from .status.state. DGDR uses Pending, Profiling, Ready, Deploying, Deployed, Failed from .status.phase.

Kubernetes enablement knobs

Metrics collection on Kubernetes is controlled by CRD annotations and Helm values — not environment variables. These knobs govern whether application and operator metrics are scraped at all.

These fields also appear in the CRD and Helm references. They are reproduced here so the observability enablement path is documented in one place. For the operator’s full Helm value set, see the Dynamo Operator guide.

nvidia.com/metrics-enabled
boolean (pod label)Defaults to true

Pod label the operator adds to every managed pod so the PodMonitor discovers and scrapes it. Present automatically; you do not set it by hand.

nvidia.com/enable-metrics
boolean (DGD annotation)Defaults to true

Annotation on a DynamoGraphDeployment to opt a deployment out of metrics collection. Set to false to disable scraping for that deployment.

dynamo-operator.metricsService.enabled
boolean (Helm value)Defaults to true

Helm value controlling whether the operator’s metrics Service and ServiceMonitor are created. Set to false to disable operator metrics collection.

dynamo-operator.dynamo.metrics.prometheusEndpoint
string (Helm value)

Helm value pointing the operator at your Prometheus endpoint, e.g. http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090.

To disable operator metrics collection:

$helm upgrade dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \
> --namespace dynamo-system \
> --set dynamo-operator.metricsService.enabled=false

To opt a single deployment out of application metrics:

1apiVersion: nvidia.com/v1
2kind: DynamoGraphDeployment
3metadata:
4 name: my-deployment
5 annotations:
6 nvidia.com/enable-metrics: "false"
7spec:
8 # …