Webhooks

This document describes the webhook functionality in the Dynamo Operator, including validation webhooks, certificate management, and troubleshooting.

Overview

The Dynamo Operator uses Kubernetes admission webhooks to provide real-time validation and mutation of custom resources. Currently, the operator implements validation webhooks that ensure invalid configurations are rejected immediately at the API server level, providing faster feedback to users compared to controller-based validation.

All webhook types (validating, mutating, conversion, etc.) share the same webhook server and TLS certificate infrastructure, making certificate management consistent across all webhook operations.

Key Features

• ✅ Always enabled - Webhooks are a required component of the operator
• ✅ Shared certificate infrastructure - All webhook types use the same TLS certificates
• ✅ Automatic certificate generation and rotation - Built-in cert-controller, no manual management required
• ✅ cert-manager integration - Optional integration for custom PKI or organizational certificate policies
• ✅ Immutability enforcement - Critical fields protected via CEL validation rules

Current Webhook Types

• Validating Webhooks: Validate custom resource specifications before persistence
  • DynamoComponentDeployment validation
  • DynamoGraphDeployment validation
  • DynamoModel validation
  • DynamoGraphDeploymentRequest validation
• Mutating Webhooks: Apply default values to resources on creation
  • DynamoGraphDeployment defaulting

Note: All webhook types use the same certificate infrastructure described in this document.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         API Server                               │
│  1. User submits CR (kubectl apply)                             │
│  2. API server calls MutatingWebhookConfiguration               │
└────────────────────────┬────────────────────────────────────────┘
                         │ HTTPS (TLS required)
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Webhook Server (in Operator Pod)                │
│  3. Applies defaults (e.g., operator version annotation)        │
│  4. Returns mutated CR                                          │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                         API Server                               │
│  5. API server calls ValidatingWebhookConfiguration             │
└────────────────────────┬────────────────────────────────────────┘
                         │ HTTPS (TLS required)
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Webhook Server (in Operator Pod)                │
│  6. Validates CR against business rules                         │
│  7. Returns admit/deny decision + warnings                      │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                      API Server                                  │
│  8. If admitted: Persist CR to etcd                             │
│  9. If denied: Return error to user                             │
└─────────────────────────────────────────────────────────────────┘

Admission Flow

1. Mutating webhooks: Apply defaults and transformations before validation
2. Validating webhooks: Validate the (possibly mutated) CR against business rules
3. CEL validation: Kubernetes-native immutability checks (always active)

Upgrading from versions with webhook.enabled: false

The webhook.enabled Helm value has been removed. Webhooks are now a required component of the operator and are always active. If you previously ran with webhook.enabled: false, take the following steps before upgrading:

1. Remove webhook.enabled from any custom values files. Helm will ignore the unknown key, but it should be cleaned up to avoid confusion.
2. Ensure port 9443 is reachable from the Kubernetes API server to the operator pod. If you have NetworkPolicy rules or firewall configurations restricting traffic, add an ingress rule allowing the API server to reach the webhook server on port 9443.
3. Ensure webhook TLS certificates are available. By default, the operator’s built-in cert-controller generates and rotates self-signed certificates automatically at startup — no action needed. If you use cert-manager or externally managed certificates, verify your configuration is in place before upgrading.

Configuration

Certificate Management Options

The operator supports three certificate management modes:

Advanced Configuration

Complete Configuration Reference

1
dynamo-operator:
2
  webhook:
3
    # Certificate management (optional, to use cert-manager instead of built-in)
4
    certManager:
5
      enabled: false
6
      issuerRef:
7
        kind: Issuer
8
        name: selfsigned-issuer
9
10
    # Certificate secret configuration
11
    certificateSecret:
12
      name: webhook-server-cert
13
      external: false           # Set to true for externally managed certificates
14
15
    # Webhook behavior configuration
16
    failurePolicy: Fail        # Fail (reject on error) or Ignore (allow on error)
17
    timeoutSeconds: 10         # Webhook timeout
18
19
    # User-configured namespace filtering is unsupported
20
    namespaceSelector: {}

Failure Policy

1
# Fail: Reject resources if webhook is unavailable (recommended for production)
2
webhook:
3
  failurePolicy: Fail
4
5
# Ignore: Allow resources if webhook is unavailable (use with caution)
6
webhook:
7
  failurePolicy: Ignore

Recommendation: Use Fail in production to ensure validation is always enforced. Only use Ignore if you need high availability and can tolerate occasional invalid resources.

Namespace Filtering

User-configured webhook namespace filtering is not supported. Helm rejects a non-empty webhook.namespaceSelector value. The cluster-wide webhook configurations cover every namespace, while namespace-restricted webhook configurations use a Helm-generated selector for their target namespace. The cluster-wide handlers skip namespaces claimed by an active ownership Lease.

Certificate Management

Automatic Certificates (Default)

Zero configuration required! The operator’s built-in cert-controller generates and rotates certificates automatically at startup.

How It Works

1. Operator starts: The CertManager checks for an existing certificate Secret (configured via webhook.certificateSecret.name, default: webhook-server-cert). If missing or invalid, it generates a self-signed Root CA and server certificate and writes them to the Secret.

2. CA bundle injection: The CABundleInjector reads ca.crt from the Secret and patches both the ValidatingWebhookConfiguration and MutatingWebhookConfiguration with the base64-encoded CA bundle.

3. Certificate rotation: The cert-controller monitors certificate validity and regenerates certificates before they expire.

4. Webhook server starts: The webhook server only begins serving after certificates are confirmed ready, preventing startup races.

Certificate Validity

• Root CA: 10 years
• Server Certificate: 10 years (same as Root CA)
• Automatic rotation: The cert-controller monitors validity and regenerates before expiration

Smart Certificate Management

The cert-controller is intelligent about certificate lifecycle:

• ✅ Checks existing certificates at startup before generating new ones
• ✅ Skips generation if valid certificates already exist in the Secret
• ✅ Regenerates only when needed (missing, expiring soon, or incorrect SANs)

This means:

• Fast operator restarts (no unnecessary cert generation)
• No dependency on Helm hooks or external Jobs
• Certificates persist across pod restarts (stored in Secret)

Manual Certificate Rotation

If you need to rotate certificates manually:

$
# Delete the certificate secret -- the operator will regenerate it on restart
$
kubectl delete secret <release>-webhook-server-cert -n <namespace>
$
$
# Restart the operator pod to trigger regeneration
$
kubectl rollout restart deployment/<release>-dynamo-operator -n <namespace>

cert-manager Integration

For clusters with cert-manager installed, you can enable automated certificate lifecycle management.

Prerequisites

1. cert-manager installed (v1.0+)
2. CA issuer configured (e.g., selfsigned-issuer)

Configuration

1
dynamo-operator:
2
  webhook:
3
    certManager:
4
      enabled: true
5
      issuerRef:
6
        kind: Issuer              # Or ClusterIssuer
7
        name: selfsigned-issuer   # Your issuer name

How It Works

1. Helm creates Certificate resource: Requests TLS certificate from cert-manager
2. cert-manager generates certificate: Based on configured issuer
3. cert-manager stores in Secret: <release>-webhook-server-cert
4. cert-manager ca-injector: Automatically injects CA bundle into ValidatingWebhookConfiguration
5. Operator pod: Mounts certificate secret and serves webhook

When to Use cert-manager

• ✅ Custom validity periods: Configure certificate lifetime to match organizational policy
• ✅ Integration with existing PKI: Use your organization’s certificate infrastructure
• ✅ Centralized certificate management: Manage all cluster certificates through cert-manager

Certificate Rotation

With cert-manager, certificate rotation is fully automated:

1. Leaf certificate rotation (default: every year)

   • cert-manager auto-renews before expiration
   • controller-runtime auto-reloads new certificate
   • No pod restart required
   • No caBundle update required (same Root CA)

2. Root CA rotation (every 10 years)

   • cert-manager rotates Root CA
   • ca-injector auto-updates caBundle in ValidatingWebhookConfiguration
   • No manual intervention required

Example: Self-Signed Issuer

1
apiVersion: cert-manager.io/v1
2
kind: Issuer
3
metadata:
4
  name: selfsigned-issuer
5
  namespace: dynamo-system
6
spec:
7
  selfSigned: {}
8
---
9
# Enable in platform values.yaml
10
dynamo-operator:
11
  webhook:
12
    certManager:
13
      enabled: true
14
      issuerRef:
15
        kind: Issuer
16
        name: selfsigned-issuer

External Certificates

Bring your own certificates for custom PKI requirements.

Steps

1. Create certificate secret manually:

$
kubectl create secret tls <release>-webhook-server-cert \
>
  --cert=tls.crt \
>
  --key=tls.key \
>
  -n <namespace>
$
$
# Also add ca.crt to the secret
$
kubectl patch secret <release>-webhook-server-cert -n <namespace> \
>
  --type='json' \
>
  -p='[{"op": "add", "path": "/data/ca.crt", "value": "'$(base64 -w0 < ca.crt)'"}]'

2. Configure operator to use external secret:

1
dynamo-operator:
2
  webhook:
3
    certificateSecret:
4
      external: true
5
    caBundle: <base64-encoded-ca-cert>  # Must manually specify

3. Deploy operator:

$
helm install dynamo-platform . -n <namespace> -f values.yaml

Certificate Requirements

• Secret name: Must match webhook.certificateSecret.name (default: webhook-server-cert)
• Secret keys: tls.crt, tls.key, ca.crt
• Certificate SAN: Must include <service-name>.<namespace>.svc
  • Example: dynamo-platform-dynamo-operator-webhook-service.dynamo-system.svc

Development and Test Multi-Operator Deployments

The operator can run one cluster-wide instance and namespace-restricted instances using a Lease-based ownership mechanism.

Scenario

Cluster:
├─ Operator A (cluster-wide, namespace: platform-system)
│  └─ Skips reconciliation and admission in team-a
└─ Operator B (namespace-restricted, namespace: team-a)
   └─ Reconciles, validates, and mutates only team-a

How It Works

1. The namespace-restricted operator creates a Lease named dynamo-operator-namespace-scope.
2. The cluster-wide operator skips reconciliation, validation, and mutation for that namespace.
3. The namespace-restricted operator’s webhook configurations select only its target namespace.
4. Each operator manages the certificate and CA bundles for its own admission configurations.
5. Only the cluster-wide operator updates CRDs, conversion service references, and conversion CA bundles.

Deployment Example

$
# 1. Deploy the cluster-wide operator
$
helm install platform-operator dynamo-platform -n platform-system
$
$
# 2. Deploy a namespace-restricted operator for team-a
$
helm install team-a-operator dynamo-platform \
>
  -n team-a \
>
  --set dynamo-operator.namespaceRestriction.enabled=true \
>
  --set dynamo-operator.upgradeCRD=false

ValidatingWebhookConfiguration Naming

The webhook configuration name reflects the deployment mode:

• Cluster-wide: <release>-validating
• Namespace-restricted: <release>-validating-<namespace>

Example:

$
# Cluster-wide
$
platform-operator-validating
$
$
# Namespace-restricted (team-a)
$
team-a-operator-validating-team-a

Helm fixes every webhook in the namespace-restricted configurations to the target namespace. Do not set webhook.namespaceSelector manually.

Lease Health

After you uninstall the namespace-restricted release, its webhook configurations are removed and its Lease expires. Cluster-wide reconciliation and admission then resume for the namespace. If only the operator Pod is unhealthy, its webhook configurations remain and can block admission according to failurePolicy until the release recovers or is uninstalled.

Run the same operator version in parallel whenever possible. The cluster-wide operator should be the same version or newer and must provide the newest APIs. A newer namespaced controller is acceptable for development when it remains compatible with the installed CRDs. Install every operator Helm release in a separate namespace so each release owns a distinct admission certificate and webhook configuration set.

Troubleshooting

Webhook Not Called

Symptoms:

• Invalid resources are accepted
• No validation errors in logs

Checks:

1. Verify webhook configuration exists:

$
kubectl get validatingwebhookconfiguration | grep dynamo

2. Check webhook configuration:

$
kubectl get validatingwebhookconfiguration <name> -o yaml
$
# Verify:
$
# - caBundle is present and non-empty
$
# - clientConfig.service points to correct service
$
# - webhooks[].namespaceSelector is absent for cluster-wide admission or matches the target namespace

3. Verify webhook service exists:

$
kubectl get service -n <namespace> | grep webhook

4. Check operator logs for webhook startup:

$
kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep webhook
$
# Should see: "Registering validation webhooks"
$
# Should see: "Starting webhook server"

Connection Refused Errors

Symptoms:

Error from server (InternalError): Internal error occurred: failed calling webhook:
Post "https://...webhook-service...:443/validate-...": dial tcp ...:443: connect: connection refused

Checks:

1. Verify operator pod is running:

$
kubectl get pods -n <namespace> -l app.kubernetes.io/name=dynamo-operator

2. Check webhook server is listening:

$
# Port-forward to pod
$
kubectl port-forward -n <namespace> pod/<operator-pod> 9443:9443
$
$
# In another terminal, test connection
$
curl -k https://localhost:9443/validate-nvidia-com-v1alpha1-dynamocomponentdeployment
$
# Should NOT get "connection refused"

3. Verify webhook port in deployment:

$
kubectl get deployment -n <namespace> <release>-dynamo-operator -o yaml | grep -A5 "containerPort: 9443"

4. Check for webhook initialization errors:

$
kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep -i error

Certificate Errors

Symptoms:

Error from server (InternalError): Internal error occurred: failed calling webhook:
x509: certificate signed by unknown authority

Checks:

1. Verify caBundle is present:

$
kubectl get validatingwebhookconfiguration <name> -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d
$
# Should output a valid PEM certificate

2. Verify certificate secret exists:

$
kubectl get secret -n <namespace> <release>-webhook-server-cert

3. Check certificate validity:

$
kubectl get secret -n <namespace> <release>-webhook-server-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text
$
# Check:
$
# - Not expired
$
# - SAN includes: <service-name>.<namespace>.svc

4. Check operator logs for CA injection errors:

$
kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep -i "cert\|ca.*bundle\|inject"

Certificate Controller Errors

Symptoms:

• Operator logs show cert-controller errors
• Certificate Secret is not created
• CA bundle is not injected into webhook configurations

Checks:

1. Check cert-controller logs:

$
kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep -i "cert-manager\|cert-rotation\|cert-controller"

2. Verify RBAC permissions:

$
# The operator needs permissions to manage Secrets, ValidatingWebhookConfigurations,
$
# MutatingWebhookConfigurations, and CustomResourceDefinitions
$
kubectl auth can-i create secrets -n <namespace> --as=system:serviceaccount:<namespace>:<release>-dynamo-operator
$
kubectl auth can-i patch validatingwebhookconfigurations --as=system:serviceaccount:<namespace>:<release>-dynamo-operator

3. Check if the certificate Secret was created:

$
kubectl get secret -n <namespace> <release>-webhook-server-cert

4. Force certificate regeneration:

$
# Delete the certificate secret and restart the operator
$
kubectl delete secret <release>-webhook-server-cert -n <namespace>
$
kubectl rollout restart deployment/<release>-dynamo-operator -n <namespace>

Validation Errors Not Clear

Symptoms:

• Webhook rejects resource but error message is unclear

Solution:

Check operator logs for detailed validation errors:

$
kubectl logs -n <namespace> deployment/<release>-dynamo-operator | grep "validate create\|validate update"

Webhook logs include:

• Resource name and namespace
• Validation errors with context
• Warnings for immutable field changes

Stuck Deleting Resources

Symptoms:

• Resource stuck in “Terminating” state
• Webhook blocks finalizer removal

Solution:

The webhook automatically skips validation for resources being deleted. If stuck:

1. Check if webhook is blocking:

$
kubectl describe <resource-type> <name> -n <namespace>
$
# Look for events mentioning webhook errors

2. Temporarily work around the webhook:

$
# Option 1: Set failurePolicy to Ignore
$
kubectl patch validatingwebhookconfiguration <name> \
>
  --type='json' \
>
  -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'
$
$
# Option 2 (last resort): Delete ValidatingWebhookConfiguration
$
kubectl delete validatingwebhookconfiguration <name>

3. Delete resource again:

$
kubectl delete <resource-type> <name> -n <namespace>

4. Restore webhook configuration:

$
helm upgrade <release> dynamo-platform -n <namespace>

Best Practices

Production Deployments

1. ✅ Use failurePolicy: Fail (default) to ensure validation is enforced
2. ✅ Monitor webhook latency - Validation adds ~10-50ms per resource operation
3. ✅ Automatic certificates work well for production - The built-in cert-controller handles generation and rotation; use cert-manager only if you need integration with organizational PKI
4. ✅ Test webhook configuration in staging before production

Development Deployments

1. ✅ Use failurePolicy: Ignore if webhook availability is problematic during development
2. ✅ Keep automatic certificates (zero configuration, built into the operator)

Multi-Tenant Deployments

1. ✅ Deploy one cluster-wide operator for platform-wide validation
2. Use namespace-restricted operators only for development or testing, never as a production tenant-isolation mechanism

Additional Resources

• Kubernetes Admission Webhooks
• cert-manager Documentation
• Kubebuilder Webhook Tutorial
• CEL Validation Rules

Support

For issues or questions:

• Check Troubleshooting section
• Review operator logs: kubectl logs -n <namespace> deployment/<release>-dynamo-operator
• Open an issue on GitHub