Skip to navigation

Advanced Control Plane Configurations

View as Markdown

Use this page to customize the NVIDIA Run:ai control plane beyond the default installation settings.

The settings on this page use NVIDIA Run:ai control plane Helm chart values. You can provide the same values through either installation path:

  • Manual Helm - Set the values in a Helm values file or pass them as Helm flags.
  • Guided Installation - Set the same values under controlPlane.customHelmValues in config.yaml. See controlPlane.

Helm Chart Values

The following Helm chart values can be used to customize the control plane for your environment. After changing a value, restart the relevant NVIDIA Run:ai pods so they can load the updated configuration.

KeyChangeDescription
global.ingress.ingressClassIngress classNVIDIA Run:ai uses NGINX as the default ingress controller. If your cluster has a different ingress controller, you can configure the ingress class to be created by NVIDIA Run:ai.
global.ingress.tlsSecretNameTLS secret nameNVIDIA Run:ai requires the creation of a secret with domain certificate. If the runai-backend namespace already had such a secret, you can set the secret name here
<service-name>.podLabelsPod labelsSet NVIDIA Run:ai and 3rd party services’ Pod Labels in a format of key/value pairs.
<service-name>.tolerationsPod tolerationsSet NVIDIA Run:ai and third-party services’ Pod Tolerations in list format. These tolerations apply only to the specified service.
global.tolerationsPod tolerationsSet NVIDIA Run:ai and 3rd party services’ Pod Tolerations in list format. These tolerations are applied globally to all supported services.
<service-name>.replicaCountPod replicasBy default, NVIDIA Run:ai services are deployed with a single replica. Set this value to run the specified service with multiple pod replicas.
global.replicaCountPod replicasBy default, NVIDIA Run:ai services are deployed with a single replica. Set this value to apply a global replica count to all services.
global.requireDefaultPodAntiAffinityEnable default pod anti-affinity

When enabled, NVIDIA Run:ai applies a default pod anti-affinity rule that attempts to prevent pods belonging to the same service from being scheduled on the same node.
Default: true

<service-name>
 resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

Pod request and limitsSet NVIDIA Run:ai and 3rd party services’ resources
disableIstioSidecarInjection.enabledDisable Istio sidecar injectionDisable the automatic injection of Istio sidecars across the entire NVIDIA Run:ai Control Plane services.
global.affinitySystem nodes

Sets the system nodes where NVIDIA Run:ai system-level services are scheduled.
Default: Prefer to schedule on nodes that are labeled with node-role.kubernetes.io/runai-system

global.customCA.enabledCertificate authorityEnables the use of a custom Certificate Authority (CA) in your deployment. When set to true, the system is configured to trust a user-provided CA certificate for secure communication.

Email Notifications Configuration

To enable and manage outbound email notifications for the NVIDIA Run:ai platform, configure the following values under the notificationsService.config.sinks.runai-email section in your Helm values. These settings are used both for workload-related notifications and for system messages from NVIDIA Run:ai. All parameters can be set in your values.yaml file during Helm deployment or via Helm upgrade.

ParameterRequiredDefault ValueDescription
notificationsService.config.sinks.runai-email.typerequiredemailType of sink; must be set as email
notificationsService.config.sinks.runai-email.smtp_hostrequiredEmptySMTP server hostname
notificationsService.config.sinks.runai-email.smtp_portoptional587SMTP server port
notificationsService.config.sinks.runai-email.useroptionalEmptySMTP authentication username
notificationsService.config.sinks.runai-email.passwordoptionalEmptySMTP authentication password
notificationsService.config.sinks.runai-email.smtp_tls_enabledoptionaltrueEnable TLS for SMTP
notificationsService.config.sinks.runai-email.from_display_nameoptionalNVIDIA Run:aiDisplay name for the sender email address
notificationsService.config.sinks.runai-email.fromoptionaltest@run.aiSender email address, e.g. notifications@my.domain
notificationsService.config.sinks.runai-email.direct_notificationsoptionalFalseSend workload notifications directly to the submitter’s inbox instead of sending all notifications to a single global email address
notificationsService.config.sinks.runai-email.auth_typeoptionalauth_loginAuthentication type: auth_login or auth_plain

Example Configuration

notificationsService:
config:
sinks:
runai-email:
type: email
smtp_host: my.smtp.host
smtp_port: 587
user: smtp_user
password: smtp_password
smtp_tls_enabled: true
from_display_name: NVIDIA Run:ai
from: notifications@my.domain
direct_notifications: true
logo_url: https://s3.amazonaws.com/www.run.ai/nvidia_runai_logo_new.png
auth_type: auth_login

Third-Party Services

The NVIDIA Run:ai control plane bundles third-party services it depends on - Keycloak for authentication, PostgreSQL as its database, and Thanos for metrics storage - and deploys them for you by default. This section covers tuning those built-in deployments and rotating their credentials. For what each service does, supported versions, and replacing one with an externally managed instance instead, see Third-Party Services.

Keycloak

Keycloak provides authentication and identity management for the NVIDIA Run:ai control plane. The Helm chart uses Keycloakx (keycloakx).

Note

The keycloakx.adminUser can only be set during the initial installation. The admin password can be changed later through the Keycloak UI, but you must also update keycloakx.adminPassword in the Helm chart using helm upgrade.

KeyChangeDescription
keycloakx.adminUserAdmin usernameDefines the username for the Keycloak administrator. Can only be set during initial installation.
keycloakx.adminPasswordAdmin passwordDefines the password for the Keycloak administrator.
keycloakx.existingSecretCredentials secretExisting secret name with authentication credentials.
global.keycloakx.hostKeycloak host pathOverrides the DNS for Keycloak. Use this to access Keycloak externally to the cluster.

Changing the Keycloak Admin Password

  1. Open the Keycloak UI at: https://<runai-domain>/auth
  2. Sign in with your existing admin credentials
  3. Go to Users and select admin (or your admin username)
  4. Open Credentials and select Reset password
  5. Set the new password and click Save
  6. Run helm upgrade and set keycloakx.adminPassword to the new password

Note

Failing to update the Helm values after changing the password can cause control plane services to encounter errors.

PostgreSQL

PostgreSQL is the centralized database for the NVIDIA Run:ai control plane, deployed as part of the control plane Helm chart using Bitnami PostgreSQL (postgresql).

The following values tune the built-in PostgreSQL deployment:

KeyChangeDescription
global.postgresql.auth.postgresPasswordPostgreSQL admin passwordPassword for the built-in PostgreSQL superuser (postgres).
postgresql.primary.initdb.passwordPostgreSQL admin passwordSet the same password as global.postgresql.auth.postgresPassword if changed.
postgresql.primary.persistence.storageClassStorage classUse a specific storage class instead of the default.

Changing PostgreSQL Passwords

  1. Connect to the PostgreSQL pod:

    kubectl exec -it runai-backend-postgresql-0 -n runai-backend -- bash
    psql -U postgres
  2. Update the passwords:

    ALTER USER "postgres" WITH PASSWORD '<new-postgres-password>';
    ALTER USER "user" WITH PASSWORD '<new-user-password>';
  3. Find the Kubernetes secret (default name: runai-backend-gen-postgresql) and update it with the new base64-encoded passwords. The keys are typically postgres-password and password.

  4. Restart all deployments to apply the changes:

    kubectl rollout restart deployment -n runai-backend

Note

Failing to restart the pods after updating the secret can cause control plane services to encounter errors.

Thanos

Thanos provides long-term metrics storage for the NVIDIA Run:ai control plane, deployed using Bitnami Thanos (thanos).

KeyChangeDescription
thanos.receive.persistence.storageClassStorage classUse a specific storage class instead of the default.