MLFlow Customizer Secrets#

NVIDIA NeMo Customizer uses MLflow for metrics tracking. This guide explains how to create and manage two required secrets:

  1. MLflow credentials secret: Contains the authentication credentials for MLflow

  2. Customizer MLflow configuration secret: Contains the connection details that NeMo Customizer uses to connect to MLflow

Create MLflow Credentials Secret#

Create a secret containing the MLflow authentication credentials:

apiVersion: v1
kind: Secret
metadata:
  name: mlflow-credentials
  namespace: mlflow-system
type: Opaque
stringData:
  username: "bn_mlflow"  # Replace with secure username
  password: "bn_mlflow"  # Replace with secure password

Apply the secret:

kubectl apply -f mlflow-secret.yaml

Create Customizer MLflow Configuration Secret#

Create a secret containing the MLflow connection details that NeMo Customizer will use:

apiVersion: v1
kind: Secret
metadata:
  name: customizer-mlflow-config
  namespace: your-customizer-namespace
type: Opaque
stringData:
  MLFLOW_URL: "http://mlflow-tracking.mlflow-system.svc.cluster.local:80"
  MLFLOW_USERNAME: "bn_mlflow"  # Must match username in MLflow credentials secret
  MLFLOW_PASSWORD: "bn_mlflow"  # Must match password in MLflow credentials secret

Reference this secret in your Customizer configuration:

customizerConfig:
  mlflowSecretName: "customizer-mlflow-config"
  mlflowURLKey: "MLFLOW_URL"
  mlflowUsernameKey: "MLFLOW_USERNAME"
  mlflowPasswordKey: "MLFLOW_PASSWORD"