Skip to navigation

Kubernetes Operators

View as Markdown

Kubernetes operators are the modern way to manage Kubernetes cluster applications (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). It is usually recommended that Kubernetes operators are used instead of the legacy applications.

At the time of writing of this section (March 2024), NVIDIA Base Command Manager provides and packages several operators which are validated to perform basic functionalities on a Kubernetes BCM setup.

  • the NVIDIA GPU Operator (section 6.4)

  • the NVIDIA Network Operator (section 6.5)

  • the NVIDIA NetQ Operator (section 6.6)

  • the Prometheus Operator Stack (section 6.7)

  • the Prometheus Adapter Operator

  • the Jupyter Kernel Operator (section 6.3)

  • the Spark Operator (section 6.9)

  • the Postgres Operator (section 6.10)

  • the MPI Operator

  • the MetalLB Operator

  • the Kubeflow Training Operator

Run:ai is no longer one of them. Section 6.8 describes what changed, and how Run:ai can still be deployed on top of a BCM 10 Kubernetes cluster.

6.1 Versions Of Operators Available

The versions of the operators that are available can be listed with the cm-kubernetes-setup --list-operators-versions command.

The versions available at the time of writing (November 2024) for the operators are shown in the following table:

Kubernetes OperatorDefaultPossible versions
NVIDIA GPU Operator24.6.2(E)24.6.2(E), 24.6.1, 24.6.0, 24.3.0, 23.9.2, 23.9.1(E), 23.9.0(E), 23.6.2, 23.6.1(E), 23.6.0, 23.3.2(E), 23.3.1(E), 23.3.0, 22.9.2, 22.9.1(E), 22.9.0(E)
MetalLB0.14.80.14.8
NetQ Operator4.9 (Kubernetes 1.27),4.10 (Kubernetes 1.28),4.11 (Kubernetes 1.29), 4.12 (Kubernetes 1.30), 4.13 (Kubernetes 1.31)4.9, 4.10, 4.11, 4.12, 4.13
Network Operator24.7.024.7.0, 24.4.1, 24.4.0, 23.10.0, 23.7.0(E), 23.5.0(E), 23.1.0(E)
OVN CNI1.1.51.1.5
Postgres Operator1.13.01.13.0, 1.12.2, 1.10.1, 1.10.0, 1.9.0
Prometheus Adapter4.11.04.11.0
Prometheus Operator Stack62.7.062.7.0
SDN1.0.31.0.3
CM Jupyter Kernel Operator0.2.10.2.1
CM Kubernetes MPI Operator0.5.00.5.0
Spark Operator1.4.61.4.6

(E) NVAIE-certified

During the initial setup, the installation wizard displays a menu to select which operators are to be installed (figure 6.1).

Kubernetes setup TUI session: selection of operator packages to be installed

Figure 6.1: Kubernetes setup TUI session: selection of operator packages to be installed

After selecting the operators to be used, the actual operator versions available depend on package management resolution.

In the case of BCM operators, the installation wizard uses a Helm chart from the OS package manager for some of the .deb or .rpm packages being deployed. In the case of cloud operators, the installation wizard uses a Helm chart from the Helm repository for deployment.

6.2 Helm Charts For The BCM Operators

Based on the operators that are to be installed, the wizard asks for configuration options to the Helm charts that are to be installed (figure 6.2):

Kubernetes setup TUI session: prompting for Helm chart configuration options

Figure 6.2: Kubernetes setup TUI session: prompting for Helm chart configuration options

The Helm charts that are selected are installed with sensible defaults. If additional tuning is needed, then the charts can be installed manually after cm-kubernetes-setup finishes:

[root@basecm10 ~]# yum install cm-jupyter-kernel-operator -y
[root@basecm10 ~]# helm install cm-jupyter-kernel-operator -n cm --wait \
    /cm/shared/apps/jupyter-kernel-operator/current/helm/*.tgz

If additional tuning is required then tunable values can be set with a command line similar to the following:

[root@basecm10 ~]# helm install cm-jupyter-kernel-operator -n cm --wait \
    --values tunables.yaml \
    /cm/shared/apps/jupyter-kernel-operator/current/helm/*.tgz

Possible values can be displayed as follows:

[root@basecm10 ~]# helm show values /cm/shared/apps/jupyter-kernel-operator/current/helm/*.tgz

Installed operators can be listed by using the CLI option --list-operators:

[root@basecm10 ~]# cm-kubernetes-setup --list-operators
...
#### stage: kubernetes: Display Available Operators
OPERATOR________________________________: api_available___________________________
cm-jupyter-kernel-operator              : 1
postgresql-operator                     : 1
spark-operator                          : 1
cm-kubernetes-mpi-operator              : 1
...

6.3 The Jupyter Kernel Operator

6.3.1 Installing The Jupyter Kernel Operator

The Kubernetes Jupyter Kernel Operator can be installed as a part of the cm-kubernetes-setup procedure (section 4.2.6), which eventually leads to the selection screen displayed in figure 6.1.

The Kubernetes Jupyter Kernel Operator can alternatively be installed later on using the OS package manager and Helm:

[root@basecm10 ~]# yum install cm-jupyter-kernel-operator -y
[root@basecm10 ~]# helm install cm-jupyter-kernel-operator -n cm --create-namespace \
    /cm/shared/apps/jupyter-kernel-operator/current/helm/cm-jupyter-kernel-operator-*.tgz

The chart installs into the Helm release namespace given with -n rather than shipping its own hardcoded namespace, so a namespace must be specified. Installations done through cm-kubernetes-setup are unaffected by this, since it already passes --namespace cm --create-namespace. When upgrading an existing installation with helm upgrade, the namespace given must be the release’s existing namespace, and not a hardcoded cm-jupyter-kernel-operator-system: an older chart version shipped its own Namespace object, and upgrading with a mismatched -n deletes the entire release, because the old chart’s Namespace object is pruned. If the target namespace is not known, the release should be uninstalled first, then installed again into the new namespace.

The Jupyter Kernel Operator can be removed with:

Example

[root@basecm10 ~]# helm uninstall cm-jupyter-kernel-operator -n cm

It is recommended to enable a pod security policy using Kyverno (section 4.10.1) for the cluster before allowing a user, for example alice, to create resources in the Kubernetes cluster.

The Kubernetes Jupyter Kernel Operator Helm chart creates a CRD that can be used in the Kubernetes API. To check the availability of the CRD, the following command can be run:

Example

[root@basecm10 ~]# module load kubernetes
[root@basecm10 ~]# kubectl get crd | grep jupyterkernels
cmjupyterkernels.apps.brightcomputing.com                                  2022-11-07T09:49:48Z
cmkubernetesoperatorpermissionsjupyterkernels.charts.brightcomputing.com   2022-11-07T09:18:32Z

6.3.2 Architecture Overview

The Kubernetes Jupyter Kernel Operator has two main components:

  • the operator itself

  • the sidecar. This is attached to every user-defined kernel pod, and communicates with Jupyter Enterprise Gateway, acting as a proxy for the kernel process.

The following is an overview of the kernel setup and pod lifecycle when the user runs the Kubernetes Jupyter Kernel Operator:

  1. User initiates creating kernel in JupyterLab.

  2. JupyterLab delegates this task to Jupyter Enterprise Gateway (JEG).

  3. JEG opens a service TCP/IP socket and creates a CRD in Kubernetes specifying this port.

  4. KubeApi notifies Jupyter Kernel Operator about the newly created CRD.

  5. Jupyter Kernel Operator creates services, configmaps, secrets.

  6. Jupyter Kernel Operator creates pod to run Jupyter kernel based on the specification. The sidecar is added to the kernel pod during this step.

  7. The sidecar waits for the connection file created by the kernel. Alternatively, it relies on the connection file created by the operator (if requested), as not all kernels create a connection file.

  8. The sidecar runs a proxy to forward kernel communications to JEG (stdin, shell, iopub, etc).

  9. The sidecar notifies JEG about connection parameters and handles kernel communications.

  10. If JEG disappears, or if communication drops, then the sidecar stops. This causes the kernel operator to get a notification via the KubeApi service.

  11. The Kubernetes Jupyter Kernel Operator removes the unneeded pod, service, configmap and secrets. It also tries to gather stdout and std error of the kernel pod for debug purposes.

The pod created in step 6 is heavily customized by the kernel operator. For security reasons, running a process inside the pod must be carried out as an unprivileged user.

For the convenience of the Jupyter user, the UID/GID of the process inside the pod should match the UID/GID of the Jupyter user. If that is not the case, then the files created in the container are inaccessible for the Jupyter user.

To achieve matching UID/GIDs, the operator dynamically creates /etc/passwd and /etc/group files inside the pod and populates them with the data from corresponding templates. At the same time the operator can create a kernel communication file, if requested—some kernels rely on that.

6.3.3 Running Jupyter Kernel Using The Operator

An example of a basic YAML definition for the CMJupyterKernel is:

[alice@basecm10 ~]$ cat cmjk.yaml
---
apiVersion: apps.brightcomputing.com/v1
kind: CMJupyterKernel
metadata:
  name: cmjk-test
  namespace: alice-restricted
spec:
  username: alice
  uid: 1001
  gid: 1001
  kernel_id: testtesttest
  homedir: /home/alice
  pod:
    volumes:
    - name: homedir
      hostPath:
        path: /home/alice
        type: DirectoryOrCreate
    containers:
    - name: kernel
      image: jupyter/datascience-notebook
      command:
      - "python"
      args:
      - "-m"
      - "ipykernel_launcher"
      - "-f"
      - "/var/tmp/kernel-parm.json"
      workingDir: /home/alice
      securityContext:
        allowPrivilegeEscalation: false
        privileged: false
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
      volumeMounts:
      - name: homedir
        mountPath: /home/alice

This can be submitted, but the operator removes it in approximately 1 minute:

[alice@basecm10 ~]$ module load kubernetes
[alice@basecm10 ~]$ kubectl apply -f cmjk.yaml

The logs of the operator can be checked for debug purposes:

[root@basecm10 ~]# module load kubernetes
[root@basecm10 ~]# kubectl logs \
    -n cm \
    -l control-plane=controller-manager \
    --tail -1 \
    -c manager
...
2022-02-14T18:26:01.005Z    INFO    controllers.CMJupyterKernel Container is
stopped. Logs are below "cmjupyterkernel": "alice-restricted/cmjk-test"
...
2022-02-14T18:26:01.190Z    INFO    controllers.CMJupyterKernel cmjksidecar:
2022/02/14 18:26:00 Timeout receiving pings from server for 60 sec.
Shutting down.  "cmjupyterkernel": "alice-restricted/cmjk-test"
...

This indicates that the sidecar was stopped because there was no connection from Jupyter Enterprise Gateway to the kernel. This is expected, since the kernel has been run manually, and not using Jupyter. After the sidecar shutdown, the kube-api server notifies the operator, which, in turn, removes objects such as CMJupyterKernel, pods, and services.

6.3.4 Jupyter Kernel Operator Tunables

Table 6.1: Tunable Options for Jupyter Kernel Operators

OptionDescription
kernel_id^RKernel identifier (random UUID) given by Jupyter server
username^OName of the user
uid^R, gid^R, homedir^O, usershell^OUID, GID, home directory and default shell of the user
image_os_flavor^ODefines template of /etc/passwd and /etc/group files, where uid, gid, homedir, and usershell will be added. Could be one of ubuntu1604, ubuntu1804, ubuntu2004, ubuntu2404, rhel7, centos7, rhel8, centos8, sles12, sles15.
etc_passwd^O, etc_group^OCustom content of the /etc/passwd or /etc/group, if necessary.
sidecar_command^O, sidecar_args^OCommands and arguments to run the sidecar. By default empty. Most of the arguments for the sidecar are passed via environment variables (section 6.3.5).
kernel_connection_file_path^OWhere to expect to find kernel connection file. Default: /var/tmp/kernel-parm.json
create_connection_file^ODoes the operator need to create and populate kernel connection file before the pod starts? Default: false
spark_pod_template_path^O, spark_pod_template^OOptions to store or override the Spark executor template
pod^RKubernetes Pod definition
service^OKubernetes Service definition

Legend:

O: Optional

R: Required

6.3.5 Sidecar Arguments And Environment Variables

6.3.5.1 Sidecar Arguments

A timeout can be set as an argument for the sidecar.

  • --timeout: Defines how long, in seconds, that the sidecar waits for the Jupyter Enterprise Gateway proxy to connect before shutdown. Default: 60

6.3.5.2 Environment Variables

The following environment variables can be used by the sidecar:

Table 6.2: Environment Variables For The Sidecar

Environment VariableDescription
CMJK_CONNECTION_FILEPath to find a connection file. The sidecar uses the file to establish a connection to the kernel and to pass data between Jupyter Enterprise Gateway and the kernel. Default: /var/tmp/kernel-parm.json.
CMJK_KERNEL_IDUnique identifier of the kernel. Usually the UUID in table 6.1.
CMJK_SHELL_PORTProxy port to open to forward shell communication. Default: 5001.
CMJK_IOPUB_PORTProxy port to open to forward iopub communication. Default: 5002.
CMJK_STDIN_PORTProxy port to open to forward stdin communication. Default: 5003.
CMJK_CONTROL_PORTProxy port to open to forward control communication. Default: 5004.
CMJK_HB_PORTProxy port to open to forward heartbeat communication. Default: 5005.
CMJK_COMM_PORTProxy port to open to forward comm communication. Default: 5006.

6.3.6 Running Spark-based Kernels In Jupyter Kernel Operator

Jupyter integration for BCM provides a kernel template (jupyter-eg-kernel-k8s-cmjkop-py-spark) and a sample container image (brightcomputing/jupyter-kernel-sample:k8s-spark-py39-2.0.0) to run Jupyter kernels in a Spark environment. The image can be altered or created from scratch based on the scripts provided in /cm/shared/examples/jupyter/kubernetes-kernel-image-spark-py39/.

The Jupyter kernel is not run directly. Instead, the kernel process is run and controlled by the spark-submit executable inside the container.

Jupyter Kernel Operator alters the provided image based on the CRD definition.

Spark-specific tunables are spark_pod_template_path and spark_pod_template. The operator creates a file inside of the Spark driver pod and puts the content of spark_pod_template in it. After that, spark-submit uses this file, via the --spark.kubernetes.executor.podTemplateFile configuration option, to create executor pods.

6.3.7 Example: Creating An R Kernel From The Kernel Template

The Jupyter Kernel Operator can be used out-of-the-box to support more kernels.

For example, an R kernel can be added.

The official jupyter/r-notebook R image can be taken from the Jupyter project, from https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-r-notebook.

The default entry point cannot be used as that would start Jupyter notebook, while the aim for this section is to use the kernel only.

Some exploratory investigation should reveal the command to start the kernel:

The pod can be run interactively in a Jupyter notebook terminal by a user:

    kubectl run -i --tty testnotebook --image=jupyter/r-notebook --restart=Never -- bash

The kernel specifications can then be investigated:

jupyter-kernelspec list
Available kernels:
  ir         /opt/conda/share/jupyter/kernels/ir
  python3    /opt/conda/share/jupyter/kernels/python3

The ir kernel is what is of interest here. The command line that is used to start the kernel can be found:

cat /opt/conda/share/jupyter/kernels/ir/kernel.json
"argv": ["R", "--slave", "-e", "IRkernel::main()", "--args", "connection_file"],
 "display_name":"R",
 "language":"R"

Based on this information the Jupyter Kernel Operator CRD can be created for the user:

cat cmjk-ir.yaml
---
apiVersion: apps.brightcomputing.com/v1
kind: CMJupyterKernel
metadata:
  name: cmjk-test
  namespace: alice-restricted
spec:
  username: alice
  uid: 1001
  gid: 1001
  kernel_id: testtesttest
  homedir: /home/alice
  create_connection_file: true  # R kernel expects connection file be created
  pod:
    volumes:
    - name: homedir
      hostPath:
        path: /home/alice
        type: DirectoryOrCreate
    containers:
    - name: kernel
      image: jupyter/r-notebook  # image
      command:
      - "R"
      args:
      - "--slave"
      - "-e"
      - "IRkernel::main()"
      - "--args"
      - "/var/tmp/kernel-parm.json"  # we have static connection file
      workingDir: /home/alice
      securityContext:
        allowPrivilegeEscalation: false
        privileged: false
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
      volumeMounts:
      - name: homedir
        mountPath: /home/alice

There are several changes from the previous (section 6.3.3) YAML, and from the IR command line:

  • create_connection_file: true

    If this is not specified then the kernel complains with the following message during startup:

    kernel: cannot open file '/var/tmp/kernel-parm.json': No such file or directory

    This means that the kernel expected this file to be created before the start.

  • image: jupyter/r-notebook

    Another image needs to be used.

  • args: ... "/var/tmp/kernel-parm.json"

    The spec file has a fixed path and name, instead of "connection_file``" as in kernel.json earlier

The resulting cmjk-ir.yaml file can be submitted to Kubernetes, but it will be removed by the operator after one minute, as it is not being started from the Jupyter Enterprise Gateway.

The next step is to create a kernel template. The Python kernel can be used as a reference:

    cd /cm/shared/apps/jupyter/current
    cd lib/python*/site-packages/cm_jupyter_kernel_creator/kerneltemplates
    cp -pr jupyter-eg-kernel-k8s-cmjkop-py jupyter-eg-kernel-k8s-cmjkop-r

The files meta.yaml, kernel.json, and templates/cmjk.yaml.j2 need to be changed in order to able to provide the correct image and command:

    vim jupyter-eg-kernel-k8s-cmjkop-r/meta.yaml
    vim jupyter-eg-kernel-k8s-cmjkop-r/kernel.json.j2
    vim templates/cmjk.yaml.j2

The changes that are applied should look similar to the following:

diff -u jupyter-eg-kernel-k8s-cmjkop-py/kernel.json.j2 jupyter-eg-kernel-k8s-cmjkop-r/kernel.json.j2
--- jupyter-eg-kernel-k8s-cmjkop-py/kernel.json.j2  2022-01-25 21:13:52.000000000 +0100
+++ jupyter-eg-kernel-k8s-cmjkop-r/kernel.json.j2   2022-02-16 12:22:23.610382929 +0100
@@ -15,8 +15,8 @@
     }
   },
   "argv": [
-    "python",
-    "-m", "ipykernel_launcher",
-    "-f", "/var/tmp/kernel-parm.json"
+    "R",
+    "--slave", "-e", "IRkernel::main()",
+    "--args", "/var/tmp/kernel-parm.json"
   ]
 }
diff -u jupyter-eg-kernel-k8s-cmjkop-py/meta.yaml jupyter-eg-kernel-k8s-cmjkop-r/meta.yaml
--- jupyter-eg-kernel-k8s-cmjkop-py/meta.yaml   2022-01-25 21:13:52.000000000 +0100
+++ jupyter-eg-kernel-k8s-cmjkop-r/meta.yaml    2022-02-16 12:20:57.500974886 +0100
@@ -1,5 +1,5 @@
 ---
-display_name: "Python on Kubernetes Operator"
+display_name: "R on Kubernetes Operator"
 features: "k8s-jupyter-operator-enabled"
 parameters:
   display_name:
@@ -7,7 +7,7 @@
     definition:
       getter: shell
       exec:
-      - echo "Python on Kubernetes Operator $(date +%y%m%d%H%M%S)"
+      - echo "R on Kubernetes Operator $(date +%y%m%d%H%M%S)"
       display_name: "Display name of the kernel"
   k8s_env_module:
     type: str
@@ -20,9 +20,9 @@
     definition:
       getter: static
       default:
-      - "jupyter/datascience-notebook"
+      - "jupyter/r-notebook"
       values:
-      - "jupyter/datascience-notebook"
+      - "jupyter/r-notebook"
       display_name: "Image to run"
     limits:
       max_len: 1
    diff -u jupyter-eg-kernel-k8s-cmjkop-py/templates/cmjk.yaml.j2\
    jupyter-eg-kernel-k8s-cmjkop-r/templates/cmjk.yaml.j2
--- jupyter-eg-kernel-k8s-cmjkop-py/templates/cmjk.yaml.j2  2022-01-25 21:13:52.000000000 +0100
+++ jupyter-eg-kernel-k8s-cmjkop-r/templates/cmjk.yaml.j2   2022-02-16 12:24:25.375373991 +0100
@@ -9,6 +9,7 @@
   gid:  gid
   kernel_id:  kernel_id
   homedir:  homedir
+  create_connection_file: true
   pod:
     volumes:
     - name: homedir

After instantiating a kernel spec from the template, the R kernel is ready for use:

Jupyter Kernel Creator: creating the IR kernel spec

Jupyter Kernel Creator: creating the IR kernel spec

JupyterLab: running the IR kernel

JupyterLab: running the IR kernel

6.3.8 Example: Letting Kubernetes Access Private Registries From The Kernel Template

To be able to pull images from private registries, Kubernetes needs to be instructed about the credentials to use.

6.3.8.1 Creating The Secret

This can be achieved by specifying the secret to spec.imagePullSecrets of the pod definition.

For Jupter Kernel Operator this is spec.pod.imagePullSecrets:

Example

  pod:
    containers:
    - name: kernel
      image:  image
      ...
      imagePullSecrets:
      - name: regcred
      ...

Creating secrets can be carried out with kubectl

Example

kubectl create \
    --namespace alice-restricted \
    secret docker-registry regcred \
    --docker-server=<your-registry-server> \
    --docker-username=<your-name> \
    --docker-password=<your-pword> \
    --docker-email=<your-email>

More details about managing certificates can be found in the Kubernetes documentation at:
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

6.3.8.2 Parameterizing The Secret

The name of the secret can be parameterized, so that users are allowed to select from secrets in their namespaces.

Jupyter Kernel Creator: secret selection

Jupyter Kernel Creator: secret selection

For parameterization, the meta.yaml and kernel.json.j2 files must also then be modified:

Example

# cat meta.yaml
...
parameters:
...
  image_pull_secret_name:
    type: str
    definition:
      getter: static
      default: ""
      display_name: "Name of the secret to pull images"
# cat kernel.json.j2
...
  "metadata": {
    "process_proxy": {
      "class_name": "cm_jupyter_kernel_creator.eg_processproxies.k8scmjkop.KubernetesCMJupyterKernelOperator",
      "config": {
...
        "image_pull_policy": "{{ image_pull_policy[0] }}",
        "namespace": "{{ kubernetes_namespace }}",
        "image_pull_secret_name": "{{ image_pull_secret_name }}",
        "gpu_limit": {{ gpu_limit }}
      }
...
# cat templates/cmjk.yaml.j2
...
  pod:
...
    {%- if image_pull_secret_name %}
    imagePullSecrets:
    - name: {{ image_pull_secret_name }}
    {%- endif %}
    containers:
    - name: kernel
      image: {{ image }}
...

6.3.9 Example: Adding The PVC Parameter To The Kernel Template

The PVC that is to be mounted can be set in the template:

Jupyter Kernel Creator, PVC selection.

Jupyter Kernel Creator, PVC selection.

The settings in meta.yaml, kernel.json.j2 and templates/cmjk.yaml.j2 for this are:

Example

# cat meta.yaml
...
parameters:
...
  pvc_name:
    type: list
    definition:
      getter: shell
      exec:
      - source /etc/profile.d/modules.sh
      - module load kubernetes
      - kubectl get pvc -o jsonpath="{range .items[*]}{.metadata.name}{'\\n'}{end}"
      display_name: "PVC to mount"
    limits:
      max_len: 1
      min_len: 1
  pvc_mountpoint:
    type: str
    definition:
      getter: static
      default: "/data"
      display_name: "Mountpoint to PVC"
# cat kernel.json.j2
...
  "metadata": {
    "process_proxy": {
      "class_name": "cm_jupyter_kernel_creator.eg_processproxies.k8scmjkop.KubernetesCMJupyterKernelOperator",
      "config": {
...
        "image_pull_policy": "{{ image_pull_policy[0] }}",
        "namespace": "{{ kubernetes_namespace }}",
        "pvc_name": "{{ pvc_name[0] }}",
        "pvc_mountpoint": "{{ pvc_mountpoint }}",
        "gpu_limit": {{ gpu_limit }}
      }
...
# cat templates/cmjk.yaml.j2
...
  pod:
    volumes:
...
    {%- if pvc_name and pvc_mountpoint %}
    - name: pvc
      persistentVolumeClaim:
        claimName: {{ pvc_name }}
    {%- endif %}
    containers:
    - name: kernel
      image: {{ image }}
      imagePullPolicy: {{ image_pull_policy }}
      command: {{ command }}
...
      volumeMounts:
      - name: homedir
        mountPath: {{ homedir }}
      {%- if pvc_name and pvc_mountpoint %}
      - name: pvc
        mountPath: {{ pvc_mountpoint }}
      {%- endif %}
...

6.4 The NVIDIA GPU Operator

6.4.1 Installing The NVIDIA GPU Operator

The NVIDIA GPU operator can be installed as a part of the installation session by the cm-kubernetes-setup wizard (section 4.2.6). During the setup session, a checkbox can be checkmarked to install and enable the GPU operator (figure 6.1). Nodes that run DGX OS are also supported by the wizard.

A specific version can be selected for the installation (figure 6.7):

GPU operator version selection screen

Figure 6.7: GPU operator version selection screen

The NVIDIA GPU operator can also be deployed on an existing BCM Kubernetes cluster, as described next.

6.4.2 Installing The NVIDIA GPU Operator On An Existing Kubernetes Cluster

If the NVIDIA GPU Operator (https://docs.nvidia.com/datacenter/cloud-native/gpu-operator) is to be installed within an existing BCM Kubernetes cluster, then it must always be deployed using Helm.

Prerequisites:

If the existing cluster uses the NVIDIA device plugin add-on, even if configured by NVIDIA Base Command Manager, then it may be necessary to disable the add-on. This add-on is now deprecated, and will be removed in a future release.

[cluster->kubernetes[default]->appgroups[system]->applications[nvidia]]% set enabled no
[cluster->kubernetes*[default*]->appgroups*[system*]->applications*[nvidia*]]% commit

One of the prerequisites for the preceding add-on is that it uses labels to identify the nodes to be managed by the add-on. These labels are unnecessary for the GPU operator, and may be removed:

[cluster->kubernetes[default]->labelsets]% remove nvidia
[cluster->kubernetes*[default*]->labelsets*]% commit

Installing The NVIDIA GPU Operator: Validation methods are described for each step of the deployment.

  • For containerd, Helm installation is carried out by the root user with the following options:

    helm install --wait -n gpu-operator --create-namespace \
                 --version v1.10.1 \
                 --set driver.enabled=false \
                 --set operator.defaultRuntime=containerd \
                 --set toolkit.enabled=true \
                 --set toolkit.env[0].name=CONTAINERD_CONFIG \
                 --set toolkit.env[0].value=/cm/local/apps/containerd/var/etc/conf.d/nvidia-cri.toml \
                 gpu-operator nvidia/gpu-operator

  • For docker, Helm installation is carried out by the root user with the following options:

    helm install --wait -n gpu-operator --create-namespace \
                 --version v1.10.1 \
                 --set driver.enabled=false \
                 --set operator.defaultRuntime=docker \
                 --set toolkit.enabled=true \
                 gpu-operator nvidia/gpu-operator

Helm chart options are documented at https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#common-chart-customization-options.

NVIDIA GPU Operator containerd configuration:

The operator provides the toolkit binaries and containerd configuration (nvidia-cri.toml) on each host where a GPU is auto-detected via a host-mount.

The flag that enables this is --set toolkit.enabled=true. The path for the configuration file should be set to: /cm/local/apps/containerd/var/etc/conf.d/nvidia-cri.toml, which is where BCM’s cm-containerd package expects to find it.

The operator provides a similar configuration functionality for the CUDA drivers. However this is not used in BCM, and it is disabled with the --set driver.enabled=false flag. This is because BCM supports CUDA on more Linux distributions and kernel versions than the NVIDIA GPU Operator does. CUDA drivers are therefore expected to already be present on the relevant nodes that have GPUs.

NVIDIA GPU Operator Docker configuration:

This is only relevant for older Kubernetes deployments that are deployed on top of Docker or BCM Docker.

Default paths are used, so nothing particularly special has to be done for the operator to deploy properly.

6.4.3 Removing The NVIDIA GPU Operator

The NVIDIA GPU Operator can be found in the gpu-operator namespace inside Helm and Kubernetes.

root@basecm10 ~# helm list -n gpu-operator
NAME             NAMESPACE     REVISION ... STATUS     CHART                  APP VERSION
gpu-operator     gpu-operator  1        ... deployed   gpu-operator-v1.10.1   v1.10.1

A helm uninstall gpu-operator command can be used to uninstall the operator.

6.4.4 Validating The NVIDIA GPU Operator

A pragmatic way to validate the NVIDIA GPU Operator is to check if the validator pods can be run. A Running status for the pods that are to have a GPU on them can be seen with:

Example

root@basecm10 ~# kubectl get pod -n gpu-operator -l app=nvidia-operator-validator -o wide
NAME                              READY  STATUS   ... IP              NODE     NOMINATED NODE  READINESS GATES
nvidia-operator-validator-2qvz6   1/1    Running  ... 172.29.152.172  node001  <none>          <none>
nvidia-operator-validator-xkwwv   1/1    Running  ... 172.29.112.154  node002  <none>          <none>

The preceding shows successfully-running pods. The log output should show all validations are successful:

Example

root@basecm10 ~# kubectl logs -n gpu-operator -l app=nvidia-operator-validator -c nvidia-operator-validator
all validations are successful
all validations are successful

6.4.5 Validating The NVIDIA GPU Operator In Detail

The set of pods associated with the NVIDIA GPU Operator can be examined in more detail. The following shows outputs from a GPU operator deployment that is working correctly:

Example

root@basecm10 ~# helm list -n gpu-operator
NAME           NAMESPACE    ... STATUS     CHART                APP VERSION
gpu-operator   gpu-operator ... deployed  gpu-operator-v1.10.1  v1.10.1
root@basecm10 ~# kubectl get all -n gpu-operator -o wide
NAME                                                              READY  STATUS      RESTARTS   ...   NODE
pod/gpu-feature-discovery-gk892                                   1/1    Running     0          ...   node001
pod/gpu-feature-discovery-rmkvj                                   1/1    Running     0          ...   node002
pod/gpu-operator-798c6ddc97-lmclm                                 1/1    Running     0          ...   basecm10
pod/gpu-operator-node-feature-discovery-master-6c65c99969-cjlpq   1/1    Running     0          ...   basecm10
pod/gpu-operator-node-feature-discovery-worker-cgxzl              1/1    Running     0          ...   node002
pod/gpu-operator-node-feature-discovery-worker-ds5mb              1/1    Running     0          ...   basecm10
pod/gpu-operator-node-feature-discovery-worker-jf65c              1/1    Running     0          ...   node001
pod/nvidia-container-toolkit-daemonset-ffbk7                      1/1    Running     1 (46m ago)...   node002
pod/nvidia-container-toolkit-daemonset-lqfkq                      1/1    Running     0          ...   node001
pod/nvidia-cuda-validator-pxs9b                                   0/1    Completed   0          ...   node001
pod/nvidia-cuda-validator-v7gfz                                   0/1    Completed   0          ...   node002
pod/nvidia-dcgm-exporter-bxjrv                                    1/1    Running     0          ...   node001
pod/nvidia-dcgm-exporter-qll9z                                    1/1    Running     0          ...   node002
pod/nvidia-device-plugin-daemonset-698hd                          1/1    Running     0          ...   node001
pod/nvidia-device-plugin-daemonset-xd4kj                          1/1    Running     0          ...   node002
pod/nvidia-device-plugin-validator-5crlc                          0/1    Completed   0          ...   node001
pod/nvidia-device-plugin-validator-wh27x                          0/1    Completed   0          ...   node002
pod/nvidia-operator-validator-2qvz6                               1/1    Running     0          ...   node001
pod/nvidia-operator-validator-xkwwv                               1/1    Running     0          ...   node002
...

On this particular example cluster, there are two compute nodes with GPUs, and there is one control plane node without a GPU:

root@basecm10 ~# kubectl get nodes
NAME       STATUS   ROLES                  AGE    VERSION
node001    Ready    worker                 3h2m   v1.24.0
node002    Ready    worker                 3h2m   v1.24.0
basecm10   Ready    control-plane,master   3h2m   v1.24.0

Feature discovery pods: Node Feature Discovery (NFD, https://intel.github.io/kubernetes-docs/nfd/index.html) is an add-on that is initiated after the operator is installed. A master pod collects discovery information from the worker pods, and schedules more pods in case GPUs have been detected.

In the preceding GPU operator output,

  • the master pod is running on node001 with the name:
    gpu-operator-node-feature-discovery-master-6c65c99969-wtzcx

  • the worker pods run on each node. For example, the worker pod for node002 is:
    gpu-operator-node-feature-discovery-worker-z4skv

The output for the pods is not very verbose by default, but if more pods under the nvidia- namespace are scheduled on a node, besides the gpu-operator-node-feature-discovery-* pods, then that means that NFD has detected one or more GPUs.

For example, a GPU discovered on node001 results in a scheduling of the following pods on that node:

  • container toolkit

  • device plugin

  • validator

Container toolkit pods: For nodes that have GPUs, the NVIDIA container toolkit installation pods are started. Pod logs show exactly what is being installed.

One of the requirements for the NVIDIA container toolkit installation pods is that the driver has to be in working order, or the init container driver-validation will fail. The following is the log from a successful installation:

Example

root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-container-toolkit-daemonset-ffbk7
Defaulted container "nvidia-container-toolkit-ctr" out of: nvidia-container-toolkit-ctr, driver-validation (init)
...
time="2022-12-06T14:31:36Z" level=info msg="Installing toolkit"
time="2022-12-06T14:31:36Z" level=info msg="Parsing arguments: [/usr/local/nvidia/toolkit]"
time="2022-12-06T14:31:36Z" level=info msg="Successfully parsed arguments"
time="2022-12-06T14:31:36Z" level=info msg="Installing NVIDIA container toolkit to '/usr/local/nvidia/toolkit'"
...
time="2022-12-06T14:31:36Z" level=info msg="Installing NVIDIA container toolkit config
                                   '/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml'"
time="2022-12-06T14:31:36Z" level=info msg="Setting up runtime"
time="2022-12-06T14:31:36Z" level=info msg="Starting 'setup' for containerd"
time="2022-12-06T14:31:36Z" level=info msg="Parsing arguments: [/usr/local/nvidia/toolkit]"
time="2022-12-06T14:31:36Z" level=info msg="Successfully parsed arguments"
time="2022-12-06T14:31:36Z" level=info msg="Loading config: /runtime/config-dir/nvidia-cri.toml"
...

Device plugin pods: The device plugin pods are started up next. These have the toolkit as a requirement. If the toolkit is not in working order, then the init container toolkit-validation fails. The following is the log from a successful startup:

Example

root@basecm10 ~# kubectl logs -f -n gpu-operator           nvidia-device-plugin-daemonset-698hd
Defaulted container "nvidia-device-plugin-ctr" out of: nvidia-device-plugin-ctr, toolkit-validation (init)
2022/12/06 14:32:20 Loading NVML
2022/12/06 14:32:20 Starting FS watcher.
2022/12/06 14:32:20 Starting OS watcher.
2022/12/06 14:32:20 Retreiving plugins.
2022/12/06 14:32:20 No MIG devices found. Falling back to mig.strategy=&
2022/12/06 14:32:20 Starting GRPC server for 'nvidia.com/gpu'
2022/12/06 14:32:20 Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia-gpu.sock
2022/12/06 14:32:20 Registered device plugin for 'nvidia.com/gpu' with Kubelet

The pod log output suggests that the GPU is now registered with the Kubelet as a resource. This can be checked by querying the Node resource:

Example

root@basecm10 ~# kubectl describe node node001 | grep nvidia
    nvidia.com/cuda.driver.major=520
    nvidia.com/cuda.driver.minor=61
    nvidia.com/cuda.driver.rev=05
    nvidia.com/cuda.runtime.major=11
    nvidia.com/cuda.runtime.minor=8
    nvidia.com/gfd.timestamp=1670337142
    nvidia.com/gpu.compute.major=7
    nvidia.com/gpu.compute.minor=0
    nvidia.com/gpu.count=1
    nvidia.com/gpu.deploy.container-toolkit=true
    nvidia.com/gpu.deploy.dcgm=true
    nvidia.com/gpu.deploy.dcgm-exporter=true
    nvidia.com/gpu.deploy.device-plugin=true
    nvidia.com/gpu.deploy.driver=true
    nvidia.com/gpu.deploy.gpu-feature-discovery=true
    nvidia.com/gpu.deploy.node-status-exporter=true
    nvidia.com/gpu.deploy.operator-validator=true
    nvidia.com/gpu.family=volta
    nvidia.com/gpu.machine=OpenStack-Nova
    nvidia.com/gpu.memory=32768
    nvidia.com/gpu.present=true
    nvidia.com/gpu.product=Tesla-V100-SXM3-32GB
    nvidia.com/mig.strategy=single
    nvidia.com/run.ai-swap.enabled=false
    nvidia.com/gpu:     1
    nvidia.com/gpu:     1
...

Validator pods: If anything goes wrong with either the driver, toolkit, CUDA, or the plugin, then validator pods are a good place to start looking.

If all goes well, the main container outputs all validations are successful:

Example

root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6
Defaulted container "nvidia-operator-validator" out of: nvidia-operator-validator, driver-validation (init),
    toolkit-validation (init), cuda-validation (init), plugin-validation (init)
all validations are successful

It is possible for an init container to fail. The output for the container should then be checked.

The following shows output from successful init containers:

root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c driver-validation
running command chroot with args [/run/nvidia/driver nvidia-smi]
Tue Dec  6 15:32:14 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM3...  On   | 00000000:00:06.0 Off |                    0 |
| N/A   32C    P0    46W / 350W |      2MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c toolkit-validation
Tue Dec  6 14:32:16 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM3...  On   | 00000000:00:06.0 Off |                    0 |
| N/A   32C    P0    46W / 350W |      2MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c cuda-validation
time="2022-12-06T14:32:17Z" level=info msg="pod nvidia-cuda-validator-pxs9b is curently in Pending phase"
time="2022-12-06T14:32:22Z" level=info msg="pod nvidia-cuda-validator-pxs9b is curently in Pending phase"
time="2022-12-06T14:32:27Z" level=info msg="pod nvidia-cuda-validator-pxs9b is curently in Pending phase"
time="2022-12-06T14:32:32Z" level=info msg="pod nvidia-cuda-validator-pxs9b have run successfully"
root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c plugin-validation
time="2022-12-06T14:32:33Z" level=info msg="pod nvidia-device-plugin-validator-5crlc is curently in Pending phase"
time="2022-12-06T14:32:38Z" level=info msg="pod nvidia-device-plugin-validator-5crlc is curently in Pending phase"
time="2022-12-06T14:32:43Z" level=info msg="pod nvidia-device-plugin-validator-5crlc have run successfully"

This also explains where the pods earlier on came from, the ones marked with status Completed. They are used as part of certain validation steps.

Which init container prints out error messages should indicate where the problem lies—either with the CUDA drivers, or the toolkit, and so on. If the driver or toolkit is not validating correctly, then it may result in a lot of pods stuck in a Pending or an Init stage. Looking at what init container is associated with the stuck pod helps in diagnosing the problem.

DCGM exporter pods: These pods expose metrics endpoints for scraping, and can be considered less critical. They are involved in GPU metrics collection, and can be utilized with, for example, Prometheus Operator Stack, or the Prometheus Adapter, for horizontal pod autoscaling based on GPU metrics.

6.4.6 Running A GPU Workload

A GPU workload can be run with the following configuration:

Example

root@basecm10 ~# cat << EOF > gpu.yaml
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  restartPolicy: Never
  containers:
  - name: cuda-container
    image: nvidia/cuda:11.0.3-base-ubuntu20.04
    command: ["nvidia-smi"]
    resources:
      limits:
        nvidia.com/gpu: 1
EOF
root@basecm10 ~# kubectl create -f gpu.yaml
pod/gpu-pod created

On a cluster with GPUs available, this pod should get scheduled, and should not stay stuck in the Pending phase.

The preceding example just invokes nvidia-smi in the container. The output can be viewed to confirm that it worked:

Example

root@basecm10 ~# kubectl logs -f gpu-pod
Tue Dec  6 15:08:03 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM3...  On   | 00000000:00:06.0 Off |                    0 |
| N/A   34C    P0    47W / 350W |      2MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

6.5 The NVIDIA Network Operator

6.5.1 Installing The NVIDIA Network Operator

The upstream documentation for the NVIDIA Network Operator is quite extensive, and available at https://docs.nvidia.com/networking/software/cloud-orchestration/index.html#network-operator.

6.5.1.1 Going Through The Kubernetes Setup Wizard

The Network operator installation is also part of the initial Kubernetes cluster setup. One of the steps in the wizard asks the cluster administrator to select operators from a list. The Network operator is one of them. If the Network operator is selected (figure 6.8), then the setup wizard continues with some follow-up checks and questions.

Kubernetes operators selection, with Network operator selected

Figure 6.8: Kubernetes operators selection, with Network operator selected

The wizard then asks which version of the operator should be installed (figure 6.9):

Network operator version selection screen

Figure 6.9: Network operator version selection screen

The wizard then prompts for a custom YAML configuration file to use for the Helm chart deployment (figure 6.10):

Network operator prompt for custom YAML configuration for Helm

Figure 6.10: Network operator prompt for custom YAML configuration for Helm

If a custom YAML configuration file is not chosen, then options for a Helm chart can be set (figure 6.11):

Network operator options for Helm chart prompt.

Figure 6.11: Network operator options for Helm chart prompt.

6.5.1.2 Result Of The Network Operator Helm Chart

If all goes well, then the Helm chart should be deployed successfully:

Example

root@basecm10 ~# module load kubernetes/default/1.27.11-1.1
root@basecm10 ~# helm list -A -a
NAME                   NAMESPACE       ... STATUS  CHART                                       APP VERSION
local-path-provisioner cm              ... deployedcm-kubernetes-local-path-provisioner-0.0.26 0.0.26
network-operator       network-operator... deployednetwork-operator-23.7.0                     v23.7.0
permissions-manager    cm              ... deployedcm-kubernetes-permissions-manager-0.4.8     0.4.8
root@bright:~# helm history -n network-operator
REVISION    UPDATED                     STATUS      CHART                   APP VERSION DESCRIPTION
1           Thu Feb 29 14:43:20 2024    deployed    network-operator-23.7.0 v23.7.0     Install complete

6.6 The NVIDIA NetQ Operator

6.6.1 NVIDIA NetQ Operator Installation

6.6.1.1 Prerequisites

  • The only supported OS is Ubuntu 20.04.

  • The number of NetQ nodes can be 1 or 3 nodes only. These must be regular compute nodes.

  • The NetQ nodes require at least 250 GiB of diskspace to be available.

  • The NetQ nodes require at least 64 GiB of memory.

  • The NVIDIA Base Command Manager default ports cannot be used.

  • Kyverno is not supported in combination with NetQ at the time of writing (March 2024) of this section.

The default ports can be changed on the pctive and passive head nodes as follows.

cm-cmd-ports --http 8083 --https 8084 && systemctl restart cmd

6.6.1.2 Going Through The Kubernetes Setup Wizard

The NetQ installation is part of the initial Kubernetes cluster setup. One of the steps in the wizard asks the cluster administrator to choose operators from a list of operators. NetQ is one of the possible operators. If the NetQ operator is selected (figure 6.12), then the setup wizard continues with some follow-up checks and questions.

Kubernetes NetQ Operator Selection

Figure 6.12: Kubernetes NetQ Operator Selection

Node selection asks which nodes are to be assigned to NetQ (see figure 6.13). The options are:

  • for a single-deployment (this is NetQ terminology), one node should be selected

  • For a cluster-deployment (also NetQ terminology), three nodes should be selected

NetQ nodes selection

Figure 6.13: NetQ nodes selection

After node selection, the nodes are inspected to see if they meet the prerequisites for running NetQ. One potential issue it detects is if NVIDIA Base Command Manager is already running on the default ports. It suggests the commands to execute to fix that, (figure 6.14). NetQ prerequisites and how to carry out changing the default ports is described further in section 6.6.1.1. After carrying out the port changes, the wizard must be restarted from the beginning.

Warning presented if NVIDIA Base Command Manager still uses default port 8081.

Figure 6.14: Warning presented if NVIDIA Base Command Manager still uses default port 8081.

lf all prerequisites are met, then the next dialog asks for a few files. These have to be provided by NetQ and stored somewhere on the active head node (figure 6.15).

Dialog asking for NetQ tarball and packages for agent and apps.

Figure 6.15: Dialog asking for NetQ tarball and packages for agent and apps.

When the required files are downloaded, and saved to, for example, /root:

Example

root@basecm10 ~# ls -alh | grep -i netq
-rw-r--r--  1 root root  35M Jan  9 09:05 netq-agent_4.9.0-ub20.04u45~1703950858.128b0741e_amd64.deb
-rw-r--r--  1 root root  32M Jan  9 09:05 netq-apps_4.9.0-ub20.04u45~1703950858.128b0741e_amd64.deb
-rw-r--r--  1 root root  13G Jan  9 09:09 NetQ-bcm-4.9.0-SNAPSHOT.tgz

the dialog can then be filled in (figure 6.16):

Filled-in dialog asking for NetQ files.

Figure 6.16: Filled-in dialog asking for NetQ files.

Depending on whether a single-node or cluster (three-node) setup is being done, the next dialog will only be shown in the case of a cluster install, and asks for a virtual IP to be used for the NetQ LoadBalancer (figure 6.17).

NetQ prompt for virtual IP to use for load balancing

Figure 6.17: NetQ prompt for virtual IP to use for load balancing

The dialog by default suggests an IP address that is not in use, according to NVIDIA Base Command Manager’s internal database. The address is part of the chosen internal network for the Kubernetes pod network.

After the wizard completes it input stage, the actual setup is executed. The setup can take significantly more time than the wizard input stage. The amount of time taken is largely dependent on file I/O speed—bigger files require some time to be synchronized to the appropriate nodes. In addition, NetQ installation itself can take around an hour to finish deploying. The deployment of NetQ is almost the very last step of the setup process. For a successful installation, the last stage of the output displayed looks similar to figure 6.18:

Kubernetes with NetQ installation finished, and relevant URLs on display

Figure 6.18: Kubernetes with NetQ installation finished, and relevant URLs on display

6.6.2 Accessing The NVIDIA NetQ Operator UI

In section 6.6.1 a NetQ deployment is described.

If the setup is for a single-node deployment, then a NodePort service is used to expose the NetQ UI. The URL is printed at the end of the setup (figure 6.18), but it can also be found by one of the following methods:

  • via kubectl

    root@basecm10 ~# module load kubernetes/default/1.27.11-1.1
    root@basecm10 ~# kubectl get svc -l app=netq-gui -n netq
    NAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
    netq-gui   NodePort   10.150.11.218   <none>        80:30029/TCP   17m

  • via searching back in the Kubernetes setup file:

    root@basecm10 ~# tac /var/log/cm-kubernetes-setup.log | grep -m 1 "NetQ GUI:"
        I 24-02-29 13:40:46 | cmsetup.plugins.kubernetes.stages.stages |   - NetQ GUI: \
        https://basecm10.openstacklocal:30029

If the setup is for a cluster deployment, then the virtual IP address also exposes the NetQ GUI on the default HTTPS port. For the cluster, that is: https://10.141.255.253. This virtual IP address is the one that is set during installation (figure 6.17). Figure 6.19 shows that IP address in use:

NetQ GUI screenshot

Figure 6.19: NetQ GUI screenshot

6.7 The Prometheus Operator Stack

6.7.1 Logging Into The Grafana Dashboard

Grafana’s admin credentials are not the cluster root password. When Grafana is deployed using the kube-prometheus-stack Helm chart, a random admin password is generated and stored, along with the admin username (admin by default), in a Kubernetes Secret. There is no cm-kubernetes-setup wizard screen for setting these credentials — they can be retrieved with:

Example

kubectl get secret -n prometheus kube-prometheus-stack-grafana \
  -o jsonpath='{.data.admin-user}' | base64 -d
kubectl get secret -n prometheus kube-prometheus-stack-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d

If no ingress controller was set up for the cluster, the Grafana dashboard can be reached by forwarding its service port to the local machine:

Example

kubectl port-forward -n prometheus svc/kube-prometheus-stack-grafana 3000:80

The dashboard is then available in a browser at http://localhost:3000, using the credentials retrieved in the preceding step.

If an ingress controller was enabled during the Kubernetes cluster setup, the dashboard may instead be reachable directly at a proxied URL configured for the cluster.

6.7.2 Exporting And Reusing Grafana Dashboards

After a dashboard has been created, the user may wish to export it for backup purposes, or to reuse it in another Grafana instance.

6.7.2.1 Exporting A Dashboard

Exporting the dashboard as a JSON file can be carried out by following the Grafana documentation instructions at:

https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels/#export-a-dashboard-as-json

6.7.2.2 Storing A Dashboard

The exported JSON file can be added to a Git repository, or stored in a ConfigMap. A ConfigMap can be created and labeled for Grafana with:

Example

kubectl create configmap my-grafana-dashboard --from-file=my-dashboard-1739813945293.json -n default
kubectl label configmap my-grafana-dashboard grafana_dashboard='1' -n default

By default, Grafana automatically loads dashboards when using ConfigMaps containing the

grafana_dashboard='1'

label in any namespace.

6.8 Run:ai

BCM dropped support for deploying Run:ai in version 10.32.0. Earlier BCM 10 releases packaged a Run:ai operator and integrated it into the cm-kubernetes-setup wizard. That operator and that integration have been removed, and the wizard no longer offers Run:ai as an option.

The reason is that the Run:ai deployment strategies have changed. They have been re-implemented for BCM 11, and that work has not been backported to BCM 10. BCM 11 therefore does have wizard integration for Run:ai, while BCM 10 does not.

Run:ai can still be installed on top of a BCM 10 Kubernetes cluster. It is then installed directly, by following the official Run:ai documentation rather than by using the BCM wizard:

6.9 Kubernetes Spark Operator

Using the Kubernetes Spark Operator is a simpler alternative to using the spark-submit tool for job submission.

6.9.1 Installing The Kubernetes Spark Operator

The Kubernetes Spark Operator can be installed as a part of the cm-kubernetes-setup procedure (section 4.2.6), which eventually leads to a display listing the operator packages that may be installed (figure 6.1).

The Kubernetes Spark Operator can alternatively be installed later on using the OS package manager and Helm:

[root@basecm10 ~]# yum install cm-kubernetes-spark-operator -y
[root@basecm10 ~]# helm install cm-kubernetes-spark-operator \
  /cm/shared/apps/kubernetes-spark-operator/current/helm/spark-operator-*.tgz

The Kubernetes Spark Operator can be removed with:

Example

[root@basecm10 ~]# helm uninstall cm-kubernetes-spark-operator

The operator installation state can be verified with --list-operators:

Example

[root@basecm10 ~]# cm-kubernetes-setup --list-operators
...
OPERATOR________________________________: api_available___________________________
cm-jupyter-kernel-operator              : 0
cm-kubernetes-postgresql-operator       : 0
cm-kubernetes-spark-operator            : 1
...

The Helm status can be checked with, for example:

Example

[root@basecm10 ~]#  helm list
NAME                            NAMESPACE  ...  STATUS     CHART                 APP VERSION
cm-kubernetes-spark-operator    default    ...  deployed   spark-operator-1.0.8  v1beta2-1.2.0-3.0.0
[root@basecm10 ~]#  helm status cm-kubernetes-spark-operator
NAME: cm-kubernetes-spark-operator
LAST DEPLOYED: Mon Sep 19 11:17:21 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
[root@basecm10 ~]#

The Permission Manager (section 4.15) and PodSecurityPolicy (PSP, section 4.10.2) must both be enabled for the cluster, before allowing a user to create resources in the Kubernetes cluster in their namespace:

Example

[root@basecm10 ~]# cm-kubernetes-setup --psp

The user alice can be allowed to use the Spark operator, and allowed to run a process as any UID in the pod:

Example

    [root@basecm10 ~]# cm-kubernetes-setup --add-user alice --operators cm-kubernetes-spark-operator \
    --allow-all-uids

The Kubernetes Spark operator Helm chart creates a CRD that can be used in the Kubernetes API.

For Alice, the CRD is available and can be used with a Spark operator YAML, to build a Spark application carry out a pi run in the restricted namespace.

6.9.2 Example Spark Operator Run: Calculating Pi

Continuing on with the user alice of the preceding section, a YAML file based on the specification at https://github.com/kubeflow/spark-operator/blob/v1beta2-1.6.2-3.5.0/examples/spark-py-pi.yaml can be used:

Example

    [root@basecm10 ~]# su - alice
[alice@basecm10 ~]$ module load kubernetes
[alice@basecm10 ~]$ cat <<EOF > pi-spark.yaml
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: pyspark-pi
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: "docker.io/apache/spark:3.5.1-python3"
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py
  sparkVersion: "3.5.1"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.5.1
    serviceAccount: spark
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 3.5.1
EOF
[alice@basecm10 ~]$ kubectl apply -f pi-spark.yaml
sparkapplication.sparkoperator.k8s.io/pyspark-pi created
[alice@basecm10 ~]$ kubectl get pods
NAME                READY   STATUS              RESTARTS   AGE
pyspark-pi-driver   0/1     ContainerCreating   0          1s
[alice@basecm10 ~]$ kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
pyspark-pi-driver   1/1     Running   0          3s
[alice@basecm10 ~]$ kubectl get sparkapplications
NAME         AGE
pyspark-pi   7s
[alice@basecm10 ~]$ kubectl get pods
NAME                               READY   STATUS              RESTARTS   AGE
pyspark-pi-driver                  1/1     Running             0          14s
pythonpi-e768128383a881b3-exec-1   0/1     ContainerCreating   0          0s
[alice@basecm10 ~]$ kubectl get pods
NAME                               READY   STATUS        RESTARTS   AGE
pyspark-pi-driver                  0/1     Completed     0          34s
pythonpi-e768128383a881b3-exec-1   0/1     Terminating   0          20s
[alice@basecm10 ~]$ kubectl get pods
NAME                READY   STATUS      RESTARTS   AGE
pyspark-pi-driver   0/1     Completed   0          36s

Instead of tracking the pod with:
kubectl get pods
as in the preceding session, or with the more convenient:
watch kubectl get pods
the pod could be tracked with the -f| --follow option to stream the driver logs:

Example

[alice@basecm10 ~]$ kubectl logs pyspark-pi-driver -f

To get intended output of the pi run—the calculated value of pi—it is sufficient to grep the log as follows:

Example

[alice@basecm10 ~]$ kubectl logs pyspark-pi-driver | grep ^Pi
Pi is roughly 3.148800

After the pi run has completed, the resources can be removed from the namespace:

[alice@basecm10 ~]$ kubectl delete -f pi-spark.yaml
sparkapplication.sparkoperator.k8s.io "pyspark-pi" deleted
[alice@basecm10 ~]$ kubectl get pods
No resources found in alice-restricted namespace.
[alice@basecm10 ~]$ kubectl get sparkapplications
No resources found in alice-restricted namespace.

6.10 The NVIDIA Postgres Operator

6.10.1 Installing The NVIDIA Postgres Operator

The upstream documentation for the Postgres Operator is at https://operatorhub.io/operator/postgres-operator.

6.10.1.1 Going Through The Kubernetes Setup Wizard

The Postgres operator installation is part of the initial Kubernetes cluster setup. Its installation is one of the options made available to the cluster administrator during the selection of operator packages (figure 6.1).

The wizard later asks which version of the Postgres operator should be installed (figure 6.20):

Postgres operator version selection screen

Figure 6.20: Postgres operator version selection screen