> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Kubernetes Operators Kubernetes operators are the modern way to manage Kubernetes cluster applications ([https://kubernetes.io/docs/concepts/extend-kubernetes/operator/](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)). It is usually recommended that Kubernetes operators are used instead of the legacy applications. At the time of writing of this section (March 2024), NVIDIA Base Command Manager provides and packages several operators which are validated to perform basic functionalities on a Kubernetes BCM setup. * the NVIDIA GPU Operator (section [6.4](#64-the-nvidia-gpu-operator)) * the NVIDIA Network Operator (section [6.5](#65-the-nvidia-network-operator)) * the NVIDIA NetQ Operator (section [6.6](#66-the-nvidia-netq-operator)) * the Prometheus Operator Stack (section [6.7](#67-the-prometheus-operator-stack)) * the Prometheus Adapter Operator * the Jupyter Kernel Operator (section [6.3](#63-the-jupyter-kernel-operator)) * the Spark Operator (section [6.9](#69-kubernetes-spark-operator)) * the Postgres Operator (section [6.10](#610-the-nvidia-postgres-operator)) * the MPI Operator * the MetalLB Operator * the Kubeflow Training Operator Run:ai is no longer one of them. Section [6.8](#68-runai) describes what changed, and how Run:ai can still be deployed on top of a BCM 10 Kubernetes cluster. ## 6.1 Versions Of Operators Available The versions of the operators that are available can be listed with the `cm-kubernetes-setup --list-operators-versions` command. The versions available at the time of writing (November 2024) for the operators are shown in the following table: | **Kubernetes Operator** | **Default** | **Possible versions** | | :------------------------- | :------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------- | | NVIDIA GPU Operator | 24.6.2(E) | 24.6.2(E), 24.6.1, 24.6.0, 24.3.0, 23.9.2, 23.9.1(E), 23.9.0(E), 23.6.2, 23.6.1(E), 23.6.0, 23.3.2(E), 23.3.1(E), 23.3.0, 22.9.2, 22.9.1(E), 22.9.0(E) | | MetalLB | 0.14.8 | 0.14.8 | | NetQ Operator | 4.9 (Kubernetes 1.27),4.10 (Kubernetes 1.28),4.11 (Kubernetes 1.29), 4.12 (Kubernetes 1.30), 4.13 (Kubernetes 1.31) | 4.9, 4.10, 4.11, 4.12, 4.13 | | Network Operator | 24.7.0 | 24.7.0, 24.4.1, 24.4.0, 23.10.0, 23.7.0(E), 23.5.0(E), 23.1.0(E) | | OVN CNI | 1.1.5 | 1.1.5 | | Postgres Operator | 1.13.0 | 1.13.0, 1.12.2, 1.10.1, 1.10.0, 1.9.0 | | Prometheus Adapter | 4.11.0 | 4.11.0 | | Prometheus Operator Stack | 62.7.0 | 62.7.0 | | SDN | 1.0.3 | 1.0.3 | | CM Jupyter Kernel Operator | 0.2.1 | 0.2.1 | | CM Kubernetes MPI Operator | 0.5.0 | 0.5.0 | | Spark Operator | 1.4.6 | 1.4.6 | (E) NVAIE-certified During the initial setup, the installation wizard displays a menu to select which operators are to be installed (figure [6.1](#fig-kube-urwid-wizard-operators-install-pkgs)). ![Kubernetes setup TUI session: selection of operator packages to be installed](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/b616d43d324af1f5497f47d708f82bbc7ef355b99bbe7a2af769ac12beb80a35/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/kube-urwid-wizard-operators-install-pkgs.png) *Figure 6.1: Kubernetes setup TUI session: selection of operator packages to be installed* After selecting the operators to be used, the actual operator versions available depend on package management resolution. In the case of BCM operators, the installation wizard uses a Helm chart from the OS package manager for some of the .deb or .rpm packages being deployed. In the case of cloud operators, the installation wizard uses a Helm chart from the Helm repository for deployment. ## 6.2 Helm Charts For The BCM Operators Based on the operators that are to be installed, the wizard asks for configuration options to the Helm charts that are to be installed (figure [6.2](#fig-kube-ncurses-wizard-operators-helm-install)): ![Kubernetes setup TUI session: prompting for Helm chart configuration options](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/d4dea4b87e769f7ea51a5b3955a75fd15d83ab8fe1b3912e3d74ef6d942a576a/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/kube-ncurses-wizard-operators-helm-install.png) *Figure 6.2: Kubernetes setup TUI session: prompting for Helm chart configuration options* The Helm charts that are selected are installed with sensible defaults. If additional tuning is needed, then the charts can be installed manually after `cm-kubernetes-setup` finishes: ```bash [root@basecm10 ~]# yum install cm-jupyter-kernel-operator -y [root@basecm10 ~]# helm install cm-jupyter-kernel-operator -n cm --wait \ /cm/shared/apps/jupyter-kernel-operator/current/helm/*.tgz ``` If additional tuning is required then tunable values can be set with a command line similar to the following: ```bash [root@basecm10 ~]# helm install cm-jupyter-kernel-operator -n cm --wait \ --values tunables.yaml \ /cm/shared/apps/jupyter-kernel-operator/current/helm/*.tgz ``` Possible values can be displayed as follows: ```bash [root@basecm10 ~]# helm show values /cm/shared/apps/jupyter-kernel-operator/current/helm/*.tgz ``` Installed operators can be listed by using the CLI option `--list-operators`: ```bash [root@basecm10 ~]# cm-kubernetes-setup --list-operators ... #### stage: kubernetes: Display Available Operators OPERATOR________________________________: api_available___________________________ cm-jupyter-kernel-operator : 1 postgresql-operator : 1 spark-operator : 1 cm-kubernetes-mpi-operator : 1 ... ``` ## 6.3 The Jupyter Kernel Operator ### 6.3.1 Installing The Jupyter Kernel Operator The Kubernetes Jupyter Kernel Operator can be installed as a part of the `cm-kubernetes-setup` procedure (section [4.2.6](/containerization/kubernetes#426-kubernetes-setup-from-a-tui-session)), which eventually leads to the selection screen displayed in figure [6.1](#fig-kube-urwid-wizard-operators-install-pkgs). The Kubernetes Jupyter Kernel Operator can alternatively be installed later on using the OS package manager and Helm: ```bash [root@basecm10 ~]# yum install cm-jupyter-kernel-operator -y [root@basecm10 ~]# helm install cm-jupyter-kernel-operator -n cm --create-namespace \ /cm/shared/apps/jupyter-kernel-operator/current/helm/cm-jupyter-kernel-operator-*.tgz ``` The chart installs into the Helm release namespace given with `-n` rather than shipping its own hardcoded namespace, so a namespace must be specified. Installations done through `cm-kubernetes-setup` are unaffected by this, since it already passes `--namespace cm --create-namespace`. When upgrading an existing installation with `helm upgrade`, the namespace given must be the release’s existing namespace, and not a hardcoded `cm-jupyter-kernel-operator-system`: an older chart version shipped its own Namespace object, and upgrading with a mismatched `-n` deletes the entire release, because the old chart’s Namespace object is pruned. If the target namespace is not known, the release should be uninstalled first, then installed again into the new namespace. The Jupyter Kernel Operator can be removed with: **Example** ```bash [root@basecm10 ~]# helm uninstall cm-jupyter-kernel-operator -n cm ``` It is recommended to enable a pod security policy using Kyverno (section [4.10.1](/containerization/kubernetes#4101-kyverno)) for the cluster before allowing a user, for example `alice`, to create resources in the Kubernetes cluster. The Kubernetes Jupyter Kernel Operator Helm chart creates a CRD that can be used in the Kubernetes API. To check the availability of the CRD, the following command can be run: **Example** ```bash [root@basecm10 ~]# module load kubernetes [root@basecm10 ~]# kubectl get crd | grep jupyterkernels cmjupyterkernels.apps.brightcomputing.com 2022-11-07T09:49:48Z cmkubernetesoperatorpermissionsjupyterkernels.charts.brightcomputing.com 2022-11-07T09:18:32Z ``` ### 6.3.2 Architecture Overview The Kubernetes Jupyter Kernel Operator has two main components: * the operator itself * the sidecar. This is attached to every user-defined kernel pod, and communicates with Jupyter Enterprise Gateway, acting as a proxy for the kernel process. The following is an overview of the kernel setup and pod lifecycle when the user runs the Kubernetes Jupyter Kernel Operator: 1. User initiates creating kernel in JupyterLab. 2. JupyterLab delegates this task to Jupyter Enterprise Gateway (JEG). 3. JEG opens a service TCP/IP socket and creates a CRD in Kubernetes specifying this port. 4. KubeApi notifies Jupyter Kernel Operator about the newly created CRD. 5. Jupyter Kernel Operator creates services, configmaps, secrets. 6. Jupyter Kernel Operator creates pod to run Jupyter kernel based on the specification. The sidecar is added to the kernel pod during this step. 7. The sidecar waits for the connection file created by the kernel. Alternatively, it relies on the connection file created by the operator (if requested), as not all kernels create a connection file. 8. The sidecar runs a proxy to forward kernel communications to JEG (stdin, shell, iopub, etc). 9. The sidecar notifies JEG about connection parameters and handles kernel communications. 10. If JEG disappears, or if communication drops, then the sidecar stops. This causes the kernel operator to get a notification via the KubeApi service. 11. The Kubernetes Jupyter Kernel Operator removes the unneeded pod, service, configmap and secrets. It also tries to gather stdout and std error of the kernel pod for debug purposes. The pod created in step 6 is heavily customized by the kernel operator. For security reasons, running a process inside the pod must be carried out as an unprivileged user. For the convenience of the Jupyter user, the UID/GID of the process inside the pod should match the UID/GID of the Jupyter user. If that is not the case, then the files created in the container are inaccessible for the Jupyter user. To achieve matching UID/GIDs, the operator dynamically creates `/etc/passwd` and `/etc/group` files inside the pod and populates them with the data from corresponding templates. At the same time the operator can create a kernel communication file, if requested—some kernels rely on that. ### 6.3.3 Running Jupyter Kernel Using The Operator An example of a basic YAML definition for the CMJupyterKernel is: ```bash [alice@basecm10 ~]$ cat cmjk.yaml --- apiVersion: apps.brightcomputing.com/v1 kind: CMJupyterKernel metadata: name: cmjk-test namespace: alice-restricted spec: username: alice uid: 1001 gid: 1001 kernel_id: testtesttest homedir: /home/alice pod: volumes: - name: homedir hostPath: path: /home/alice type: DirectoryOrCreate containers: - name: kernel image: jupyter/datascience-notebook command: - "python" args: - "-m" - "ipykernel_launcher" - "-f" - "/var/tmp/kernel-parm.json" workingDir: /home/alice securityContext: allowPrivilegeEscalation: false privileged: false runAsNonRoot: true runAsUser: 1001 runAsGroup: 1001 volumeMounts: - name: homedir mountPath: /home/alice ``` This can be submitted, but the operator removes it in approximately 1 minute: ```bash [alice@basecm10 ~]$ module load kubernetes [alice@basecm10 ~]$ kubectl apply -f cmjk.yaml ``` The logs of the operator can be checked for debug purposes: ```bash [root@basecm10 ~]# module load kubernetes [root@basecm10 ~]# kubectl logs \ -n cm \ -l control-plane=controller-manager \ --tail -1 \ -c manager ... 2022-02-14T18:26:01.005Z INFO controllers.CMJupyterKernel Container is stopped. Logs are below "cmjupyterkernel": "alice-restricted/cmjk-test" ... 2022-02-14T18:26:01.190Z INFO controllers.CMJupyterKernel cmjksidecar: 2022/02/14 18:26:00 Timeout receiving pings from server for 60 sec. Shutting down. "cmjupyterkernel": "alice-restricted/cmjk-test" ... ``` This indicates that the sidecar was stopped because there was no connection from Jupyter Enterprise Gateway to the kernel. This is expected, since the kernel has been run manually, and not using Jupyter. After the sidecar shutdown, the `kube-api` server notifies the operator, which, in turn, removes objects such as CMJupyterKernel, pods, and services. ### 6.3.4 Jupyter Kernel Operator Tunables *Table 6.1: Tunable Options for Jupyter Kernel Operators* | **Option** | **Description** | | :-------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kernel_id`^R | Kernel identifier (random UUID) given by Jupyter server | | `username`^O | Name of the user | | `uid`^R, `gid`^R, `homedir`^O, `usershell`^O | UID, GID, home directory and default shell of the user | | `image_os_flavor`^O | Defines template of `/etc/passwd` and `/etc/group` files, where `uid`, `gid`, `homedir`, and `usershell` will be added. Could be one of `ubuntu1604`, `ubuntu1804`, `ubuntu2004`, `ubuntu2404`, `rhel7`, `centos7`, `rhel8`, `centos8`, `sles12`, `sles15`. | | `etc_passwd`^O, `etc_group`^O | Custom content of the `/etc/passwd` or `/etc/group`, if necessary. | | `sidecar_command`^O, `sidecar_args`^O | Commands and arguments to run the sidecar. By default empty. Most of the arguments for the sidecar are passed via environment variables (section [6.3.5](#635-sidecar-arguments-and-environment-variables)). | | `kernel_connection_file_path`^O | Where to expect to find kernel connection file. Default: `/var/tmp/kernel-parm.json` | | `create_connection_file`^O | Does the operator need to create and populate kernel connection file before the pod starts? Default: `false` | | `spark_pod_template_path`^O, `spark_pod_template`^O | Options to store or override the Spark executor template | | `pod`^R | Kubernetes Pod definition | | `service`^O | Kubernetes Service definition | Legend: O: Optional R: Required ### 6.3.5 Sidecar Arguments And Environment Variables #### 6.3.5.1 Sidecar Arguments A timeout can be set as an argument for the sidecar. * `--timeout`: Defines how long, in seconds, that the sidecar waits for the Jupyter Enterprise Gateway proxy to connect before shutdown. Default: `60` #### 6.3.5.2 Environment Variables The following environment variables can be used by the sidecar: *Table 6.2: Environment Variables For The Sidecar*
**Environment Variable** **Description**
`CMJK_CONNECTION_FILE` Path to find a connection file. The sidecar uses the file to establish a connection to the kernel and to pass data between Jupyter Enterprise Gateway and the kernel. Default: `/var/tmp/kernel-parm.json` .
`CMJK_KERNEL_ID` Unique identifier of the kernel. Usually the UUID in table [6.1](#tunable-options-for-kernel-operators) .
`CMJK_SHELL_PORT` Proxy port to open to forward shell communication. Default: `5001` .
`CMJK_IOPUB_PORT` Proxy port to open to forward iopub communication. Default: `5002` .
`CMJK_STDIN_PORT` Proxy port to open to forward stdin communication. Default: `5003` .
`CMJK_CONTROL_PORT` Proxy port to open to forward control communication. Default: `5004` .
`CMJK_HB_PORT` Proxy port to open to forward heartbeat communication. Default: `5005` .
`CMJK_COMM_PORT` Proxy port to open to forward comm communication. Default: `5006` .
### 6.3.6 Running Spark-based Kernels In Jupyter Kernel Operator Jupyter integration for BCM provides a kernel template (`jupyter-eg-kernel-k8s-cmjkop-py-spark`) and a sample container image (`brightcomputing/jupyter-kernel-sample:k8s-spark-py39-2.0.0`) to run Jupyter kernels in a Spark environment. The image can be altered or created from scratch based on the scripts provided in `/cm/shared/examples/jupyter/kubernetes-kernel-image-spark-py39/`. The Jupyter kernel is not run directly. Instead, the kernel process is run and controlled by the `spark-submit` executable inside the container. Jupyter Kernel Operator alters the provided image based on the CRD definition. Spark-specific tunables are `spark_pod_template_path` and `spark_pod_template`. The operator creates a file inside of the Spark driver pod and puts the content of `spark_pod_template` in it. After that, `spark-submit` uses this file, via the `--spark.kubernetes.executor.podTemplateFile` configuration option, to create executor pods. ### 6.3.7 Example: Creating An R Kernel From The Kernel Template The Jupyter Kernel Operator can be used out-of-the-box to support more kernels. For example, an R kernel can be added. The official `jupyter/r-notebook` R image can be taken from the Jupyter project, from [https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-r-notebook](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-r-notebook). The default entry point cannot be used as that would start Jupyter notebook, while the aim for this section is to use the kernel only. Some exploratory investigation should reveal the command to start the kernel: The pod can be run interactively in a Jupyter notebook terminal by a user: ```bash kubectl run -i --tty testnotebook --image=jupyter/r-notebook --restart=Never -- bash ``` The kernel specifications can then be investigated: ```bash jupyter-kernelspec list Available kernels: ir /opt/conda/share/jupyter/kernels/ir python3 /opt/conda/share/jupyter/kernels/python3 ``` The `ir` kernel is what is of interest here. The command line that is used to start the kernel can be found: ```bash cat /opt/conda/share/jupyter/kernels/ir/kernel.json "argv": ["R", "--slave", "-e", "IRkernel::main()", "--args", "connection_file"], "display_name":"R", "language":"R" ``` Based on this information the Jupyter Kernel Operator CRD can be created for the user: ```bash cat cmjk-ir.yaml --- apiVersion: apps.brightcomputing.com/v1 kind: CMJupyterKernel metadata: name: cmjk-test namespace: alice-restricted spec: username: alice uid: 1001 gid: 1001 kernel_id: testtesttest homedir: /home/alice create_connection_file: true # R kernel expects connection file be created pod: volumes: - name: homedir hostPath: path: /home/alice type: DirectoryOrCreate containers: - name: kernel image: jupyter/r-notebook # image command: - "R" args: - "--slave" - "-e" - "IRkernel::main()" - "--args" - "/var/tmp/kernel-parm.json" # we have static connection file workingDir: /home/alice securityContext: allowPrivilegeEscalation: false privileged: false runAsNonRoot: true runAsUser: 1001 runAsGroup: 1001 volumeMounts: - name: homedir mountPath: /home/alice ``` There are several changes from the previous (section [6.3.3](#633-running-jupyter-kernel-using-the-operator)) YAML, and from the IR command line: * `create_connection_file: true` If this is not specified then the kernel complains with the following message during startup: `kernel: cannot open file '/var/tmp/kernel-parm.json': No such file or directory` This means that the kernel expected this file to be created before the start. * `image: jupyter/r-notebook` Another image needs to be used. * `args: ... "/var/tmp/kernel-parm.json"` The spec file has a fixed path and name, instead of `"connection_file``"` as in `kernel.json` earlier The resulting `cmjk-ir.yaml` file can be submitted to Kubernetes, but it will be removed by the operator after one minute, as it is not being started from the Jupyter Enterprise Gateway. The next step is to create a kernel template. The Python kernel can be used as a reference: ```bash cd /cm/shared/apps/jupyter/current cd lib/python*/site-packages/cm_jupyter_kernel_creator/kerneltemplates cp -pr jupyter-eg-kernel-k8s-cmjkop-py jupyter-eg-kernel-k8s-cmjkop-r ``` The files `meta.yaml`, `kernel.json`, and `templates/cmjk.yaml.j2` need to be changed in order to able to provide the correct image and command: ```bash vim jupyter-eg-kernel-k8s-cmjkop-r/meta.yaml vim jupyter-eg-kernel-k8s-cmjkop-r/kernel.json.j2 vim templates/cmjk.yaml.j2 ``` The changes that are applied should look similar to the following: ```bash diff -u jupyter-eg-kernel-k8s-cmjkop-py/kernel.json.j2 jupyter-eg-kernel-k8s-cmjkop-r/kernel.json.j2 --- jupyter-eg-kernel-k8s-cmjkop-py/kernel.json.j2 2022-01-25 21:13:52.000000000 +0100 +++ jupyter-eg-kernel-k8s-cmjkop-r/kernel.json.j2 2022-02-16 12:22:23.610382929 +0100 @@ -15,8 +15,8 @@ } }, "argv": [ - "python", - "-m", "ipykernel_launcher", - "-f", "/var/tmp/kernel-parm.json" + "R", + "--slave", "-e", "IRkernel::main()", + "--args", "/var/tmp/kernel-parm.json" ] } ``` ```bash diff -u jupyter-eg-kernel-k8s-cmjkop-py/meta.yaml jupyter-eg-kernel-k8s-cmjkop-r/meta.yaml --- jupyter-eg-kernel-k8s-cmjkop-py/meta.yaml 2022-01-25 21:13:52.000000000 +0100 +++ jupyter-eg-kernel-k8s-cmjkop-r/meta.yaml 2022-02-16 12:20:57.500974886 +0100 @@ -1,5 +1,5 @@ --- -display_name: "Python on Kubernetes Operator" +display_name: "R on Kubernetes Operator" features: "k8s-jupyter-operator-enabled" parameters: display_name: @@ -7,7 +7,7 @@ definition: getter: shell exec: - - echo "Python on Kubernetes Operator $(date +%y%m%d%H%M%S)" + - echo "R on Kubernetes Operator $(date +%y%m%d%H%M%S)" display_name: "Display name of the kernel" k8s_env_module: type: str @@ -20,9 +20,9 @@ definition: getter: static default: - - "jupyter/datascience-notebook" + - "jupyter/r-notebook" values: - - "jupyter/datascience-notebook" + - "jupyter/r-notebook" display_name: "Image to run" limits: max_len: 1 ``` ```bash diff -u jupyter-eg-kernel-k8s-cmjkop-py/templates/cmjk.yaml.j2\ jupyter-eg-kernel-k8s-cmjkop-r/templates/cmjk.yaml.j2 --- jupyter-eg-kernel-k8s-cmjkop-py/templates/cmjk.yaml.j2 2022-01-25 21:13:52.000000000 +0100 +++ jupyter-eg-kernel-k8s-cmjkop-r/templates/cmjk.yaml.j2 2022-02-16 12:24:25.375373991 +0100 @@ -9,6 +9,7 @@ gid: gid kernel_id: kernel_id homedir: homedir + create_connection_file: true pod: volumes: - name: homedir ``` After instantiating a kernel spec from the template, the R kernel is ready for use: ![Jupyter Kernel Creator: creating the IR kernel spec](/base-command-manager/product-manuals/_fern-img/e1f5b4bd156876b5a734cd3607143e06e311bd82cd39bf99c670c82514c6fddf.webp) *Jupyter Kernel Creator: creating the IR kernel spec* ![JupyterLab: running the IR kernel](/base-command-manager/product-manuals/_fern-img/621e0678df83338fee88e00811cbc275d563c2f63eff02f1f2d5525b0e70f2c6.webp) *JupyterLab: running the IR kernel* ### 6.3.8 Example: Letting Kubernetes Access Private Registries From The Kernel Template To be able to pull images from private registries, Kubernetes needs to be instructed about the credentials to use. #### 6.3.8.1 Creating The Secret This can be achieved by specifying the secret to `spec.imagePullSecrets` of the pod definition. For Jupter Kernel Operator this is `spec.pod.imagePullSecrets`: **Example** ```bash pod: containers: - name: kernel image: image ... imagePullSecrets: - name: regcred ... ``` Creating secrets can be carried out with `kubectl` **Example** ```bash kubectl create \ --namespace alice-restricted \ secret docker-registry regcred \ --docker-server= \ --docker-username= \ --docker-password= \ --docker-email= ``` More details about managing certificates can be found in the Kubernetes documentation at:\ [https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) #### 6.3.8.2 Parameterizing The Secret The name of the secret can be parameterized, so that users are allowed to select from secrets in their namespaces. ![Jupyter Kernel Creator: secret selection](/base-command-manager/product-manuals/_fern-img/f19fad98ca1afc11790c222fbddc52b61d143c0ec84e981b31f07d32de28d780.webp) *Jupyter Kernel Creator: secret selection* For parameterization, the `meta.yaml` and `kernel.json.j2` files must also then be modified: **Example** ```bash # cat meta.yaml ... parameters: ... image_pull_secret_name: type: str definition: getter: static default: "" display_name: "Name of the secret to pull images" ``` ```bash # cat kernel.json.j2 ... "metadata": { "process_proxy": { "class_name": "cm_jupyter_kernel_creator.eg_processproxies.k8scmjkop.KubernetesCMJupyterKernelOperator", "config": { ... "image_pull_policy": "{{ image_pull_policy[0] }}", "namespace": "{{ kubernetes_namespace }}", "image_pull_secret_name": "{{ image_pull_secret_name }}", "gpu_limit": {{ gpu_limit }} } ... ``` ```bash # cat templates/cmjk.yaml.j2 ... pod: ... {%- if image_pull_secret_name %} imagePullSecrets: - name: {{ image_pull_secret_name }} {%- endif %} containers: - name: kernel image: {{ image }} ... ``` ### 6.3.9 Example: Adding The PVC Parameter To The Kernel Template The PVC that is to be mounted can be set in the template: ![Jupyter Kernel Creator, PVC selection.](/base-command-manager/product-manuals/_fern-img/2db38ba1a358c3e72d6a6f38560f9388d812b6ddbead86efe08e8f17fa2296e4.webp) *Jupyter Kernel Creator, PVC selection.* The settings in `meta.yaml`, `kernel.json.j2` and `templates/cmjk.yaml.j2` for this are: **Example** ```bash # cat meta.yaml ... parameters: ... pvc_name: type: list definition: getter: shell exec: - source /etc/profile.d/modules.sh - module load kubernetes - kubectl get pvc -o jsonpath="{range .items[*]}{.metadata.name}{'\\n'}{end}" display_name: "PVC to mount" limits: max_len: 1 min_len: 1 pvc_mountpoint: type: str definition: getter: static default: "/data" display_name: "Mountpoint to PVC" ``` ```bash # cat kernel.json.j2 ... "metadata": { "process_proxy": { "class_name": "cm_jupyter_kernel_creator.eg_processproxies.k8scmjkop.KubernetesCMJupyterKernelOperator", "config": { ... "image_pull_policy": "{{ image_pull_policy[0] }}", "namespace": "{{ kubernetes_namespace }}", "pvc_name": "{{ pvc_name[0] }}", "pvc_mountpoint": "{{ pvc_mountpoint }}", "gpu_limit": {{ gpu_limit }} } ... ``` ```bash # cat templates/cmjk.yaml.j2 ... pod: volumes: ... {%- if pvc_name and pvc_mountpoint %} - name: pvc persistentVolumeClaim: claimName: {{ pvc_name }} {%- endif %} containers: - name: kernel image: {{ image }} imagePullPolicy: {{ image_pull_policy }} command: {{ command }} ... volumeMounts: - name: homedir mountPath: {{ homedir }} {%- if pvc_name and pvc_mountpoint %} - name: pvc mountPath: {{ pvc_mountpoint }} {%- endif %} ... ``` ## 6.4 The NVIDIA GPU Operator ### 6.4.1 Installing The NVIDIA GPU Operator The NVIDIA GPU operator can be installed as a part of the installation session by the `cm-kubernetes-setup` wizard (section [4.2.6](/containerization/kubernetes#426-kubernetes-setup-from-a-tui-session)). During the setup session, a checkbox can be checkmarked to install and enable the GPU operator (figure [6.1](#fig-kube-urwid-wizard-operators-install-pkgs)). Nodes that run DGX OS are also supported by the wizard. A specific version can be selected for the installation (figure [6.7](#fig-gpu-operator-version-select)): ![GPU operator version selection screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/8d784ea9e051284e4dc092c7928cf76d08f0c7cab134b249bce46e589f6d0ff8/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/gpu-operator-version-select.png) *Figure 6.7: GPU operator version selection screen* The NVIDIA GPU operator can also be deployed on an existing BCM Kubernetes cluster, as described next. ### 6.4.2 Installing The NVIDIA GPU Operator On An Existing Kubernetes Cluster If the NVIDIA GPU Operator ([https://docs.nvidia.com/datacenter/cloud-native/gpu-operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator)) is to be installed within an existing BCM Kubernetes cluster, then it must always be deployed using Helm. **Prerequisites:** If the existing cluster uses the NVIDIA device plugin add-on, even if configured by NVIDIA Base Command Manager, then it may be necessary to disable the add-on. This add-on is now deprecated, and will be removed in a future release. ```bash [cluster->kubernetes[default]->appgroups[system]->applications[nvidia]]% set enabled no [cluster->kubernetes*[default*]->appgroups*[system*]->applications*[nvidia*]]% commit ``` One of the prerequisites for the preceding add-on is that it uses labels to identify the nodes to be managed by the add-on. These labels are unnecessary for the GPU operator, and may be removed: ```bash [cluster->kubernetes[default]->labelsets]% remove nvidia [cluster->kubernetes*[default*]->labelsets*]% commit ``` **Installing The NVIDIA GPU Operator:** Validation methods are described for each step of the deployment. * For `containerd`, Helm installation is carried out by the root user with the following options: ```bash helm install --wait -n gpu-operator --create-namespace \ --version v1.10.1 \ --set driver.enabled=false \ --set operator.defaultRuntime=containerd \ --set toolkit.enabled=true \ --set toolkit.env[0].name=CONTAINERD_CONFIG \ --set toolkit.env[0].value=/cm/local/apps/containerd/var/etc/conf.d/nvidia-cri.toml \ gpu-operator nvidia/gpu-operator ``` * For `docker`, Helm installation is carried out by the root user with the following options: ```bash helm install --wait -n gpu-operator --create-namespace \ --version v1.10.1 \ --set driver.enabled=false \ --set operator.defaultRuntime=docker \ --set toolkit.enabled=true \ gpu-operator nvidia/gpu-operator ``` Helm chart options are documented at [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#common-chart-customization-options](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#common-chart-customization-options). **NVIDIA GPU Operator containerd configuration:** The operator provides the toolkit binaries and containerd configuration (`nvidia-cri.toml`) on each host where a GPU is auto-detected via a host-mount. The flag that enables this is `--set toolkit.enabled=true`. The path for the configuration file should be set to: `/cm/local/apps/containerd/var/etc/conf.d/nvidia-cri.toml`, which is where BCM’s `cm-containerd` package expects to find it. The operator provides a similar configuration functionality for the CUDA drivers. However this is not used in BCM, and it is disabled with the `--set driver.enabled=false` flag. This is because BCM supports CUDA on more Linux distributions and kernel versions than the NVIDIA GPU Operator does. CUDA drivers are therefore expected to already be present on the relevant nodes that have GPUs. **NVIDIA GPU Operator Docker configuration:** This is only relevant for older Kubernetes deployments that are deployed on top of Docker or BCM Docker. Default paths are used, so nothing particularly special has to be done for the operator to deploy properly. ### 6.4.3 Removing The NVIDIA GPU Operator The NVIDIA GPU Operator can be found in the `gpu-operator` namespace inside Helm and Kubernetes. ```bash root@basecm10 ~# helm list -n gpu-operator NAME NAMESPACE REVISION ... STATUS CHART APP VERSION gpu-operator gpu-operator 1 ... deployed gpu-operator-v1.10.1 v1.10.1 ``` A `helm uninstall gpu-operator` command can be used to uninstall the operator. ### 6.4.4 Validating The NVIDIA GPU Operator A pragmatic way to validate the NVIDIA GPU Operator is to check if the validator pods can be run. A `Running` status for the pods that are to have a GPU on them can be seen with: **Example** ```bash root@basecm10 ~# kubectl get pod -n gpu-operator -l app=nvidia-operator-validator -o wide NAME READY STATUS ... IP NODE NOMINATED NODE READINESS GATES nvidia-operator-validator-2qvz6 1/1 Running ... 172.29.152.172 node001 nvidia-operator-validator-xkwwv 1/1 Running ... 172.29.112.154 node002 ``` The preceding shows successfully-running pods. The log output should show `all validations are successful`: **Example** ```bash root@basecm10 ~# kubectl logs -n gpu-operator -l app=nvidia-operator-validator -c nvidia-operator-validator all validations are successful all validations are successful ``` ### 6.4.5 Validating The NVIDIA GPU Operator In Detail The set of pods associated with the NVIDIA GPU Operator can be examined in more detail. The following shows outputs from a GPU operator deployment that is working correctly: **Example** ```bash root@basecm10 ~# helm list -n gpu-operator NAME NAMESPACE ... STATUS CHART APP VERSION gpu-operator gpu-operator ... deployed gpu-operator-v1.10.1 v1.10.1 root@basecm10 ~# kubectl get all -n gpu-operator -o wide NAME READY STATUS RESTARTS ... NODE pod/gpu-feature-discovery-gk892 1/1 Running 0 ... node001 pod/gpu-feature-discovery-rmkvj 1/1 Running 0 ... node002 pod/gpu-operator-798c6ddc97-lmclm 1/1 Running 0 ... basecm10 pod/gpu-operator-node-feature-discovery-master-6c65c99969-cjlpq 1/1 Running 0 ... basecm10 pod/gpu-operator-node-feature-discovery-worker-cgxzl 1/1 Running 0 ... node002 pod/gpu-operator-node-feature-discovery-worker-ds5mb 1/1 Running 0 ... basecm10 pod/gpu-operator-node-feature-discovery-worker-jf65c 1/1 Running 0 ... node001 pod/nvidia-container-toolkit-daemonset-ffbk7 1/1 Running 1 (46m ago)... node002 pod/nvidia-container-toolkit-daemonset-lqfkq 1/1 Running 0 ... node001 pod/nvidia-cuda-validator-pxs9b 0/1 Completed 0 ... node001 pod/nvidia-cuda-validator-v7gfz 0/1 Completed 0 ... node002 pod/nvidia-dcgm-exporter-bxjrv 1/1 Running 0 ... node001 pod/nvidia-dcgm-exporter-qll9z 1/1 Running 0 ... node002 pod/nvidia-device-plugin-daemonset-698hd 1/1 Running 0 ... node001 pod/nvidia-device-plugin-daemonset-xd4kj 1/1 Running 0 ... node002 pod/nvidia-device-plugin-validator-5crlc 0/1 Completed 0 ... node001 pod/nvidia-device-plugin-validator-wh27x 0/1 Completed 0 ... node002 pod/nvidia-operator-validator-2qvz6 1/1 Running 0 ... node001 pod/nvidia-operator-validator-xkwwv 1/1 Running 0 ... node002 ... ``` On this particular example cluster, there are two compute nodes with GPUs, and there is one control plane node without a GPU: ```bash root@basecm10 ~# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready worker 3h2m v1.24.0 node002 Ready worker 3h2m v1.24.0 basecm10 Ready control-plane,master 3h2m v1.24.0 ``` **Feature discovery pods:** *Node Feature Discovery* (NFD, [https://intel.github.io/kubernetes-docs/nfd/index.html](https://intel.github.io/kubernetes-docs/nfd/index.html)) is an add-on that is initiated after the operator is installed. A master pod collects discovery information from the worker pods, and schedules more pods in case GPUs have been detected. In the preceding GPU operator output, * the master pod is running on `node001` with the name:\ `gpu-operator-node-feature-discovery-master-6c65c99969-wtzcx` * the worker pods run on each node. For example, the worker pod for `node002` is:\ `gpu-operator-node-feature-discovery-worker-z4skv` The output for the pods is not very verbose by default, but if more pods under the `nvidia-` namespace are scheduled on a node, besides the `gpu-operator-node-feature-discovery-*` pods, then that means that NFD has detected one or more GPUs. For example, a GPU discovered on node001 results in a scheduling of the following pods on that node: * container toolkit * device plugin * validator **Container toolkit pods:** For nodes that have GPUs, the NVIDIA container toolkit installation pods are started. Pod logs show exactly what is being installed. One of the requirements for the NVIDIA container toolkit installation pods is that the driver has to be in working order, or the `init` container `driver-validation` will fail. The following is the log from a successful installation: **Example** ```bash root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-container-toolkit-daemonset-ffbk7 Defaulted container "nvidia-container-toolkit-ctr" out of: nvidia-container-toolkit-ctr, driver-validation (init) ... time="2022-12-06T14:31:36Z" level=info msg="Installing toolkit" time="2022-12-06T14:31:36Z" level=info msg="Parsing arguments: [/usr/local/nvidia/toolkit]" time="2022-12-06T14:31:36Z" level=info msg="Successfully parsed arguments" time="2022-12-06T14:31:36Z" level=info msg="Installing NVIDIA container toolkit to '/usr/local/nvidia/toolkit'" ... time="2022-12-06T14:31:36Z" level=info msg="Installing NVIDIA container toolkit config '/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml'" time="2022-12-06T14:31:36Z" level=info msg="Setting up runtime" time="2022-12-06T14:31:36Z" level=info msg="Starting 'setup' for containerd" time="2022-12-06T14:31:36Z" level=info msg="Parsing arguments: [/usr/local/nvidia/toolkit]" time="2022-12-06T14:31:36Z" level=info msg="Successfully parsed arguments" time="2022-12-06T14:31:36Z" level=info msg="Loading config: /runtime/config-dir/nvidia-cri.toml" ... ``` **Device plugin pods:** The device plugin pods are started up next. These have the toolkit as a requirement. If the toolkit is not in working order, then the `init` container `toolkit-validation` fails. The following is the log from a successful startup: **Example** ```bash root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-device-plugin-daemonset-698hd Defaulted container "nvidia-device-plugin-ctr" out of: nvidia-device-plugin-ctr, toolkit-validation (init) 2022/12/06 14:32:20 Loading NVML 2022/12/06 14:32:20 Starting FS watcher. 2022/12/06 14:32:20 Starting OS watcher. 2022/12/06 14:32:20 Retreiving plugins. 2022/12/06 14:32:20 No MIG devices found. Falling back to mig.strategy=& 2022/12/06 14:32:20 Starting GRPC server for 'nvidia.com/gpu' 2022/12/06 14:32:20 Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia-gpu.sock 2022/12/06 14:32:20 Registered device plugin for 'nvidia.com/gpu' with Kubelet ``` The pod log output suggests that the GPU is now registered with the Kubelet as a resource. This can be checked by querying the Node resource: **Example** ```bash root@basecm10 ~# kubectl describe node node001 | grep nvidia nvidia.com/cuda.driver.major=520 nvidia.com/cuda.driver.minor=61 nvidia.com/cuda.driver.rev=05 nvidia.com/cuda.runtime.major=11 nvidia.com/cuda.runtime.minor=8 nvidia.com/gfd.timestamp=1670337142 nvidia.com/gpu.compute.major=7 nvidia.com/gpu.compute.minor=0 nvidia.com/gpu.count=1 nvidia.com/gpu.deploy.container-toolkit=true nvidia.com/gpu.deploy.dcgm=true nvidia.com/gpu.deploy.dcgm-exporter=true nvidia.com/gpu.deploy.device-plugin=true nvidia.com/gpu.deploy.driver=true nvidia.com/gpu.deploy.gpu-feature-discovery=true nvidia.com/gpu.deploy.node-status-exporter=true nvidia.com/gpu.deploy.operator-validator=true nvidia.com/gpu.family=volta nvidia.com/gpu.machine=OpenStack-Nova nvidia.com/gpu.memory=32768 nvidia.com/gpu.present=true nvidia.com/gpu.product=Tesla-V100-SXM3-32GB nvidia.com/mig.strategy=single nvidia.com/run.ai-swap.enabled=false nvidia.com/gpu: 1 nvidia.com/gpu: 1 ... ``` **Validator pods:** If anything goes wrong with either the driver, toolkit, CUDA, or the plugin, then validator pods are a good place to start looking. If all goes well, the main container outputs `all validations are successful`: **Example** ```bash root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 Defaulted container "nvidia-operator-validator" out of: nvidia-operator-validator, driver-validation (init), toolkit-validation (init), cuda-validation (init), plugin-validation (init) all validations are successful ``` It is possible for an `init` container to fail. The output for the container should then be checked. The following shows output from successful `init` containers: ```bash root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c driver-validation running command chroot with args [/run/nvidia/driver nvidia-smi] Tue Dec 6 15:32:14 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-SXM3... On | 00000000:00:06.0 Off | 0 | | N/A 32C P0 46W / 350W | 2MiB / 32768MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c toolkit-validation Tue Dec 6 14:32:16 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-SXM3... On | 00000000:00:06.0 Off | 0 | | N/A 32C P0 46W / 350W | 2MiB / 32768MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c cuda-validation time="2022-12-06T14:32:17Z" level=info msg="pod nvidia-cuda-validator-pxs9b is curently in Pending phase" time="2022-12-06T14:32:22Z" level=info msg="pod nvidia-cuda-validator-pxs9b is curently in Pending phase" time="2022-12-06T14:32:27Z" level=info msg="pod nvidia-cuda-validator-pxs9b is curently in Pending phase" time="2022-12-06T14:32:32Z" level=info msg="pod nvidia-cuda-validator-pxs9b have run successfully" root@basecm10 ~# kubectl logs -f -n gpu-operator nvidia-operator-validator-2qvz6 -c plugin-validation time="2022-12-06T14:32:33Z" level=info msg="pod nvidia-device-plugin-validator-5crlc is curently in Pending phase" time="2022-12-06T14:32:38Z" level=info msg="pod nvidia-device-plugin-validator-5crlc is curently in Pending phase" time="2022-12-06T14:32:43Z" level=info msg="pod nvidia-device-plugin-validator-5crlc have run successfully" ``` This also explains where the pods earlier on came from, the ones marked with status `Completed`. They are used as part of certain validation steps. Which `init` container prints out error messages should indicate where the problem lies—either with the CUDA drivers, or the toolkit, and so on. If the driver or toolkit is not validating correctly, then it may result in a lot of pods stuck in a `Pending` or an `Init` stage. Looking at what `init` container is associated with the stuck pod helps in diagnosing the problem. **DCGM exporter pods:** These pods expose metrics endpoints for scraping, and can be considered less critical. They are involved in GPU metrics collection, and can be utilized with, for example, Prometheus Operator Stack, or the Prometheus Adapter, for *horizontal pod autoscaling* based on GPU metrics. ### 6.4.6 Running A GPU Workload A GPU workload can be run with the following configuration: **Example** ```bash root@basecm10 ~# cat << EOF > gpu.yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod spec: restartPolicy: Never containers: - name: cuda-container image: nvidia/cuda:11.0.3-base-ubuntu20.04 command: ["nvidia-smi"] resources: limits: nvidia.com/gpu: 1 EOF root@basecm10 ~# kubectl create -f gpu.yaml pod/gpu-pod created ``` On a cluster with GPUs available, this pod should get scheduled, and should not stay stuck in the `Pending` phase. The preceding example just invokes `nvidia-smi` in the container. The output can be viewed to confirm that it worked: **Example** ```bash root@basecm10 ~# kubectl logs -f gpu-pod Tue Dec 6 15:08:03 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-SXM3... On | 00000000:00:06.0 Off | 0 | | N/A 34C P0 47W / 350W | 2MiB / 32768MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ``` ## 6.5 The NVIDIA Network Operator ### 6.5.1 Installing The NVIDIA Network Operator The upstream documentation for the NVIDIA Network Operator is quite extensive, and available at [https://docs.nvidia.com/networking/software/cloud-orchestration/index.html#network-operator](https://docs.nvidia.com/networking/software/cloud-orchestration/index.html#network-operator). #### 6.5.1.1 Going Through The Kubernetes Setup Wizard The Network operator installation is also part of the initial Kubernetes cluster setup. One of the steps in the wizard asks the cluster administrator to select operators from a list. The Network operator is one of them. If the Network operator is selected (figure [6.8](#fig-network-operator-selection)), then the setup wizard continues with some follow-up checks and questions. ![Kubernetes operators selection, with Network operator selected](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/883c90be86bcbced87b796abac2d93460cf9f6d02dc41cca195439a3a8d2eb91/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/network-operator-selection.png) *Figure 6.8: Kubernetes operators selection, with Network operator selected* The wizard then asks which version of the operator should be installed (figure [6.9](#fig-network-operator-version-select)): ![Network operator version selection screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/7111a2376a4800d6fddf9cc0bd4c04c6f6e77dec3472e7829cf28483faca1693/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/network-operator-version-select.png) *Figure 6.9: Network operator version selection screen* The wizard then prompts for a custom YAML configuration file to use for the Helm chart deployment (figure [6.10](#fig-network-operator-yaml-config)): ![Network operator prompt for custom YAML configuration for Helm](/base-command-manager/product-manuals/_fern-img/dca581b43a79c2e6d825ff553b1ee9677f9436dcfd8631b264f5cbd36480aa36.webp) *Figure 6.10: Network operator prompt for custom YAML configuration for Helm* If a custom YAML configuration file is not chosen, then options for a Helm chart can be set (figure [6.11](#fig-network-operator-options)): ![Network operator options for Helm chart prompt.](/base-command-manager/product-manuals/_fern-img/9848fd557f7d5bdfd68691c966d4e8badae49ad661fa2dd1ba7993a3bbeb2577.webp) *Figure 6.11: Network operator options for Helm chart prompt.* #### 6.5.1.2 Result Of The Network Operator Helm Chart If all goes well, then the Helm chart should be deployed successfully: **Example** ```bash root@basecm10 ~# module load kubernetes/default/1.27.11-1.1 root@basecm10 ~# helm list -A -a NAME NAMESPACE ... STATUS CHART APP VERSION local-path-provisioner cm ... deployedcm-kubernetes-local-path-provisioner-0.0.26 0.0.26 network-operator network-operator... deployednetwork-operator-23.7.0 v23.7.0 permissions-manager cm ... deployedcm-kubernetes-permissions-manager-0.4.8 0.4.8 root@bright:~# helm history -n network-operator REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Thu Feb 29 14:43:20 2024 deployed network-operator-23.7.0 v23.7.0 Install complete ``` ## 6.6 The NVIDIA NetQ Operator ### 6.6.1 NVIDIA NetQ Operator Installation #### 6.6.1.1 Prerequisites * The only supported OS is Ubuntu 20.04. * The number of NetQ nodes can be 1 or 3 nodes only. These must be regular compute nodes. * The NetQ nodes require at least 250 GiB of diskspace to be available. * The NetQ nodes require at least 64 GiB of memory. * The NVIDIA Base Command Manager default ports cannot be used. * Kyverno is not supported in combination with NetQ at the time of writing (March 2024) of this section. The default ports can be changed on the pctive and passive head nodes as follows. ```bash cm-cmd-ports --http 8083 --https 8084 && systemctl restart cmd ``` #### 6.6.1.2 Going Through The Kubernetes Setup Wizard The NetQ installation is part of the initial Kubernetes cluster setup. One of the steps in the wizard asks the cluster administrator to choose operators from a list of operators. NetQ is one of the possible operators. If the NetQ operator is selected (figure [6.12](#fig-netq-operator)), then the setup wizard continues with some follow-up checks and questions. ![Kubernetes NetQ Operator Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/c16076e3560c4977393404b25884857051ebad10cdbcc1b3ee0ac3959588423f/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/netq-operator.png) *Figure 6.12: Kubernetes NetQ Operator Selection* Node selection asks which nodes are to be assigned to NetQ (see figure [6.13](#fig-netq-nodes)). The options are: * for a single-deployment (this is NetQ terminology), one node should be selected * For a cluster-deployment (also NetQ terminology), three nodes should be selected ![NetQ nodes selection](/base-command-manager/product-manuals/_fern-img/4919e53cb78811bac2bcf70297d2bf8f3af3caf5cff0f5b6bea8b33e4de71aac.webp) *Figure 6.13: NetQ nodes selection* After node selection, the nodes are inspected to see if they meet the prerequisites for running NetQ. One potential issue it detects is if NVIDIA Base Command Manager is already running on the default ports. It suggests the commands to execute to fix that, (figure [6.14](#fig-netq-cmd-conflict)). NetQ prerequisites and how to carry out changing the default ports is described further in section [6.6.1.1](#6611-prerequisites). After carrying out the port changes, the wizard must be restarted from the beginning. ![Warning presented if NVIDIA Base Command Manager still uses default port 8081.](/base-command-manager/product-manuals/_fern-img/b27c0c2a80a6a323ab1bcddfbdce2f18e73a9d610f06e70ce17460be62f854de.webp) *Figure 6.14: Warning presented if NVIDIA Base Command Manager still uses default port 8081.* lf all prerequisites are met, then the next dialog asks for a few files. These have to be provided by NetQ and stored somewhere on the active head node (figure [6.15](#fig-netq-input-files)). ![Dialog asking for NetQ tarball and packages for agent and apps.](/base-command-manager/product-manuals/_fern-img/fa27c9898ab68dbcc4e4fba45a52afaf1a5d5cf97e9717d14870b4b8306fc564.webp) *Figure 6.15: Dialog asking for NetQ tarball and packages for agent and apps.* When the required files are downloaded, and saved to, for example, `/root`: **Example** ```bash root@basecm10 ~# ls -alh | grep -i netq -rw-r--r-- 1 root root 35M Jan 9 09:05 netq-agent_4.9.0-ub20.04u45~1703950858.128b0741e_amd64.deb -rw-r--r-- 1 root root 32M Jan 9 09:05 netq-apps_4.9.0-ub20.04u45~1703950858.128b0741e_amd64.deb -rw-r--r-- 1 root root 13G Jan 9 09:09 NetQ-bcm-4.9.0-SNAPSHOT.tgz ``` the dialog can then be filled in (figure [6.16](#fig-netq-input-files-filled-in)): ![Filled-in dialog asking for NetQ files.](/base-command-manager/product-manuals/_fern-img/7f0ca9f03ba44561c4e356eb852d41ba0b35d3e5a7a09a9fce5d0db0ddc90081.webp) *Figure 6.16: Filled-in dialog asking for NetQ files.* Depending on whether a single-node or cluster (three-node) setup is being done, the next dialog will only be shown in the case of a cluster install, and asks for a virtual IP to be used for the NetQ LoadBalancer (figure [6.17](#fig-netq-virtual-ip)). ![NetQ prompt for virtual IP to use for load balancing](/base-command-manager/product-manuals/_fern-img/c8499f03489af18602f2bb02052ce9febac68cc0097c6933626b40fcad25c06d.webp) *Figure 6.17: NetQ prompt for virtual IP to use for load balancing* The dialog by default suggests an IP address that is not in use, according to NVIDIA Base Command Manager’s internal database. The address is part of the chosen internal network for the Kubernetes pod network. After the wizard completes it input stage, the actual setup is executed. The setup can take significantly more time than the wizard input stage. The amount of time taken is largely dependent on file I/O speed—bigger files require some time to be synchronized to the appropriate nodes. In addition, NetQ installation itself can take around an hour to finish deploying. The deployment of NetQ is almost the very last step of the setup process. For a successful installation, the last stage of the output displayed looks similar to figure [6.18](#fig-netq-finished-setup): ![Kubernetes with NetQ installation finished, and relevant URLs on display](/base-command-manager/product-manuals/_fern-img/d78daa7ff92b44a9df4d290d1f374e439c640e94f4c761efb25b42fd1a68f9e3.webp) *Figure 6.18: Kubernetes with NetQ installation finished, and relevant URLs on display* ### 6.6.2 Accessing The NVIDIA NetQ Operator UI In section [6.6.1](#661-nvidia-netq-operator-installation) a NetQ deployment is described. If the setup is for a single-node deployment, then a NodePort service is used to expose the NetQ UI. The URL is printed at the end of the setup (figure [6.18](#fig-netq-finished-setup)), but it can also be found by one of the following methods: * via kubectl ```bash root@basecm10 ~# module load kubernetes/default/1.27.11-1.1 root@basecm10 ~# kubectl get svc -l app=netq-gui -n netq NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE netq-gui NodePort 10.150.11.218 80:30029/TCP 17m ``` * via searching back in the Kubernetes setup file: ```bash root@basecm10 ~# tac /var/log/cm-kubernetes-setup.log | grep -m 1 "NetQ GUI:" I 24-02-29 13:40:46 | cmsetup.plugins.kubernetes.stages.stages | - NetQ GUI: \ https://basecm10.openstacklocal:30029 ``` If the setup is for a cluster deployment, then the virtual IP address also exposes the NetQ GUI on the default HTTPS port. For the cluster, that is: [https://10.141.255.253](https://10.141.255.253). This virtual IP address is the one that is set during installation (figure [6.17](#fig-netq-virtual-ip)). Figure [6.19](#fig-netq-screenshot-gui) shows that IP address in use: ![NetQ GUI screenshot](/base-command-manager/product-manuals/_fern-img/550c1d041c5b12a3b86c7b015cf87ddfe73138ad8f7443bf2d7d87a5c6152572.webp) *Figure 6.19: NetQ GUI screenshot* ## 6.7 The Prometheus Operator Stack ### 6.7.1 Logging Into The Grafana Dashboard Grafana’s admin credentials are not the cluster root password. When Grafana is deployed using the kube-prometheus-stack Helm chart, a random admin password is generated and stored, along with the admin username (`admin` by default), in a Kubernetes `Secret`. There is no `cm-kubernetes-setup` wizard screen for setting these credentials — they can be retrieved with: **Example** ```bash kubectl get secret -n prometheus kube-prometheus-stack-grafana \ -o jsonpath='{.data.admin-user}' | base64 -d kubectl get secret -n prometheus kube-prometheus-stack-grafana \ -o jsonpath='{.data.admin-password}' | base64 -d ``` If no ingress controller was set up for the cluster, the Grafana dashboard can be reached by forwarding its service port to the local machine: **Example** ```bash kubectl port-forward -n prometheus svc/kube-prometheus-stack-grafana 3000:80 ``` The dashboard is then available in a browser at [http://localhost:3000](http://localhost:3000), using the credentials retrieved in the preceding step. If an ingress controller was enabled during the Kubernetes cluster setup, the dashboard may instead be reachable directly at a proxied URL configured for the cluster. ### 6.7.2 Exporting And Reusing Grafana Dashboards After a dashboard has been created, the user may wish to export it for backup purposes, or to reuse it in another Grafana instance. #### 6.7.2.1 Exporting A Dashboard Exporting the dashboard as a JSON file can be carried out by following the Grafana documentation instructions at: [`https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels/#export-a-dashboard-as-json`](https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels/#export-a-dashboard-as-json) #### 6.7.2.2 Storing A Dashboard The exported JSON file can be added to a Git repository, or stored in a ConfigMap. A ConfigMap can be created and labeled for Grafana with: **Example** ```bash kubectl create configmap my-grafana-dashboard --from-file=my-dashboard-1739813945293.json -n default kubectl label configmap my-grafana-dashboard grafana_dashboard='1' -n default ``` By default, Grafana automatically loads dashboards when using ConfigMaps containing the `grafana_dashboard='1'` label in any namespace. ## 6.8 Run:ai BCM dropped support for deploying Run:ai in version 10.32.0. Earlier BCM 10 releases packaged a Run:ai operator and integrated it into the `cm-kubernetes-setup` wizard. That operator and that integration have been removed, and the wizard no longer offers Run:ai as an option. The reason is that the Run:ai deployment strategies have changed. They have been re-implemented for BCM 11, and that work has not been backported to BCM 10. BCM 11 therefore does have wizard integration for Run:ai, while BCM 10 does not. Run:ai can still be installed on top of a BCM 10 Kubernetes cluster. It is then installed directly, by following the official Run:ai documentation rather than by using the BCM wizard: * SaaS: [https://run-ai-docs.nvidia.com/saas/getting-started/installation/install-using-helm](https://run-ai-docs.nvidia.com/saas/getting-started/installation/install-using-helm) * Self-hosted: [https://run-ai-docs.nvidia.com/self-hosted/getting-started/installation/install-using-helm](https://run-ai-docs.nvidia.com/self-hosted/getting-started/installation/install-using-helm) ## 6.9 Kubernetes Spark Operator Using the Kubernetes Spark Operator is a simpler alternative to using the `spark-submit` tool for job submission. ### 6.9.1 Installing The Kubernetes Spark Operator The Kubernetes Spark Operator can be installed as a part of the `cm-kubernetes-setup` procedure (section [4.2.6](/containerization/kubernetes#426-kubernetes-setup-from-a-tui-session)), which eventually leads to a display listing the operator packages that may be installed (figure [6.1](#fig-kube-urwid-wizard-operators-install-pkgs)). The Kubernetes Spark Operator can alternatively be installed later on using the OS package manager and Helm: ```bash [root@basecm10 ~]# yum install cm-kubernetes-spark-operator -y [root@basecm10 ~]# helm install cm-kubernetes-spark-operator \ /cm/shared/apps/kubernetes-spark-operator/current/helm/spark-operator-*.tgz ``` The Kubernetes Spark Operator can be removed with: **Example** ```bash [root@basecm10 ~]# helm uninstall cm-kubernetes-spark-operator ``` The operator installation state can be verified with `--list-operators`: **Example** ```bash [root@basecm10 ~]# cm-kubernetes-setup --list-operators ... OPERATOR________________________________: api_available___________________________ cm-jupyter-kernel-operator : 0 cm-kubernetes-postgresql-operator : 0 cm-kubernetes-spark-operator : 1 ... ``` The Helm status can be checked with, for example: **Example** ```bash [root@basecm10 ~]# helm list NAME NAMESPACE ... STATUS CHART APP VERSION cm-kubernetes-spark-operator default ... deployed spark-operator-1.0.8 v1beta2-1.2.0-3.0.0 [root@basecm10 ~]# helm status cm-kubernetes-spark-operator NAME: cm-kubernetes-spark-operator LAST DEPLOYED: Mon Sep 19 11:17:21 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None [root@basecm10 ~]# ``` The Permission Manager (section [4.15](/containerization/kubernetes#415-kubernetes-permission-manager)) and PodSecurityPolicy (PSP, section [4.10.2](/containerization/kubernetes#4102-podsecuritypolicy)) must both be enabled for the cluster, before allowing a user to create resources in the Kubernetes cluster in their namespace: **Example** ```bash [root@basecm10 ~]# cm-kubernetes-setup --psp ``` The user `alice` can be allowed to use the Spark operator, and allowed to run a process as any UID in the pod: **Example** ```bash [root@basecm10 ~]# cm-kubernetes-setup --add-user alice --operators cm-kubernetes-spark-operator \ --allow-all-uids ``` The Kubernetes Spark operator Helm chart creates a CRD that can be used in the Kubernetes API. For Alice, the CRD is available and can be used with a Spark operator YAML, to build a Spark application carry out a pi run in the restricted namespace. ### 6.9.2 Example Spark Operator Run: Calculating Pi Continuing on with the user `alice` of the preceding section, a YAML file based on the specification at [https://github.com/kubeflow/spark-operator/blob/v1beta2-1.6.2-3.5.0/examples/spark-py-pi.yaml](https://github.com/kubeflow/spark-operator/blob/v1beta2-1.6.2-3.5.0/examples/spark-py-pi.yaml) can be used: **Example** ```bash [root@basecm10 ~]# su - alice [alice@basecm10 ~]$ module load kubernetes [alice@basecm10 ~]$ cat < pi-spark.yaml apiVersion: "sparkoperator.k8s.io/v1beta2" kind: SparkApplication metadata: name: pyspark-pi spec: type: Python pythonVersion: "3" mode: cluster image: "docker.io/apache/spark:3.5.1-python3" imagePullPolicy: Always mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py sparkVersion: "3.5.1" restartPolicy: type: OnFailure onFailureRetries: 3 onFailureRetryInterval: 10 onSubmissionFailureRetries: 5 onSubmissionFailureRetryInterval: 20 driver: cores: 1 coreLimit: "1200m" memory: "512m" labels: version: 3.5.1 serviceAccount: spark executor: cores: 1 instances: 1 memory: "512m" labels: version: 3.5.1 EOF [alice@basecm10 ~]$ kubectl apply -f pi-spark.yaml sparkapplication.sparkoperator.k8s.io/pyspark-pi created [alice@basecm10 ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE pyspark-pi-driver 0/1 ContainerCreating 0 1s [alice@basecm10 ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE pyspark-pi-driver 1/1 Running 0 3s [alice@basecm10 ~]$ kubectl get sparkapplications NAME AGE pyspark-pi 7s [alice@basecm10 ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE pyspark-pi-driver 1/1 Running 0 14s pythonpi-e768128383a881b3-exec-1 0/1 ContainerCreating 0 0s [alice@basecm10 ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE pyspark-pi-driver 0/1 Completed 0 34s pythonpi-e768128383a881b3-exec-1 0/1 Terminating 0 20s [alice@basecm10 ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE pyspark-pi-driver 0/1 Completed 0 36s ``` Instead of tracking the pod with:\ `kubectl get pods`\ as in the preceding session, or with the more convenient:\ `watch kubectl get pods`\ the pod could be tracked with the `-f| --follow` option to stream the driver logs: **Example** ```bash [alice@basecm10 ~]$ kubectl logs pyspark-pi-driver -f ``` To get intended output of the pi run—the calculated value of pi—it is sufficient to grep the log as follows: **Example** ```bash [alice@basecm10 ~]$ kubectl logs pyspark-pi-driver | grep ^Pi Pi is roughly 3.148800 ``` After the pi run has completed, the resources can be removed from the namespace: ```bash [alice@basecm10 ~]$ kubectl delete -f pi-spark.yaml sparkapplication.sparkoperator.k8s.io "pyspark-pi" deleted [alice@basecm10 ~]$ kubectl get pods No resources found in alice-restricted namespace. [alice@basecm10 ~]$ kubectl get sparkapplications No resources found in alice-restricted namespace. ``` ## 6.10 The NVIDIA Postgres Operator ### 6.10.1 Installing The NVIDIA Postgres Operator The upstream documentation for the Postgres Operator is at [https://operatorhub.io/operator/postgres-operator](https://operatorhub.io/operator/postgres-operator). #### 6.10.1.1 Going Through The Kubernetes Setup Wizard The Postgres operator installation is part of the initial Kubernetes cluster setup. Its installation is one of the options made available to the cluster administrator during the selection of operator packages (figure [6.1](#fig-kube-urwid-wizard-operators-install-pkgs)). The wizard later asks which version of the Postgres operator should be installed (figure [6.20](#fig-postgres-operator-version-select)): ![Postgres operator version selection screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/927fb2d05025d37ab5c3d1e1f1a069237f35b99f30123d02856c4d9c9dd348a4/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/postgres-operator-version-select.png) *Figure 6.20: Postgres operator version selection screen*