> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # The Jupyter Notebook Environment Integration ## 18.1 Introduction This chapter covers the installation and usage of the Jupyter Notebook environment in BCM. An updated list of the supported Linux distributions and Jupyter functionalities can be found in the feature matrix at [https://service.bcm.nvidia.com/feature-matrix/](https://service.bcm.nvidia.com/feature-matrix/), under the `Feature` column, in the section for Jupyter features. An overview of the concepts and terminology follows. #### 18.1.0.1 What Is Jupyter Notebook? *Jupyter Notebook* ([https://jupyter-notebook.readthedocs.io/](https://jupyter-notebook.readthedocs.io/)), or Jupyter, is a client-server open-source application that provides a convenient way for a cluster user to write and execute *notebook documents* in an interactive environment. In Jupyter, a notebook document, or notebook, is content that can be managed by the application. Notebooks are organized in units called *cells* and can contain both executable code, as well as items that are not meant for execution. Items not meant for execution can be, for example: explanatory text, figures, formulas, or tables. Notebooks can also store the inputs and outputs of an interactive session. Notebooks can thus serve as a complete record of a user session, interleaving code with rich representations of resulting objects. These documents are encoded as JSON files and saved with the `.ipynb` extension. Since JSON is a plain text format, notebooks can be version-controlled, shared with other users and exported to other formats, such as HTML, LaTeX, PDF, and slide shows. #### 18.1.0.2 What Is A Notebook Kernel? A *notebook kernel* (often shortened to *kernel*) is a computational engine that handles the various types of requests in a notebook (e.g. code execution, code completions, inspection) and provides replies to the user ([https://jupyter.readthedocs.io/en/latest/projects/kernels.html](https://jupyter.readthedocs.io/en/latest/projects/kernels.html)). Usually kernels only allow execution of a single language. There are kernels available for many languages, of varying quality and features. #### 18.1.0.3 What Is JupyterHub? Jupyter on its own provides a single user service. *JupyterHub* ([https://jupyterhub.readthedocs.io/](https://jupyterhub.readthedocs.io/)) allows Jupyter to provide a multi-user service, and is therefore commonly installed with it. JupyterHub is an open-source project that supports a number of authentication protocols, and can be configured in order to provide access to a subset of users. #### 18.1.0.4 What Is JupyterLab? *JupyterLab* ([https://jupyterlab.readthedocs.io/](https://jupyterlab.readthedocs.io/)) is a modern and powerful interface for Jupyter. It enables users to work with notebooks and other applications, such as terminals or file browsers. It is open-source, flexible, integrated, and extensible. JupyterLab works out of the box with JupyterHub. It can be used to arrange the user interface to support a wide range of workflows in data science, scientific computing, and machine learning. JupyterLab is extensible with plugins that can customize or enhance any part of the interface. Plugins exist for themes, file editors, keyboard shortcuts, as well as for other components. #### 18.1.0.5 What Is A Jupyter Extension? Several components of the Jupyter environment can be customized in different ways with extensions. Some types of extensions are: * IPython extensions ([https://ipython.readthedocs.io/en/stable/config/extensions/#ipython-extensions](https://ipython.readthedocs.io/en/stable/config/extensions/#ipython-extensions)) * Jupyter Notebook server extensions ([https://jupyter-notebook.readthedocs.io/en/stable/extending/index.html](https://jupyter-notebook.readthedocs.io/en/stable/extending/index.html)) * JupyterLab extensions ([https://jupyterlab.readthedocs.io/en/stable/user/extensions.html](https://jupyterlab.readthedocs.io/en/stable/user/extensions.html)) Extensions are usually developed, bundled, released, installed, and enabled in different ways. Each extension provides a new functionality for a specific component. For example, JupyterLab extensions can customize or enhance any part of the JupyterLab user interface. Extensions can provide new themes, file viewers, editors and renderers for rich output in notebooks. They can also add settings, add keyboard shortcuts, or add items to the menu or command palette. #### 18.1.0.6 What Is Jupyter Enterprise Gateway? By default, Jupyter runs kernels locally, which can exhaust server resources. A resource manager can be used to deal with this issue. *Jupyter Enterprise Gateway* ([https://jupyter-enterprise-gateway.readthedocs.io/](https://jupyter-enterprise-gateway.readthedocs.io/)) (JEG) is a pluggable open-source framework that leverages local underlying resource managers, such as Slurm or Kubernetes, to distribute kernels across the compute cluster. In addition to scalability, JEG also provides an improved multi-user support, and more granular security for Jupyter. This makes Jupyter more useful for enterprise, scientific, and academic implementations. In BCM, all the technologies mentioned in these sections are combined to provide a powerful, customizable and user-friendly Jupyter Notebook web interface running on a lightweight, multi-tenant, multi-language, scalable and secure environment, ready for a wide range of enterprise scenarios. For convenience, in the following sections, *Jupyter* is generally used to collectively refer to Jupyter Notebook, JupyterHub, JupyterLab and Jupyter Enterprise Gateway. ## 18.2 Jupyter Environment Installation BCM distributes Jupyter via two packages: `cm-jupyter` and `cm-jupyter-local`. They are typically installed via the `cm-jupyter-setup` script (section [18.2.1](#1821-jupyter-setup)). `cm-jupyter` is installed in the `/cm/shared` directory, which is by default exported over NFS. As a result, Jupyter kernels can run on all the compute nodes, without a separate installation to those nodes. `cm-jupyter` provides Jupyter Notebook, JupyterHub, JupyterLab, Jupyter Enterprise Gateway and some extensions. `cm-jupyter-local` provides the JupyterHub system service (`cm-jupyterhub.service`), and is therefore designed to be installed only on the node exposing users to the web login page for Jupyter. For convenience, this node is called the *login node*. A login node is typically the head node, but any cluster node can be used. Since compute nodes are not reachable via a web interface by default, it is the responsibility of the cluster administrator to configure access to these nodes if they are configured to be login nodes while Jupyter runs. That is, login nodes that are compute nodes must have their access configured by assigning IP addresses, configuring the firewall, opening Jupyter ports, and so on. However, if the Jupyter login node is the head node, then BCM takes care of configuring the firewall to open the required ports and of ensuring that the resulting environment is working out of the box. #### 18.2.0.1 BCM Jupyter Extensions For a default deployment of Jupyter, BCM installs and enables the following extensions to the Jupyter environment: * Jupyter Addons: A Jupyter Notebook server extension that performs API calls to CMDaemon and manages other server extensions; * Jupyter Kernel Creator (section [18.5](#185-jupyter-kernel-creator-extension)): A Jupyter Notebook server extension that provides a new interactive and user-friendly way to create kernels; * Jupyter VNC (section [18.7](#187-jupyter-vnc-extension)): A Jupyter Notebook server extension that enables remote desktops with VNC from notebooks; * JupyterLab Tools: A JupyterLab extension that exposes BCM server extensions functionalities to the users and shows the Cluster View section; * Jupyter WLM Magic (section [18.8](#188-jupyter-wlm-magic-extension)): An IPython extension that simplifies scheduling of workload manager jobs from the notebook; * Jupyter Kubernetes Operators Manager (section [18.9](#189-jupyter-kubernetes-operators-manager)): An extension that integrates with Kubernetes clusters, and for which it provides basic overview and management features. ### 18.2.1 Jupyter Setup The `cm-jupyter-setup` script can be run on the head node of the cluster to deploy a working Jupyter environment with minimal effort. The script comes with BCM’s `cm-setup` package. It has no prerequisites, and can be run before or after configuring any resource manager, such as Kubernetes or Slurm. The `cm-jupyter-setup` script installs the two packages `cm-jupyter` and `cm-jupyter-local`. By default, the Jupyter environment initially contains only Jupyter’s default Python 3 kernel, which runs on the login node. During setup, an administrator can deploy the Jupyter login interface on multiple nodes to evenly distribute the load across them. In this case the administrator must configure a load balancer to route users’ requests across those nodes. These login nodes become members of the same `configurationoverlay`, and therefore share the same Jupyter configuration, such as port numbers, authenticator, and so on. By default, the Jupyter configuration file points to local SSL certificates. This means that if there are multiple Jupyter login interfaces, then each node uses its own SSL certificate. ### 18.2.2 Jupyter Architecture The default Jupyter architecture deployed by `cm-jupyter-setup` is shown in figure [18.1](#fig-jupyter-architecture). ![Jupyter architecture](/base-command-manager/product-manuals/_fern-img/d9e765f414fa9bcb6b400c23f4bd1b4976c7e2082369f9864ad0f0430d3821f0.webp) *Figure 18.1: Jupyter architecture* In this architecture, the `cm-jupyterhub.service` provided by `cm-jupyter-local` starts JupyterHub on the port specified by `c.JupyterHub.hub_port` (default: 8901) on a login node (typically the head node). JupyterHub then automatically spawns a dynamic proxy to route HTTP requests via BCM’s `cm-npm-configurable-http-proxy` package. The proxy is the only process that listens for clients’ requests on Jupyter’s public interface, as specified by `c.JupyterHub.port` (default: 8000). JupyterHub instructs the proxy on how requests should be dispatched by using its REST API. This is exposed at the port specified by `c.ConfigurableHTTPProxy.port` (default: 8902), typically defined within `c.ConfigurableHTTPProxy.api_url`. At this stage, users accessing Jupyter with their browsers are either redirected to the shared Hub (e.g. for authentication), or to their dedicated single-user servers to run notebooks. By default, single-user servers are accessed at `/user/` \<*username*>, while JupyterHub is accessed at `/hub` ([https://jupyterhub.readthedocs.io/en/stable/reference/technical-overview.html](https://jupyterhub.readthedocs.io/en/stable/reference/technical-overview.html)). The `cm-jupyter-setup` script automatically installs JupyterLab and sets `/lab` as the default URL (`c.Spawner.default_url`) to redirect users to the new interface. Finally, Jupyter Enterprise Gateway is integrated to spawn kernels on different nodes by configuring `jupyterhub.spawner.LocalProcessSpawner` as the default spawning mechanism (`c.JupyterHub.spawner_class`), and `jupyterhub-singleuser-gw` as the default spawning command (`c.Spawner.cmd`). When a new notebook is started, JEG scans for an available port, and spawns a kernel chosen by the user on an appropriate cluster node. This process is then connected to JupyterLab. JupyterHub and its HTTP proxy are run as two root processes, while JupyterLab and Jupyter Enterprise Gateway are run as user processes. The privileges of the kernels spawned by Jupyter Enterprise Gateway can be configured by cluster administrators, and depend on the underlying computational engine (e.g. Kubernetes). By default, all the kernels configurable by BCM are run as user processes, and no privilege escalation is possible. Communication between users’ browsers, JupyterHub, its HTTP proxy, and JupyterLab, is secured by default by JupyterHub. On the other hand, communication between JupyterLab, Jupyter Enterprise Gateway, and kernels, is secured by BCM. Administrators can customize their Jupyter integration by setting some of the aforementioned options when the `cm-jupyter-setup` script runs. New values are automatically handled by BCM and written to Jupyter configuration files. Other configuration options can be found in `/cm/local/apps/jupyter/current/conf/jupyterhub_config.py`. ### 18.2.3 Verifying Jupyter Installation The `cm-jupyter-setup` script automatically starts the `cm-jupyterhub` service. Any user (not necessarily root) can then verify the installation is working as expected. Here an ordinary user, `jupyterhubuser` runs the checks. It can take some time until the service is fully up and running, even if `systemctl status cm-jupyterhub -l` shows that the service is active: **Example** ```bash [jupyterhubuser@basecm10 ~]$ systemctl status cm-jupyterhub -l cm-jupyterhub.service - JupyterHub Loaded: loaded (/usr/lib/systemd/system/cm-jupyterhub.service; static; vendor preset: disabled) Active: active (running) since Thu 2022-11-03 17:30:13 CET; 24min ago Main PID: 16162 (run.sh) Tasks: 11 (limit: 23269) Memory: 75.8M CGroup: /system.slice/cm-jupyterhub.service |-16162 /bin/bash /cm/shared/apps/jupyter/14.0.0/bin/run.sh \mbox{ |-16178 /cm/local/apps/python39/bin/python3 /cm/shared/apps/jupyter/14.0.0/bin/jupyterhub --Jupyter>} |-16179 tee -a /var/log/jupyterhub.log \mbox{ '-16191 node /cm/shared/apps/jupyter/14.0.0/bin/configurable-http-proxy --ip --port 8000 --api-ip 1>} ``` A check can then be done to see that the Jupyter extensions provided by BCM are installed and enabled: **Example** ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter Loading jupyter/15.2.0 Loading requirement: python39 [jupyterhubuser@basecm10 ~]$ jupyter serverextension list config dir: /cm/shared/apps/jupyter/15.2.0/etc/jupyter jupyter_server_ydoc enabled - Validating... X is jupyter_server_ydoc importable? jupyterlab enabled - Validating... jupyterlab 3.6.7 OK cm_jupyter_kernel_creator enabled - Validating... X is cm_jupyter_kernel_creator importable? cm_jupyter_addons enabled - Validating... X is cm_jupyter_addons importable? cm_jupyter_vnc enabled - Validating... X is cm_jupyter_vnc importable? cm_jupyter_kubernetes_operators_manager enabled - Validating... X is cm_jupyter_kubernetes_operators_manager importable? [jupyterhubuser@basecm10 ~]$ jupyter labextension list JupyterLab v3.1.19 /cm/shared/apps/jupyter/15.2.0/share/jupyter/labextensions jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments) @brightcomputing/jupyterlab-tools v0.2.7 enabled OK (python, brightcomputing_jupyterlab_tools) ``` ### 18.2.4 Login Configuration #### 18.2.4.1 User Access To JupyterHub With The Default Configuration Once JupyterHub is functioning, its web login interface is accessible with a browser using the HTTPS protocol on the specified port (figure [18.2](#fig-jupyter-home)): ![JupyterHub login screen](/base-command-manager/product-manuals/_fern-img/8dd72f853527d8de794ef409bd69b9d7d11482fde06dd414cb6185b65908a7fd.webp) *Figure 18.2: JupyterHub login screen* Running as root under JupyterHub is not recommended, so logging in to JupyterHub as root is not allowed by default. Any other PAM user can log in to JupyterHub. If needed, a test user `jupyterhubuser` with password `jupyterhubuser` can be created with, for example: **Example** ```bash [root@basecm10 ~]# cmsh -c "user; add jupyterhubuser; set password jupyterhubuser; commit" ``` #### 18.2.4.2 Restricting User Access To JupyterHub JupyterHub logins can be limited to members of particular groups. For example, `alice` could be made a member of the group `jupyterusers`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% group [basecm10->group]% add jupyterusers [basecm10->group]*[jupyterusers*]% append members alice [basecm10->group]*[jupyterusers*]% commit ``` The group `jupyterusers` could then be set to be allowed to authenticate to JupyterHub with an authentication policy: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% configurationoverlay [basecm10->configurationoverlay]% use jupyterhub [basecm10->configurationoverlay[jupyterhub]]% roles [basecm10->configurationoverlay[jupyterhub]->roles]% use jupyterhub [basecm10->configurationoverlay[jupyterhub]->roles[jupyterhub]]% configs [basecm10->...roles[jupyterhub]->configs]% add c.BrightAuthenticator.groups_allow [basecm10->...roles*[jupyterhub*]->configs*[c.BrightAuthenticator.groups_allow*]]% set value [\"jupyterusers\"] [basecm10->...roles*[jupyterhub*]->configs*[c.BrightAuthenticator.groups_allow*]]% commit ``` Similarly, restricting authentication to JupyterHub to a specific group of users can be done by configuring `c.BrightAuthenticator.groups_deny`. Another way to restrict user access is based on the CMDaemon profile set for the user. In that case the JupyterHub configuration settings are `c.BrightAuthenticator.cmd_profiles_allow` and `c.BrightAuthenticator.cmd_profiles_deny`. In the following example, users with a `readonly` profile are not allowed to authenticate for JupyterHub. **Example** ```bash [basecm10->...roles[jupyterhub]->configs[c.BrightAuthenticator.groups_allow]]% .. [basecm10->...roles[jupyterhub]->configs]% add c.BrightAuthenticator.cmd_profiles_deny [basecm10->...roles*[jupyterhub*]->configs*[c.BrightAuthenticator.cmd_profiles_deny*]]% set value [\"readonly\"] [basecm10->...roles*[jupyterhub*]->configs*[c.BrightAuthenticator.cmd_profiles_deny*]]% commit ``` A table summarizing the authentication policies is: | **Key** | **Value** | | :----------------------------------------- | :---------------------------------------------------- | | `c.BrightAuthenticator.groups_allow` | users in the specified groups can authenticate | | `c.BrightAuthenticator.groups_deny` | users in the specified groups cannot authenticate | | `c.BrightAuthenticator.cmd_profiles_allow` | users with the specified profiles can authenticate | | `c.BrightAuthenticator.cmd_profiles_deny` | users with the specified profiles cannot authenticate | ### 18.2.5 JupyterHub Screen After Login After the first login, a new single-user server is spawned (figure [18.3](#fig-jupyter-starting-user-server)): ![JupyterHub starting single-user server](/base-command-manager/product-manuals/_fern-img/9ae3405db95f1473c02140c03a7e9063e37c58e0329702a658b851b9a49276a7.webp) *Figure 18.3: JupyterHub starting single-user server* Users are redirected to the JupyterLab interface and have access to Jupyter’s default Python 3 kernel (figure [18.4](#fig-jupyter-without-kernels)): ![JupyterLab Launcher](/base-command-manager/product-manuals/_fern-img/c1a96bf01b1188e62b5e0349ca464e57ef8cf50eb69017adf47e820250d59cd4.webp) *Figure 18.4: JupyterLab Launcher* If using Kubernetes under Jupyter, then a user registered under the Linux-PAM system must be added separately via Kubernetes with `cm-kubernetes-setup`. ([section 4.11 of the *Containerization Manual*](/containerization/kubernetes#411-addition-of-new-kubernetes-users)). ## 18.3 Jupyter Notebook Examples The `cm-jupyter` package (section [18.2.1](#1821-jupyter-setup)) provides a number of machine learning notebook examples that can be executed with Jupyter. The notebooks include some applications developed with TensorFlow, PyTorch, MXNet, and other frameworks. The applications can be found in the `/cm/shared/examples/jupyter/notebooks/` directory: ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/examples/jupyter/notebooks/ Keras+TensorFlow2-addition.ipynb R-iris.ipynb TensorFlow-minigo.ipynb MXNet-superresolution.ipynb Spark-pipeline.ipynb Pytorch-cartpole.ipynb Spark+XGBoost-mortgage.ipynb ``` The datasets needed to execute these notebooks can be found in the `/cm/shared/examples/jupyter/datasets/` directory: ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/examples/jupyter/datasets/ 880f8b8a6fd-mortgage-small.tar.gz kaggle-iris.csv ``` Users can copy these examples to their home directories, create or choose appropriate kernels to execute them, and interactively run them from Jupyter. In order to edit notebooks, the write permissions must be kept during the copy. The distributed examples typically only require the packages provided by BCM, such as TensorFlow and PyTorch. It is the responsibility of users to make sure that the required modules are loaded by their Jupyter kernels. The list of frameworks and libraries required to run an example is usually available at the beginning of each notebook. ## 18.4 Jupyter Kernels In Jupyter, kernels are defined as JSON files. Any user that the cluster administrator has registered in the Linux-PAM system can list the available Jupyter kernels via the command line. The following example is run in the initial Jupyter environment: **Example** ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ jupyter kernelspec list Available kernels: python3 /cm/shared/apps/jupyter/14.0.0/share/jupyter/kernels/python3 ``` Each kernel directory contains a `kernel.json` file describing how Jupyter spawns that kernel: **Example** ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/apps/jupyter/current/share/jupyter/kernels/*/kernel.json /cm/shared/apps/jupyter/current/share/jupyter/kernels/python3/kernel.json ``` In addition to specifications for shared kernels, each user can define new personal ones in the home directory. By default, the Jupyter data directory for a user is located at `$HOME``/.local/share/jupyter`. This path can be verified with Jupyter by using the `--paths` option: **Example** ```bash [jupyterhubuser@basecm10 ~]$ jupyter --paths config: /home/jupyterhubuser/.jupyter /home/jupyterhubuser/.local/etc/jupyter /cm/local/apps/python39/etc/jupyter /cm/shared/apps/jupyter/14.0.0/etc/jupyter /etc/jupyter data: /home/jupyterhubuser/.local/share/jupyter /cm/local/apps/python39/share/jupyter /cm/shared/apps/jupyter/14.0.0/share/jupyter /usr/share/jupyter runtime: /home/jupyterhubuser/.local/share/jupyter/runtime ``` The simplest definition for a Python3 kernel designed to run on the login node is: ```bash { "argv": ["python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python" } ``` In the preceding kernel definition: * `argv`: is the command to be executed to locally spawn the kernel * `"display_name`": is the name to be displayed in the JupyterLab interface * `"language"`: is the supported programming language (`"language"`) * `"{connection_file}"` ([https://jupyter-client.readthedocs.io/en/stable/kernels.html#connection-files](https://jupyter-client.readthedocs.io/en/stable/kernels.html#connection-files)) is a placeholder, and is replaced by Jupyter with the actual path to the connection file before starting the kernel. The following kernel is Jupyter’s default Python 3 kernel distributed by BCM in the initial environment: **Example** ```bash [jupyterhubuser@basecm10 ~]$ cat /cm/shared/apps/jupyter/current/share/jupyter/kernels/python3/kernel.json "argv": [ "/cm/local/apps/python37/bin/python3.9", "-m", "ipykernel_launcher", "--InteractiveShellApp.extra_extension=cm_jupyter_wlm_magic", "--TerminalIPythonApp.extra_extension=cm_jupyter_wlm_magic", "-f", "connection_file" ], "display_name": "Python 3", "language": "python", "env": "PYTHONPATH": "/cm/shared/apps/jupyter/current/lib64/python3.9/site-packages:/cm/shared/apps/jupyter/ current/lib/python3.9/site-packages" ``` The two kernels are not very different. They differ from each other in the Python 3 binary path, the IPython extension (Jupyter WLM Magic), and the exported `PYTHONPATH` environment variable (`"env"`). ### 18.4.1 Jupyter Enterprise Gateway Kernels Jupyter is designed to run both the kernel processes, as well as the user interface (JupyterLab or Jupyter Notebook) on the same host. The kernel `{connection_file}` is therefore stored in the `~/.local/share/jupyter/runtime` directory, or in the `/run` directory. JupyterLab can delegate the task of spawning kernels to another component. During the BCM installation it is Jupyter Enterprise Gateway (JEG) that takes on this role, performing a complete lifecycle management of the kernel. That is: start, status monitoring, and termination. JEG requires an extended `kernel.json` definition to describe a particular process-proxy module to handle the kernel. A simple definition for a Python3 kernel designed to be scheduled via JEG is: ```bash { "argv": ["python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.processproxy.LocalProcessProxy", "config": {} } } } ``` In this example, the `"metadata"` entry has been added. It includes `"class_name"`, which defines a path to a Python class compatible with the JEG API; and `"config"`, an initial arguments dictionary passed to the JEG process. JEG can use several pluggable modules that are included in the BCM environment to interact with a wide range of resource managers, such as Kubernetes or Slurm. By doing so, kernels can be scheduled across compute nodes. BCM recommends that JEG kernels are created and used with the `Jupyter Kernel Creator` (section [18.5](#185-jupyter-kernel-creator-extension)) extension. ## 18.5 Jupyter Kernel Creator Extension Creating or editing kernels can be cumbersome and error-prone for users, depending on the features of the execution context desired for their notebooks. To provide a more user-friendly experience, BCM includes the *Jupyter Kernel Creator* extension in JupyterLab. This extension is accessed from the navigation pane in the JupyterLab interface, by clicking on the BCM icon. Jupyter Kernel Creator allows users to create kernels using the JupyterLab interface, without the need to directly edit JSON files. With this interface users can create kernels by customizing an available *template* according to their needs. A template can be considered to be the skeleton of a kernel, with several preconfigured options, and others options that are yet to be specified. Common customizations for templates include environment modules to be loaded, workload manager queues to be used, number and type of GPUs to acquire, and so on. Templates are usually defined by administrators according to cluster capabilities, programming languages and user requirements. Each template can provide different options for customizations. Administrators often create different templates to take advantage of different workload managers, programming languages and hardware resources. For example, an administrator may define a template for scheduling Python kernels via Kubernetes, another one for R kernels via Slurm, and yet another one for Bash kernels via Platform LSF. ### 18.5.1 BCM Predefined Kernel Templates To simplify Jupyter configuration for administrators, BCM distributes a number of pre-defined templates with Jupyter Kernel Creator. These templates can be used for default configurations of BCM workload managers, and can be customized and extended for more advanced use. Kernel templates defined by BCM can be found in the `Jupyter Kernel Creator` installation directory, under the `kerneltemplates` directory: ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/apps/jupyter/current/lib/python3.9/site-packages/\ cm_jupyter_kernel_creator/kerneltemplates/ filter.yaml jupyter-eg-kernel-k8s-py-spark jupyter-eg-kernel-pbspro-py39 jupyter-eg-kernel-k8s-bash jupyter-eg-kernel-k8s-r jupyter-eg-kernel-slurm-bash jupyter-eg-kernel-k8s-cmjkop-julia jupyter-eg-kernel-lsf-bash jupyter-eg-kernel-slurm-py39 jupyter-eg-kernel-k8s-cmjkop-ngc-py jupyter-eg-kernel-lsf-py39 jupyter-eg-kernel-slurm-pyxis-py39 jupyter-eg-kernel-k8s-cmjkop-py jupyter-eg-kernel-openpbs-bash jupyter-eg-kernel-uge-bash jupyter-eg-kernel-k8s-cmjkop-py-spark jupyter-eg-kernel-openpbs-py39 jupyter-eg-kernel-uge-py39 jupyter-eg-kernel-k8s-py jupyter-eg-kernel-pbspro-bash jupyter-eg-kernel-slurm-py-conda ``` #### 18.5.1.1 BCM Predefined Kernel Templates Seen By Users Users can view the available predefined kernel templates in the Jupyter web browser interface, within the `KERNEL TEMPLATES` section of the dedicated BCM extensions panel (figure [18.5](#fig-jupyter-template-list)): ![JupyterLab BCM extensions section with kernel templates](/base-command-manager/product-manuals/_fern-img/19eea8f2206916f2281bc4b172d999373eb903c639c7f1cf72a9a58c5e4058a7.webp) *Figure 18.5: JupyterLab BCM extensions section with kernel templates* However, the templates provided by BCM are listed in the panel only if they can be used on the cluster. This means that the templates are listed only after the associated workload manager instance, or associated Kubernetes configuration (such as a Kubernetes operator), have been deployed by cluster manager utilities. For example: * After running the `cm-wlm-setup` cluster manager utility to deploy an OpenPBS workload manager, the `jupyter-eg-kernel-openpbs-bash` and `jupyter-eg-kernel-openpbs-py39` templates become available. The templates are listed as: * `Bash via OpenPBS` * `Python 3.9 via OpenPBS` and accessed via the navigation path: *menu* > *dedicated BCM extension panel* > *kernel templates section*. * After running the `cm-kubernetes-setup` cluster manager utility to deploy a Kubernetes cluster, the Kubernetes cluster instance is displayed (navigation path: *menu* > *dedicated BCM extension panel* > *Kubernetes clusters section*) Then, after running the `cm-jupyter-kernel-operator` cluster manager utility to deploy a Jupyter kernel operator package, and configuring a user ([section 6.3 of the *Containerization Manual*](/containerization/kubernetesoperators#63-the-jupyter-kernel-operator)), the templates `jupyter-eg-kernel-k8s-cmjkop-julia`, `jupyter-eg-kernel-k8s-cmjkop-py`, and `jupyter-eg-kernel-k8s-cmjkop-py-spark` become available. The templates are listed as: * `Julia on Kubernetes Operator` * `Python on Kubernete Operator` * `Python+Spark on Kubernetes Operator` and accessed via the navigation path: *menu* > *dedicated BCM extension panel* > *kernel templates section*. Users can instantiate a kernel template to create an actual kernel from the dedicated BCM extensions section using the `+` button of the template. A dialog is dynamically generated for the template being instantiated, and users are asked to fill a number of customization options defined by administrators (figure [18.6](#fig-jupyter-template-customization)): ![Jupyter kernel template customization screen](/base-command-manager/product-manuals/_fern-img/45f7592c0a838e12aa2fddcb8a9a4d37e28c2414d90d86612ce33aab0528283a.webp) *Figure 18.6: Jupyter kernel template customization screen* Once the template is completely customized, the kernel can be created. It automatically appears in the JupyterLab Launcher screen (figure [18.7](#fig-jupyter-template-created)) and can be used to run notebooks or a console session (figure [18.7](#fig-jupyter-template-created)): ![JupyterLab Launcher screen with new custom kernel](/base-command-manager/product-manuals/_fern-img/239b32aecaf366ec69c36b2b210bb445779ce1355c153e2e722bda733b27a561.webp) *Figure 18.7: JupyterLab Launcher screen with new custom kernel* A user who lists available Jupyter kernels via the command line now sees the newly-created kernel: ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ jupyter kernelspec list Available kernels: jupyter-eg-kernel-k8s-cmjkop-py-1gii7nm01 /home/alice/.local/share/jupyter/kernels/jupyter-eg-kernel\ -k8s-cmjkop-py-1gii7nm01 python3 /cm/shared/apps/jupyter/14.0.0/share/jupyter/kernels/python3 ``` The new kernel directory will contain the JSON definition generated by the Jupyter Kernel Creator: ```bash [jupyterhubuser@basecm10 ~]$ cat .local/share/jupyter/kernels/jupyter-eg-kernel-k8s-cmjkop-py-\ 1gii7nm01/kernel.json ``` ```bash { "language": "python", "display_name": "Python on Kubernetes Operator 221123133450", "metadata": { "process_proxy": { "class_name": "cm_jupyter_kernel_creator.eg_processproxies.k8scmjkop.KubernetesCMJupyterKernelOperator", "config": { "k8s_env_module": "kubernetes", "cmjk_template": "templates/cmjk.yaml.j2", "image": "jupyter/datascience-notebook", "image_pull_policy": "IfNotPresent", "namespace": "alice-restricted", "gpu_limit": 0 } } }, "argv": [ "python", "-m", "ipykernel_launcher", "-f", "/var/tmp/kernel-parm.json" ] } ``` Jupyter kernel names need not be unique. Users should therefore choose meaningful and distinguishable display names for their kernels. Doing so makes the JupyterLab Launcher screen easier to use. For convenience, a summary of the available kernel templates and their requirements is shown in table [18.1](#jupyter-kernel-templates): *Table 18.1: Available Jupyter kernel templates for BCM and their requirements* | **Template name** | **Requirement** | **Description** | | :-------------------------------------- | :--------------- | :------------------------------------------------------------------ | | `jupyter-eg-kernel-k8s-bash` | Kubernetes¹ | Bash via Kubernetes | | `jupyter-eg-kernel-k8s-py` | Kubernetes¹ | Python 3.9 via Kubernetes | | `jupyter-eg-kernel-k8s-py-spark` | Kubernetes² | Python 3.9 + Spark 3 via Kubernetes | | `jupyter-eg-kernel-k8s-r` | Kubernetes³ | R 3.4 via Kubernetes | | `jupyter-eg-kernel-k8s-cmjkop-ngc-py` | Kubernetes | NGC images via Jupyter Kernel Operator | | `jupyter-eg-kernel-k8s-cmjkop-julia` | Kubernetes | Jupyter official image via Jupyter Kernel Operator (using Julia) | | `jupyter-eg-kernel-k8s-cmjkop-py` | Kubernetes | Jupyter official image via Jupyter Kernel Operator (Python) | | `jupyter-eg-kernel-k8s-cmjkop-py-spark` | Kubernetes³ | Python + Spark via Jupyter Kernel Operator (using Python and Spark) | | `jupyter-eg-kernel-lsf-bash` | Platform LSF | Bash via Platform LSF | | `jupyter-eg-kernel-lsf-py39` | Platform LSF | Python 3.9 via Platform LSF | | `jupyter-eg-kernel-openpbs-bash` | Open PBS | Bash via Open PBS | | `jupyter-eg-kernel-openpbs-py39` | Open PBS | Python 3.9 via Open PBS | | `jupyter-eg-kernel-pbspro-bash` | PBS Professional | Bash via PBS Professional | | `jupyter-eg-kernel-pbspro-py39` | PBS Professional | Python 3.9 via PBS Professional | | `jupyter-eg-kernel-slurm-bash` | Slurm | Bash via Slurm | | `jupyter-eg-kernel-slurm-py-conda` | Slurm | Python and Conda 3.9 via Slurm | | `jupyter-eg-kernel-slurm-py39` | Slurm | Python 3.9 via Slurm | | `jupyter-eg-kernel-uge-bash` | GE | Bash via GE | | `jupyter-eg-kernel-uge-py39` | GE | Python 3.9 via GE | ¹ Docker image: *brightcomputing/jupyter-kernel-sample:k8s-py36-1.1.1* ² Docker image: *brightcomputing/jupyter-kernel-sample:k8s-spark-py37-1.2.1* ³ Docker image: *brightcomputing/jupyter-kernel-sample:k8s-r-1.0.0* DockerHub kernels page: [https://hub.docker.com/r/brightcomputing/jupyter-kernel-sample/tags](https://hub.docker.com/r/brightcomputing/jupyter-kernel-sample/tags) ### 18.5.2 Running Jupyter Kernels With Two Factor Authentication If PAM and CMDaemon are configured with two-factor authentication (2FA), then JupyterHub needs to be instructed to support it. This can be done using `cmsh` as follows: ```bash [root@basecm10 ~]# cmsh [basecm10]% configurationoverlay [basecm10->configurationoverlay]% use jupyterhub [basecm10->configurationoverlay[jupyterhub]]% roles [basecm10->configurationoverlay[jupyterhub]->roles]% use jupyterhub [basecm10->configurationoverlay[jupyterhub]->roles[jupyterhub]]% configs [basecm10->...]->roles[jupyterhub]->configs]% add c.BrightAuthenticator.twofa [basecm10->...]->roles*[jupyterhub*]->configs*[c.BrightAuthenticator.twofa*]]% set value True [basecm10->...]->roles*[jupyterhub*]->configs*[c.BrightAuthenticator.twofa*]]% commit ``` ### 18.5.3 Running Jupyter Kernels With Kubernetes The Jupyter Kernel Operator ([section 6.3 of the *Containerization Manual*](/containerization/kubernetesoperators#63-the-jupyter-kernel-operator)) is the recommended way to run kernels in Kubernetes in BCM. It allows users to run unmodified images; it takes care of communication with Jupyter Enterprise Gateway; and it manages the Jupyter kernel life-cycle, including cleaning up dead or orphaned kernels. After installation and configuration by `cm-kubernetes-setup`, Jupyter Kernel Operator kernels (`jupyter-eg-kernel-k8s-cmjkop-*`) appear in the JuputerLab interface in the template section. Kernels which do not support Jupyter Kernel Operator are hidden. The following BCM templates allow users to create and run Jupyter Enterprise Gateway kernels on compute nodes via Kubernetes: *Table 18.2: BCM templates for creating and running Jupyter Enterprise Gateway kernels on cluster nodes via Kubernetes* | **Template** | **Description** | | :-------------------------------------- | :---------------------------------------------------------- | | `jupyter-eg-kernel-k8s-cmjkop-ngc-py` | NGC images via Jupyter Kernel Operator | | `jupyter-eg-kernel-k8s-cmjkop-py` | Jupyter official image via Jupyter Kernel Operator (Python) | | `jupyter-eg-kernel-k8s-cmjkop-julia` | Jupyter official image via Jupyter Kernel Operator (Julia) | | `jupyter-eg-kernel-k8s-cmjkop-py-spark` | Python + Spark via Jupyter Kernel Operator | | `jupyter-eg-kernel-k8s-bash` | Custom kernel with Bash | | `jupyter-eg-kernel-k8s-py` | Custom kernel with Python | | `jupyter-eg-kernel-k8s-py-spark` | Custom kernel with Python and Spark 3 | | `jupyter-eg-kernel-k8s-r` | Custom kernel with R 3.4 | By default, templates that do not support Jupyter Kernel Operator appear only if Kubernetes Jupyter Kernel Operator is not installed. The administrator has to make sure that Kubernetes is correctly configured on the cluster, and that Pod Security Policy (PSP) or Kyverno is enabled ([section 4.10.2 of the *Containerization Manual*](/containerization/kubernetes#4102-podsecuritypolicy)). Details on Kubernetes installation are provided in [Chapter 4 of the *Containerization Manual*](/containerization/kubernetes). The default configuration proposed by `cm-kubernetes-setup` is usually sufficient to run Kubernetes kernels created from BCM’s templates. However, it is the responsibility of the administrator to add users registered in the Linux-PAM system to Kubernetes. For example, a test user `jupyterhubuser` can be added to Kubernetes with: **Example** ```bash [root@basecm10 ~]# cm-kubernetes-setup --add-user jupyterhubuser ``` For every new user added, `cm-kubernetes-setup` automatically generates a dedicated namespace with a name in the form \<*`user`*>`-restricted`. For instance, the command in the example above, creates the namespace: `jupyterhubuser-restricted`. Cluster administrators are strongly recommended to review pods security policies for these dedicated namespaces. In order to speed up kernel creation for the first user logging into JupyterLab when using BCM, it is recommended that all the relevant Kubernetes images are pre-loaded on the compute nodes that Jupyter Enterprise Gateway can contact. Jupyter notebooks using Kubernetes kernels cannot natively use machine learning libraries and frameworks provided by BCM, even if installed on the software image used by Kubernetes. The machine learning libraries and frameworks have to be included in the chosen Kubernetes image. ### 18.5.4 Running Jupyter Kernels Based On NGC Containers Jupyter NGC templates are available in the list of templates if: * Kubernetes is set up * NVIDIA GPUs are available on the Kubernetes cluster * Jupyter Kernel Operator ([section 6.3 of the *Containerization Manual*](/containerization/kubernetesoperators#63-the-jupyter-kernel-operator)) is installed ![Jupyter Kernel Operator selection with cm-kubernetes-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/d4c3ed0704f40d949a2ccf88f5dfdbe95e56e79f6592316117a42edfae71e1f6/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/jupyter-ngc-kernel-operator-urwid.png) *Jupyter Kernel Operator selection with cm-kubernetes-setup* It is also strongly advised to enable Kyverno or PSP (Pod Security Policy). ![Kyverno policy engine selection in cm-kubernetes-setup](/base-command-manager/product-manuals/_fern-img/892a92c76024b3aaf4e82b2f203f72fdcc06bd4047e8c464092d0bc828f5bb73.webp) *Kyverno policy engine selection in cm-kubernetes-setup* The user needs to be given permission to access to the Jupyter Kernel Operator. This can be done via the `cm-kubernetes-setup` TUI, by adding a user, and then setting the permissions in the permissions screen that shows up (figure [18.10](#User-permissions-in-cm-kubernetes-setup)): ![User permissions in cm-kubernetes-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/d7731eec0537911ab40c2ecbf7a3fba3fd6a9f3128d8432f564e6c7abf400d00/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/jupyter-ngc-user-permissions-urwid.png) *Figure 18.10: User permissions in cm-kubernetes-setup* The permissions can also be configured using the CLI: ```bash # cm-kubernetes-setup --add-user=alice --operators=cm-jupyter-kernel-operator ``` Once Jupyter kernel Operator is available, then kernel templates appear in the list: ![Jupyter Kernel Operator templates](/base-command-manager/product-manuals/_fern-img/381b203d154611f855e9093c88993156ffe9f36668091792580b1fedd2c9f4df.webp) *Jupyter Kernel Operator templates* A kernel can be created from the template: ![Creating kernel from template](/base-command-manager/product-manuals/_fern-img/4a21d247d0ab955c319fc988bd4addf6cc659c42796e500ee7790d34f3072a9f.webp) *Creating kernel from template* If authentication is required to access container images from the private registry, then the template can be modified as described in [section 6.3.8.2 of the *Containerization Manual*](/containerization/kubernetesoperators#6382-parameterizing-the-secret). ### 18.5.5 Running Jupyter Kernels With Workload Managers BCM’s Jupyter Enterprise Gateway kernels can be created and run by users on compute nodes via workload managers (WLMs). For convenience, Slurm is used as an example in this section. However, the same instructions are valid for the other WLMs listed in table [18.1](#jupyter-kernel-templates). The templates used to create Slurm kernels are BCM’s * `jupyter-eg-kernel-slurm-bash` and * `jupyter-eg-kernel-slurm-py39`, which offer a Bash and a Python 3.9 environment respectively. The administrator has to make sure that Slurm is correctly configured on the cluster. Slurm installation is described in section [7.3](/admin/workloadmanagement#73-installation-of-workload-managers). The default configuration that `cm-wlm-setup` suggests in `Express` mode is usually sufficient to run Slurm kernels created from BCM’s templates. If Slurm is configured to automatically detect GPUs, they will be listed as available resources while instantiating Slurm templates. The Jupyter login node must be authorized to submit Slurm jobs. This is typically the case, since the JupyterHub login node is by default the head node, and `cm-wlm-setup` by default assigns the `slurmsubmit` role to the head node. Finally, the administrator must make sure that relevant dependency packages are installed on the software image used by Slurm clients. The image used is typically for the compute nodes. Missing packages may cause kernels to fail at startup or at run time. In particular, administrators need to install `cm-python39` to use kernels based on the `jupyter-eg-kernel-slurm-py39` template. Jupyter notebooks using Slurm kernels can natively use the machine learning libraries and frameworks provided by BCM on Slurm clients, if these are available. Further details on machine learning packages installation on both the head node and the compute nodes are given in section 1.4 of the *Machine Learning Manual*. ## 18.6 Jupyter Kernel Creator Extension Customization The Jupyter kernel templates described in section [18.5](#185-jupyter-kernel-creator-extension) are stored under the directory `/cm/shared/apps/jupyter/current/lib/python3.9/site-packages/cm_jupyter_kernel_creator/kerneltemplates/`. The template of a particular Jupyter kernel in BCM’s Jupyter Kernel Creator extension is a directory containing at least two files: `meta.yaml` and `kernel.json.j2`. The kernel template can also require other files placed in the same directory, such as icons. These files are copied to the target user kernels directory upon kernel creation, after the template is instantiated. The `meta.yaml` file includes all the parameters that can be substituted into `kernel.json.j2` and is defined with the YAML format ([https://yaml.org/](https://yaml.org/)). The `kernel.json.j2` file is a skeleton of the `kernel.json` file to be generated. It can contain some placeholders, and is defined with the Jinja2 format ([https://jinja.palletsprojects.com/](https://jinja.palletsprojects.com/)). ### 18.6.1 Kernel Template Parameters Definition The kernel template `meta.yaml` file defines all the parameters that can be used in `kernel.json.j2`. It should contain three entries: * `display_name`: the name that will be displayed for the kernel in the JupyterLab Launcher; * `features`: a list of features that must be available on the cluster to show this kernel template to JupyterLab users in the BCM extensions panel; * `parameters`: the variables for `kernel.json.j2`. The `display_name` entry is an arbitrary string: ```bash [...] display_name: "A simple kernel" [...] ``` The `features` entry is a list of strings. The only possible values for these strings are: * `"kubernetes"` * `"lsf"` * `"pbspro"` * `"slurm"` * `"uge"` For example, a kernel template that only requires Kubernetes to work contains this line in its `meta.yaml`: ```bash [...] features: ["kubernetes"] [...] ``` The same principle applies to the other workload managers. If multiple features are listed, then the kernel template is displayed if at least one of them is available. Finally, the `parameters` entry contains a dictionary of kernel parameters. The keys of the dictionary are parameter names that are used in `kernel.json.j2`. The values of the dictionary are options that help users choose a correct value for the parameter. Parameter options are also dictionaries. A kernel template `meta.yaml` is thus structured as: ```bash [...] parameters: : : ... : : : ... : ... : : ... : [...] ``` Parameter names are arbitrary strings. Option keys are strings. The only possible values for these strings are: * `type` * `definition` * `limits` The option keys `type` and `definition` are mandatory. The option key `limits` is optional. For example, a kernel template with two parameters `foo` and `bar` looks as follows: **Example** ```bash [...] parameters: foo: type: definition: limits: bar: type: definition: limits: [...] ``` #### 18.6.1.1 The `type` Option The `type` option key defines the type of the kernel parameter. The `type` option key only accepts one of the following string values: * `num`: for numeric values (both float and integer are supported); * `str`: for arbitrary strings; * `bool`: for boolean values (a checkbox is presented in the user interface); * `list`: for lists of pre-defined or dynamically-generated settings; * `uri`: for interactive RESTful endpoints (only `/kernelcreator/envmodules` is currently supported). #### 18.6.1.2 The `definition` Option The `definition` option key defines how the parameter value is retrieved and displayed. The `definition` option key accepts dictionary-like values. Allowed string keys for the dictionary are: * `display_name`: the name that is displayed for the parameter in the kernel customization dialog. It accepts an arbitrary string as a value; * `getter`: how parameter values are retrieved. It only accepts as value one of the following strings: * `static`: the values for the parameter are pre-defined; * `shell`: the values for the parameter are the output of the shell script; * `python`: the values for the parameter are the output of a Python script; * `values`: possible values for the parameter that are displayed in the kernel customization dialog. It accepts a list of arbitrary values; * `default`: default value for the parameter. It accepts a value from `values`; * `exec`: the script to be executed to fill `values` when `getter` is `shell` or `python`. It accepts a shell/Python script. If `type` option value is `list`, then every line of `getter` with a setting of `shell` or `python` is treated as an element of the list. #### 18.6.1.3 The `limits` Option The `limits` option key offers a way to apply bounds on values provided by users. This usually reduces the chances of making mistakes and helps users defining correct kernels before actually running them. The `limits` option key accepts dictionary-like values according to the value chosen for the `type` option key. If `num` is the `type`, then `limits` can contain: * `min`: the minimum numeric value; * `max`: the maximum numeric value. If `str` or `list` is the `type`, then `limits` can contain: * `min_len`: the minimum length of the string or the list; * `max_len`: the maximum length of the string or the list. For example, if every node in the cluster has no more than 4 GPUs, then the upper limit on the requested GPU number can be set to 4: ```bash [...] parameters: gpus: type: num definition: limits: min: 1 max: 4 [...] ``` These limits are not a security measure. They should be considered as convenient sanity checks for values entered while instantiating a kernel template. This is because users are always able to later directly edit the generated `kernel.json`, thereby ignoring such limits. ### 18.6.2 Kernel Template Parameters Usage In the `kernel.json.j2` file of a kernel template, two types of variables can be defined: Jinja2 variables ([https://jinja.palletsprojects.com/en/master/templates/#variables](https://jinja.palletsprojects.com/en/master/templates/#variables)) and Python formatted string literals (also known as f-strings) ([https://docs.python.org/3/reference/lexical_analysis.html#f-strings](https://docs.python.org/3/reference/lexical_analysis.html#f-strings)). They should not be mixed as they work on different steps of the template lifecycle: * `{{..}}` and `{%..%}`: These formats are used to specify Jinja2 expressions and statements. Jinja2 expressions and statements are substituted or computed once, when the user creates a kernel with BCM’s Jupyter Kernel Creator extension. The actual values will be those specified in the kernel customization dialog; * `{..}`: This format is used to specify Python f-strings. F-strings are substituted every time a user starts the kernel for a Jupyter notebook. The actual values will be a path to the kernel connection file, a job ID in a WLM, a UID of the running kernel process, a unique name for a Kubernetes pod, etc. Some process-proxies in Jupyter Enterprise Gateway may require another type of variables template. For example, the Kubernetes process-proxy requires multiple objects to be set up in its API. Several unique configuration files need to be created at runtime. Templates for such objects are located in the `templates/` directory, along with the `meta.yaml` and `kernel.json.j2` files. These templates are rendered every time a Kubernetes kernel is started for a Jupyter notebook. Variables for these templates are located inside the `config` section of the `kernel.json` file. ### 18.6.3 Kernel Template Creation Example To illustrate the kernel variables syntax described in sections [18.6.1](#1861-kernel-template-parameters-definition) and [18.6.2](#1862-kernel-template-parameters-usage), a kernel template is now created from Jupyter’s default Python 3 kernel. The new kernel template has to be placed in a suitable directory for BCM’s Jupyter Kernel Creator extension. This directory is the `kerneltemplates/` directory in the extension’s Python module: ```bash [root@basecm10 ~]# ls /cm/shared/apps/jupyter/current/lib/python3.9/site-packages/\ cm_jupyter_kernel_creator/kerneltemplates/ jupyter-eg-kernel-k8s-bash jupyter-eg-kernel-lsf-bash jupyter-eg-kernel-slurm-bash jupyter-eg-kernel-k8s-py jupyter-eg-kernel-lsf-py39 jupyter-eg-kernel-slurm-py39 jupyter-eg-kernel-k8s-py-spark jupyter-eg-kernel-pbspro-bash jupyter-eg-kernel-uge-bash jupyter-eg-kernel-k8s-r jupyter-eg-kernel-pbspro-py39 jupyter-eg-kernel-uge-py39 ``` The content of the original Jupyter kernel directory, including its images, is copied. The `kernel.json` file is then renamed to `kernel.json.j2` to make it a Jinja2 template: ```bash [root@basecm10 ~]# cd /cm/shared/apps/jupyter/current/lib/python3.9/site-packages/\ cm_jupyter_kernel_creator/kerneltemplates/ [root@basecm10 kerneltemplates]# cp -pr /cm/shared/apps/jupyter/current/share/jupyter/kernels/\ python3 python3-test [root@basecm10 kerneltemplates]# mv python3-test/kernel.json{,.j2} ``` The next step is to create the `meta.yaml` file with the kernel template parameters. In this example a request is made to have Slurm deployed on the cluster: ```bash [root@basecm10 kerneltemplates]# cat << EOF >python3-test/meta.yaml --- display_name: "Python 3 demo kernel" features: ["slurm"] EOF ``` The new kernel template is now available in JupyterLab (figure [18.13](#fig-jupytyter-new-template-menu)): ![New kernel template in the JupyterLab interface](/base-command-manager/product-manuals/_fern-img/46a01de844cf5cf0456675b2f200ca956774a3540c1eb6679c1a1b19c709e8fe.webp) *Figure 18.13: New kernel template in the JupyterLab interface* The next step is to define kernel template parameters. The first one being added is `kernel_name`, which is the name of the kernel in the JupyterLab Launcher. The `meta.yaml` file is updated: ```bash --- display_name: "Python 3 demo kernel" features: ["slurm"] parameters: kernel_name: type: str definition: getter: static default: "Demo kernel" display_name: "Display name of the kernel" ``` The `display_name` definition in the `kernel.json.j2` is now changed to a Jinja2 placeholder matching this `meta.yaml` parameter with the appropriate Jinja2 syntax: ```bash { "display_name": "{{ kernel_name }}", [...] } ``` The JupyterLab interface changes and now prompts the user to enter a value, while providing a default option (figure [18.14](#fig-jupytyter-new-template-user-ui)): ![New kernel template dialog during template instantiation](/base-command-manager/product-manuals/_fern-img/81cb8aba258d5ea4528a3961332dc07d4590f74c501a2fc12f254918a46e1fd1.webp) *Figure 18.14: New kernel template dialog during template instantiation* If the user clicks on the `Create` button, then a new directory is created in `~/.local/share/jupyter/kernels`. The content of the kernel template directory, except for the `meta.yaml` and `kernel.json.j2` files, is copied into this new directory. The kernel name provided by the user is substituted into the `kernel.json.j2` template, and the resulting `kernel.json` file is written into the new user kernel directory. As a result, the new kernel then appears in the JupyterLab Launcher. ### 18.6.4 Extending Kubernetes Kernel Templates BCM’s JEG kernel templates for Kubernetes can be extended with additional parameters. In the following example, a Kubernetes kernel template is extended, to show a dropdown menu with additional environment variables in the kernel customization dialog. The selected variables are exported to the context of the running kernel and made available in the Jupyter notebook. In order to preserve the original kernel template, the existing `jupyter-eg-kernel-k8s-py` template is copied to a user directory. The new customized kernel template directory could be later copied back to the original BCM Jupyter Kernel Creator extension templates directory, to make it available for all the users. ```bash [jupyterhubuser@basecm10 ~]$ mkdir -p ~/.local/share/jupyter/kerneltemplates [jupyterhubuser@basecm10 ~]$ cd ~/.local/share/jupyter/kerneltemplates [jupyterhubuser@basecm10 kerneltemplates]$ cp -pr /cm/shared/apps/jupyter/current/lib/python3.9/\ site-packages/cm_jupyter_kernel_creator/kerneltemplates/jupyter-eg-kernel-k8s-py\ ./jupyter-eg-kernel-k8s-py-demo ``` To distinguish the new kernel template from the original one, it is convenient to change the display name: ```bash --- display_name: "Python on Kubernetes Demo" features: ["kubernetes"] [...] ``` The next step is to add a new parameter in the `meta.yaml` file: ```bash [...] parameters: [...] additional_vars: type: list definition: getter: shell default: [] display_name: "Additional variables to export in the pod" exec: - echo -e "VAR1=value1\nCUR_DATE=$(date +'%Y-%m-%d')" [...] ``` Now, if the kernel template is instantiated, then a new parameter is displayed in the kernel customization dialog (figure [18.15](#fig-jupyter-kubernetes-template-vars-dropdown)): ![New environment variables dropdown menu](/base-command-manager/product-manuals/_fern-img/d9ef060a71b9e96d56d98dfbda208bbf08f9473bf0625a08bbfea2a772f0b2a3.webp) *Figure 18.15: New environment variables dropdown menu* The kernel customization dialog also shows that every output line of the script was rendered as an item of the menu. To pass the selected environment variables to the actual container in the running pod, the `kernel.json.j2` and `templates/pod.yaml.j2` files must both be modified. This is due to the two-step template rendering mechanism: 1. the first rendering takes place when the user instantiates the kernel template. The `kernel.json.j2` file becomes `kernel.json`, and kernel parameters are replaced with the values specified in the dialog; 2. the second rendering takes place when the user starts the kernel for a Jupyter notebook. The `pod.yaml.j2` file and other Jinja2 files in the `templates/` directory are rendered to the actual YAML representations of the Kubernetes objects. In the example that follows, the `kernel.json.j2` file is extended with the "`additional_vars`" kernel parameter: ```bash { [...] "metadata": { "process_proxy": { [...] "config": { [...] "additional_vars": {{ additional_vars | tojson }}, [...] } } }, [...] } ``` The last step requires the use of the Jinja2 loop syntax in the `templates/pod.yaml.j2` file: ```bash [...] spec: [...] containers: - name: notebook [...] env: [...] {% for env_var in additional_vars %} {% set var, value = env_var.split("=") %} - name: {{ var }} value: "{{ value }}" {% endfor %} [...] ``` Now, if a kernel is instantiated with both the environment variables (figure [18.16](#fig-jupyter-kubernetes-template-vars-dropdown-chosen)) and then started from a notebook, then the variables are available in the process (figure [18.17](#fig-jupyter-kubernetes-template-kernel-w-vars)): ![Environment variables are chosen in the kernel customization dialog](/base-command-manager/product-manuals/_fern-img/442dc5ecbece6d75b33f1c9ebf09f3ab9abc4075bac24ab9d8c7d352bf0f5b60.webp) *Figure 18.16: Environment variables are chosen in the kernel customization dialog* ![Environment variables are available in the running kernel](/base-command-manager/product-manuals/_fern-img/ca0a3ce3fe3dae866ec4f92827f474e2a76c49d35349400b438332681aa52251.webp) *Figure 18.17: Environment variables are available in the running kernel* ### 18.6.5 Extending Workload Managers Kernel Templates BCM’s JEG kernel templates for workload managers can be extended with additional parameters. For convenience, Slurm is used as example in this section. However, the same instructions are valid for the other WLMs listed in table [18.1](#jupyter-kernel-templates). In the following example, a Slurm kernel template is extended, to show a dropdown menu with additional environment variables in the kernel customization dialog. The selected variables are exported to the context of the running kernel and made available in the Jupyter notebook. In order to preserve the original kernel template, the existing `jupyter-eg-kernel-slurm-py39` template is copied to a user directory. The new customized kernel template directory could be later copied back to the original BCM Jupyter Kernel Creator extension templates directory, to make it available for all users. ```bash [jupyterhubuser@basecm10 ~]# mkdir -p ~/.local/share/jupyter/kerneltemplates [jupyterhubuser@basecm10 ~]# cd ~/.local/share/jupyter/kerneltemplates [jupyterhubuser@basecm10 kerneltemplates]# cp -pr /cm/shared/apps/jupyter/current/lib/python3.9/\ site-packages/cm_jupyter_kernel_creator/kerneltemplates/jupyter-eg-kernel-slurm-py39\ ./jupyter-eg-kernel-slurm-py39-demo ``` To distinguish the new kernel template from the original one, it is convenient to change the display name: ```bash --- display_name: "Python 3.9 via SLURM Demo" features: ["slurm"] [...] ``` The next step is to add a new parameter in the `meta.yaml` file: ```bash [...] parameters: [...] additional_vars: type: list definition: getter: shell default: [] display_name: "Additional variables to export in the job" exec: - echo -e "VAR1=value1\nCUR_DATE=$(date +'%Y-%m-%d')" [...] ``` Now, if the kernel template is instantiated, then a new parameter is displayed in the kernel customization dialog (figure [18.18](#fig-jupyter-slurm-template-vars-dropdown)): ![New environment variables dropdown menu](/base-command-manager/product-manuals/_fern-img/d4146e2a537a00a971b87299d2672c1605329ec2e858ed7293dede75c11cf2c8.webp) *Figure 18.18: New environment variables dropdown menu* The kernel customization dialog also shows that every output line of the script was rendered as an item of the menu. To pass the selected environment variables to the actual workload manager job, the `"submit_script"` section inside the `kernel.json.j2` file must be modified to take into account the new "`additional_vars`" kernel parameter: ```bash { [...] "metadata": { "process_proxy": { [...] "config": { [...] "submit_script": [ [...] "set -xe", "{environ}", {% for env in additional_vars %} "export {{ env }}", {% endfor %} "{kernel_cmd}" ] } } }, [...] } ``` Now, if a kernel is instantiated with both the environment variables (figure [18.19](#fig-jupyter-slurm-template-vars-dropdown-chosen)) and then started from a notebook, then the variables are available in the process (figure [18.20](#fig-jupyter-slurm-template-kernel-w-vars)): ![Environment variables are chosen in the kernel customization dialog](/base-command-manager/product-manuals/_fern-img/c5347cb835a8a4c1ff08e18b6d346a9e5b2413c3d3bbd4876ce1434e257e330b.webp) *Figure 18.19: Environment variables are chosen in the kernel customization dialog* ![Environment variables are available in the running kernel](/base-command-manager/product-manuals/_fern-img/ca0a3ce3fe3dae866ec4f92827f474e2a76c49d35349400b438332681aa52251.webp) *Figure 18.20: Environment variables are available in the running kernel* ### 18.6.6 Filtering Out Irrelevant Templates From The Interface For Users The list of kernel templates that are available for users of the Jupyter Kernel Creator can be modified as follows: In the parent directory of the templates, the file `filter.yaml` describes items called `features`. These can be statically defined or they can represent scripts to be executed. The result of an executed script can be `true` or `false`. If the script code exits with zero, then the result is `true`. Every kernel template can have the `feature` field, which is a boolean expression that is calculated when the JupyterLab extension is started. **Example** ```bash # cat filter.yaml ... slurm: getter: shell timeout: 5 exec: - "source /etc/profile.d/modules.sh" - "module load slurm" - "sinfo" ... # cat jupyter-eg-kernel-slurm-py39/meta.yaml ... features: "slurm" ... ``` In the preceding example, above `jupyter-eg-kernel-slurm-py39` template is shown in the JupyterLab interface if the script in the `exec:` section is able to finish successfully. It is also possible to define more complicated rules: **Example** ```bash # cat filter.yaml features: kubernetes: getter: shell timeout: 5 exec: - "source /etc/profile.d/modules.sh" - "module load kubernetes" - "kubectl get pods" k8s-jupyter-operator-installed: getter: shell timeout: 5 exec: - "source /etc/profile.d/modules.sh" - "module load kubernetes" - "(kubectl get cmjupyterkernels 2>&1 || true) \\" - "| egrep -q '^Error from server \\(Forbidden\\)'" k8s-jupyter-operator-enabled: getter: shell timeout: 5 exec: - "source /etc/profile.d/modules.sh" - "module load kubernetes" - "kubectl get cmjupyterkernels" ... # cat jupyter-eg-kernel-k8s-py/meta.yaml --- display_name: "Python on Kubernetes" features: "kubernetes and not k8s-jupyter-operator-installed and not k8s-jupyter-operator-enabled" ... ``` In the preceding example, `jupyter-eg-kernel-k8s-py` is shown when Kubernetes is installed, and is hidden if Jupyter Kubernetes Operator is available for the user. The `filter.yaml` file also supports statically defined features: **Example** ```bash features: always-enabled: getter: static default: True always-disabled: getter: static default: False ... ``` and supports Python code: **Example** ```bash python3-available: getter: python timeout: 5 exec: - "import sys" - "sys.exit(0) if sys.version_info.major == 3 else sys.exit(1)" ... ``` The Kubernetes Jupyter Kernel Operator is discussed in [section 6.3 of the *Containerization Manual*](/containerization/kubernetesoperators#63-the-jupyter-kernel-operator). ## 18.7 Jupyter VNC Extension ### 18.7.1 What Is Jupyter VNC Extension About? VNC (Virtual Network Computing) is a screen sharing service that can work in a browser. If VNC is allowed by the cluster administrator, then the Jupyter environment configured by BCM can be used to start and control remote desktops via VNC with the *Jupyter VNC* extension. Several kernels created from BCM’s templates are capable of running VNC sessions so that users can run GUI applications. In order to do so, the cluster nodes where kernels are executed must support VNC. During the `cm-jupyter-setup` run, the TUI prompts for installation of VNC servers on the nodes. The cluster administrator can select the nodes on which it installs. On these nodes Jupyter kernels can work with several types of VNC clients. ### 18.7.2 Enabling User Lingering User lingering is a systemd setting that sets a user manager for a user at boot and keeps it around after logout. This allows that user to run long-running sessions despite not being logged in. Enabling user lingering may be required for Jupyter VNC extension to run for a relatively complicated desktop such as KDE or GNOME. For each user on each machine where these environments are installed, the following command must be run: ```bash loginctl enable-linger ``` The command may also be carried out using prolog/epilog scripts in the chosen WLM. ### 18.7.3 Starting A VNC Session With The Jupyter VNC Extension Users can start a VNC session with the button added by Jupyter VNC (figure [18.21](#fig-jupyter-vnc-starting)). Additional VNC parameters can be optionally specified. ![Starting Jupyter VNC session from kernel](/base-command-manager/product-manuals/_fern-img/7060da3e2998d861cdbf8131333daab962ccc0f54c7c3af696523b16a324aae0.webp) *Figure 18.21: Starting Jupyter VNC session from kernel* If VNC is available and correctly configured on the node where the kernel is running, then a new tab is automatically created by Jupyter VNC containing the new session (figure [18.22](#fig-jupyter-vnc-started)). A user can now freely interact in JupyterLab both with the notebook and with the desktop environment. ![Running Jupyter VNC session from kernel](/base-command-manager/product-manuals/_fern-img/bdd5acb8ebc7bff1e97d36d9f00e7a0e13c7873def75b335da8ec5c2c09ea8bf.webp) *Figure 18.22: Running Jupyter VNC session from kernel* To provide a user-friendly experience, Jupyter VNC also allows the graphical viewport to be resized, so that the desktop application can run full-screen (figure [18.23](#fig-jupyter-vnc-fullscreen)). ![Running Jupyter VNC session from kernel (full-screen)](/base-command-manager/product-manuals/_fern-img/0db7c42b4b7e04d33446169e91448309f7de709ef61997f8b6924761b6d8cae3.webp) *Figure 18.23: Running Jupyter VNC session from kernel (full-screen)* ### 18.7.4 Running Examples And Applications In The VNC Session With The Jupyter VNC Extension Once the VNC session is correctly started and the new JupyterLab tab has been created, Jupyter VNC automatically exports the `DISPLAY` environment variable to the running notebook (figure [18.24](#fig-jupyter-vnc-display)). Doing so means that any application or library running in the notebook can make use of the freshly created desktop environment. An example of such a library is OpenAI Gym, a toolkit for developing and comparing reinforcement learning algorithms, that is distributed by BCM. Among the examples distributed by BCM (section [18.3](#183-jupyter-notebook-examples)), a notebook running PyTorch in the OpenAI Gym CartPole environment can be found. If executed after a VNC session has been started, a user can then observe the model being trained in real time in the graphical environment. ![Automatic configuration of DISPLAY environment variable](/base-command-manager/product-manuals/_fern-img/bb5e33735424c7738286805ae2448233bdeba4fc5625197ca9d136bd8d1f7dbc.webp) *Figure 18.24: Automatic configuration of DISPLAY environment variable* ## 18.8 Jupyter WLM Magic Extension In the Jupyter environment configured by BCM, the *Jupyter WLM Magic* extension can be used to schedule workload manager jobs from notebooks. The Jupyter WLM Magic extension is an IPython extension. It is designed to improve the capabilities of Jupyter’s default Python 3 kernel, which runs on the login node. The Jupyter WLM Magic extension should therefore not be used from kernels running on compute nodes, such as those typically created with BCM’s Jupyter Kernel Creator extension (section [18.5](#185-jupyter-kernel-creator-extension)), and submitted via Jupyter Enterprise Gateway. Indeed, compute nodes running these kernels are often incapable of starting workload manager jobs in many default WLM configurations. Jupyter WLM Magic extension makes it possible for users to programmatically submit WLM jobs, and then interact with their results. This can be done while using the Python programming language and its libraries, which are available in the notebook. Users submit jobs and check their progress from the login node. The actual computation is distributed by the underlying workload manager across compute nodes, which means that server resources are spared. Jupyter WLM Magic commands are available in the IPython kernel as *magic functions* ([https://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magic-functions](https://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magic-functions)). A new line magic (`%`) and a new cell magic (`%%`) are now added in the kernel, according to the workload manager: * Platform LSF: `%lsf_job` and `%%lsf_job` * PBS Professional: `%pbspro_job` and `%%pbspro_job` * Slurm: `%slurm_job` and `%%slurm_job` * GE: `%uge_job` and `%%uge_job` A user can list the magic functions in the kernel to see if they are available, with Jupyter’s builtin command `%lsmagic` ([https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-lsmagic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-lsmagic)): **Example** ```bash In []: %lsmagic Out []: root: line: automagic:"AutoMagics" autocall:"AutoMagics" [...] slurm_job:"SLURMMagic" pbspro_job:"PBSProMagic" lsf_job:"LSFMagic" uge_job:"UGEMagic" cell: js:"DisplayMagics" javascript:"DisplayMagics" [...] slurm_job:"SLURMMagic" pbspro_job:"PBSProMagic" lsf_job:"LSFMagic" uge_job:"UGEMagic" ``` The magic functions introduced by this BCM extension share a similar syntax. For convenience, Slurm is used as an example in this section. However, the same instructions are valid for the other WLMs. Users can check which options are available for a WLM function with the line magic helper: **Example** ```bash In []: %slurm_job --help Out []: usage: %slurm_job [-h] [--module MODULE] [--module-load-cmd MODULE_LOAD_CMD] [--shell SHELL] [--submit-command SUBMIT_COMMAND] [--cancel-command CANCEL_COMMAND] [--control-command CONTROL_COMMAND] [--stdout-file STDOUT_FILE] [--stderr-file STDERR_FILE] [--preamble PREAMBLE] [--timeout TIMEOUT] [--check-condition-var CHECK_CONDITION_VAR] [--job-id-var JOB_ID_VAR] [--stdout-file-var STDOUT_FILE_VAR] [--stderr-file-var STDERR_FILE_VAR] [--dont-wait] [--write-updates WRITE_UPDATES] [--check-status-every CHECK_STATUS_EVERY] optional arguments: -h, --help show this help message and exit [...] ``` Line magic functions are typically used to set options with a global scope in the notebook. By doing so, a user will not need to specify the same option every time a job will be submitted via cell magic. For example, if two Slurm instances are deployed on the cluster and their associated environment modules are `slurm-primary` and `slurm-secondary`, a user could run the following line magic once to configure the Jupyter WLM Magic extension to always use the second deployment: **Example** ```bash In []: %slurm_job --module slurm-secondary Out []: ``` Now, jobs will always be submitted to `slurm-secondary`. This is more convenient than repeatedly defining the same module option for every cell magic upon scheduling a job: **Example** ```bash In []: %%slurm_job --module slurm-secondary Out []: In []: %%slurm_job --module slurm-secondary Out []: ``` It should be noted that line magic functions cannot be used to submit WLM jobs. Cell magic functions have to be used instead. A well-defined cell contains the WLM cell magic function provided by the extension, followed by the traditional job definition. For example, a simple MPI job running on two nodes can be submitted to Slurm by defining and running this cell: **Example** ```bash In []: %%slurm_job #SBATCH -J mpi-job-example #SBATCH -N 2 module load openmpi mpirun hostname Out []: COMPLETED STDOUT file content: /home/demo/.jupyter/wlm_magic/slurm-1.out node001 node001 node002 node002 ``` Users can take advantage of the Jupyter WLM Magic extension to store some information into Python variables about the job being submitted. The information could be the ID or the output file name, for example. Users can then later programmatically interact with them in Python. This feature is convenient when a user wants to, for example, programmatically carry out new actions depending on the job output: **Example** ```bash In []: %%slurm_job --job-id-var my_job_id --stdout-file-var my_job_out #SBATCH -J mpi-job-example #SBATCH -N 2 module load openmpi mpirun hostname Out []: COMPLETED STDOUT file content: /home/demo/.jupyter/wlm_magic/slurm-2.out node001 node001 node002 node002 In []: print(f"Job id {my_job_id} was written to {my_job_out}") print(f"Output lines: {open(my_job_out).readlines()}") Out []: Job id 2 was written to /home/demo/.jupyter/wlm_magic/slurm-2.out Output lines: ['node001\n', 'node001\n', 'node002\n', 'node002\n'] ``` Users can also exploit Python variables to define the behavior of the Jupyter WLM Magic extension. For example, they can define a Python boolean variable to submit a WLM job only if a condition is true: **Example** ```bash In []: run_job = 1 == 2 Out []: In []: %%slurm_job --check-condition-var run_job #SBATCH -J mpi-job-example #SBATCH -N 2 module load openmpi mpirun hostname Out []: Variable run_job is 'False'. Skipping submit. ``` ## 18.9 Jupyter Kubernetes Operators Manager The Jupyter Kubernetes Operators Manager tool is designed to make it easier to handle everyday tasks involving Kubernetes. Some of the items that the tool can manage are: * Pods * PostgreSQL databases * Spark tasks (jobs that process a lot of data) * Persistent Volume Claims (requests for specific storage space). Particularly handy is the ability to move data between user folders and Persistent Volumes (special types of storage space in Kubernetes). Jupyter Kubernetes Operators Manager can be accessed as follows in the Jupyter Notebooks web interface (figure [18.25](#fig-Jupyter-Kubernetes-Operators-Manager-List-And-Selection)): ![Jupyter Kubernetes Operators Manager: Kubernetes cluster list and selection](/base-command-manager/product-manuals/_fern-img/37a57a181b42f7757cc72007c06887c119d322f6c212264a4555e798204d12d8.webp) *Figure 18.25: Jupyter Kubernetes Operators Manager: Kubernetes cluster list and selection* 1. The NVIDIA Base Command Manager logo (on the left side of the screen, ![image](/base-command-manager/product-manuals/_fern-img/f3248fd6b6c5a2a63d0606a711abc4501542e525832e8736add6da7f5ff8277e.webp)) is clicked. A list of Kubernetes clusters is then displayed. 2. The `Cluster View` button is used to select the cluster that is to be worked with. Jupyter Kubernetes Operators Manager displays resources and objects via a tabbed view. This view is for the restricted namespace to which the user has access. With Kubernetes as set up by BCM, this is the namespace of the form \<*`user`*>`-restricted`. ### 18.9.1 Overview Tab The `Overview` tab (figure [18.26](#fig-Jupyter-Kubernetes-Operators-Manager--overview)) provides a high-level overview of the current state of the selected Kubernetes cluster. ![Jupyter Kubernetes Operators Manager: overview](/base-command-manager/product-manuals/_fern-img/1c229e244f83371135735d784d1b29ea3ad347c6f805916d1399d36bb80a8ad3.webp) *Figure 18.26: Jupyter Kubernetes Operators Manager: overview* Among other useful details, it displays: * the cluster name * the Kubernetes version that is running * the number of namespaces and pods ### 18.9.2 Jupyter Kernel Overview Tab The `Jupyter Kernel Overview` tab (figure [18.27](#fig-Jupyter-Kubernetes-Operators-Manager--Jupyter-kernel-overview)) lists active Jupyter kernel instances along with their associated events. ![Jupyter Kubernetes Operators Manager: Jupyter kernel overview tab](/base-command-manager/product-manuals/_fern-img/16e745abfbfb4145472e4f5c595d99615fce3af1548ce89b369e9a27a9e12ae4.webp) *Figure 18.27: Jupyter Kubernetes Operators Manager: Jupyter kernel overview tab* After a kernel is stopped or removed, these events can be found under the `Events` tab. Events are discussed later on. ### 18.9.3 Jobs Tab The `Jobs` tab (figure [18.28](#fig-Jupyter-Kubernetes-Operators-Manager--Jupyter-data-migration-jobs-tab)) allows a data migration job to be run. A data migration job manages the transfer of data between the user-accessible directories of the filesystem and the Persistent Volumes. The transfer can be managed in either direction. The data transfer is needed to enable access to the data for Spark instances, and for situations where pods need to be run under a different user ID/group ID (UID/GID) from the original user. ![Jupyter Kubernetes Operators Manager: Jupyter data migration jobs tab](/base-command-manager/product-manuals/_fern-img/020d813eb2955bd023af5465c95fd4505aa8aa5f2c84c74a8bb57e9243e8235d.webp) *Figure 18.28: Jupyter Kubernetes Operators Manager: Jupyter data migration jobs tab* To initiate a data migration job, several fields must be set. This can be done via a pop-up dialog, that comes up on clicking either of these buttons: * `New job from Config`: allows a YAML file configuration to be submitted * `New Job`: allows direct configuration (figure [18.29](#fig-Jupyter-Kubernetes-Operators-Manager--data-migration-jobs-creation)) ![Jupyter Kubernetes Operators Manager: data migration jobs creation](/base-command-manager/product-manuals/_fern-img/a06b64870a10b35ff474a9e70064cd91e403f81faa818d37fd0e17b59d873980.webp) *Figure 18.29: Jupyter Kubernetes Operators Manager: data migration jobs creation* Fields that may be set include: * **Job name**: specifies the name of the migration job. * **Direction**: sets the direction of the data migration. * `to-k8s`: copies data to the Persistent Volume Claim (PVC) from the file system * `from-k8s`: copies data from the PVC back to the file system * **Path**: provides the path on the filesystem to/from where the data is migrated * **PVC**: sets the name of the Persistent Volume Claim that is involved in the data migration. Optionally, the `Force Delete` checkbox can be ticked. If ticked, then the migration job deletes data from the target location if the corresponding file or directory does not exist in the source location. In particular, if the source location is completely empty, then setting `Force Delete` results in the removal of all target data. Setting `Force Delete` should therefore be done with caution due to the significant data loss that it can cause. The execution status of a job can be monitored in the status and events area. This section displays the number of worker pods and their respective statuses, which include: * **Running**: the number of pods currently in operation. * **Completion**: the desired number of pods that should successfully complete the job. Not applicable for migration jobs, but it can be used if a custom manifest is specified, as described later. * **Succeeded**: The number of pods that have successfully completed their tasks. * **Failed**: The number of pods that have failed to complete their tasks. Additionally, a custom job manifest can be uploaded from the user’s workstation if required. ### 18.9.4 Pods Tab The `Pods` tab (figure [18.30](#fig-Jupyter-Kubernetes-Operators-Manager--pods-overview-tab)) displays user pods, reads stdout, and shows associated events from pods. ![Jupyter Kubernetes Operators Manager: pods overview tab](/base-command-manager/product-manuals/_fern-img/92d007e9a44ac278536fd4f95b6119da02db54c93bb74652229236b07af4efbd.webp) *Figure 18.30: Jupyter Kubernetes Operators Manager: pods overview tab* The tab also allows pod creation from a custom definition manifest uploaded from the user’s workstation, or by filling out a simplified form with commonly used fields: 1. **Name**: name of the Pod. 2. **Command**: program and its arguments that are to run inside the Pod. 3. **Image**: container image. 4. **Host Path** (Optional): path to mount from the filesystem. If used, the UID/GID of the running process matches the UID/GID of the user. 5. **Environment Variables** (Optional): These variables are set for the process inside the running Pod. 6. **PVCs** (Optional): A list of Persistent Volume Claims and the path to mount them inside the Pod. In figure [18.31](#fig-Jupyter-Kubernetes-Operators-Manager--pod-creation) `main.py` has been downloaded earlier from the official PyTorch repository and placed in the user’s home directory using `wget`: ```bash wget https://raw.githubusercontent.com/pytorch/examples/main/mnist/main.py ``` ![Jupyter Kubernetes Operators Manager: pod creation](/base-command-manager/product-manuals/_fern-img/4ceaacb552dcf90c50b0605bc2095c92facb8e38a93f8df66b55fa73758ca0d0.webp) *Figure 18.31: Jupyter Kubernetes Operators Manager: pod creation* ### 18.9.5 PVCs Tab A Persistent Volume Claim (PVC) is a user’s request for a specific amount of storage space within a Kubernetes cluster with defined characteristics. The `PVCs` tab (figure [18.32](#fig-Jupyter-Kubernetes-Operators-Manager--PVCs-overview-tab)) displays an overview of the PVCs. ![Jupyter Kubernetes Operators Manager: PVCs overview tab](/base-command-manager/product-manuals/_fern-img/a5baff830617a39a4a42cfc5c5d28f939c0cedba3976b9ecd06fbb2a6784120d.webp) *Figure 18.32: Jupyter Kubernetes Operators Manager: PVCs overview tab* The tab allows users to manage existing PVCs or create new ones via a dialog. To create a new PVC, several fields must be set. This can be done via a pop-up dialog, that comes up on clicking either of these buttons: * the `New PVC from Config` button: This allows a YAML file configuration to be submitted to set the fields. * the `New PVC` button: This allows the configuration to be set directly (figure [18.33](#fig-Jupyter-Kubernetes-Operators-Manager--PVC-creation)). ![Jupyter Kubernetes Operators Manager: PVC creation](/base-command-manager/product-manuals/_fern-img/86edf239d843261fb6f8a290cd802cf6adffdad9a1694a0c7674e47cb4eeadea.webp) *Figure 18.33: Jupyter Kubernetes Operators Manager: PVC creation* To create a new PVC directly, the following fields need to be set in the form: 1. **Storage class**: Storage classes need to be configured cluster-wide during installation, and can be either * `local-path`, or * `bright-ceph` 2. **Volume mode**: can be either * `Filesystem`, where the requested space is accessible as a formatted filesystem, or * `Block`, where the space is accessible as a raw block device. For the `local-path` storage class, only `Filesystem` is available. 3. **Instance name**: arbitrary name of the PVC. 4. **Instance size (GB)**: requested size. It can be ignored for `local-path`, as quotas are not supported for shared filesystems. The resulting volumes for a cluster \<*cluster_name*> are located under `/cm/shared/apps/kubernetes/` \<*`cluster_name`*>`/var/volumes` and are configurable during cluster setup. 5. **Select access mode**: * **ReadWriteOnce**: The volume can be mounted as read-write by a single node. Multiple pods can still access the volume if they are running on the same node. * **ReadOnlyMany**: The volume can be mounted as read-only by many nodes. * **ReadWriteMany**: The volume can be mounted as read-write by many nodes. ### 18.9.6 PSQL Tab The PSQL tab (figure [18.34](#fig-Jupyter-Kubernetes-Operators-Manager--PSQL-overview)) provides a simplified interface for interacting with the Zalando PostgreSQL Operator. It offers an overview of instances running on the cluster, it displays associated events, and provides credentials and access points for running PSQL databases. This information can be used later in Jupyter Notebooks or other PSQL clients. ![Jupyter Kubernetes Operators Manager: PSQL overview](/base-command-manager/product-manuals/_fern-img/60390d1af6f50dc476838f108bfb4be9a65d31fa112bc87748279f03b96bd9e2.webp) *Figure 18.34: Jupyter Kubernetes Operators Manager: PSQL overview* The tab allows users to manage existing PostgreSQL databases or to create new ones via a dialog. An example notebook is located at `/cm/shared/examples/jupyter/notebooks/psql-example.ipynb` (figure [18.35](#fig-Jupyter-Kubernetes-Operators-Manager--PSQL-notebook)). ![Jupyter Kubernetes Operators Manager: PSQL notebook](/base-command-manager/product-manuals/_fern-img/3808d6e17507887890abf3336abeb24e610dc69a94f481f29991eaaffd1131df.webp) *Figure 18.35: Jupyter Kubernetes Operators Manager: PSQL notebook* There are two buttons in figure [18.34](#fig-Jupyter-Kubernetes-Operators-Manager--PSQL-overview): * `New PSQL instance from Config`: allows a YAML file configuration that sets the fields to be submitted * `New instance`: allows the fields to be set directly (figure [18.36](#fig-Jupyter-Kubernetes-Operators-Manager--PSQL-instance-creation)) Clicking either button displays a pop-up dialog for creating a new PostgreSQL instance. For example, the `New instance` button displays the creation form in figure [18.36](#fig-Jupyter-Kubernetes-Operators-Manager--PSQL-instance-creation). ![Jupyter Kubernetes Operators Manager: PSQL instance creation](/base-command-manager/product-manuals/_fern-img/ff24500b92bdea2eda99ac6e3d46bf7cf01079ac8dc45b9d89bed9a65d62ea89.webp) *Figure 18.36: Jupyter Kubernetes Operators Manager: PSQL instance creation* The creation form accepts the following values: 1. **Instance name**: PSQL instance name. An auto-generated team name is prefixed to this name 2. **Database name**: arbitary database name 3. **Admin user** (Optional): if not set, the name of the admin user is set to the database name 4. **Instance size (GB)**: data storage size requirement for the volumes ### 18.9.7 Spark Tab The `Spark Jobs` tab (figure [18.37](#fig-Jupyter-Kubernetes-Operators-Manager--Spark-overview)) simplifies dealing with Spark clusters managed by the Google Spark Operator. The tab lets users list, remove, and create Spark instances using a simplified interface. If a particular option is not available in the simplified form, then a custom manifest can be uploaded. ![Jupyter Kubernetes Operators Manager: Spark overview](/base-command-manager/product-manuals/_fern-img/33b2f9b9c789ff21f8d86f39e88a76c446a46f958ac8c3e4d1afb11265f069df.webp) *Figure 18.37: Jupyter Kubernetes Operators Manager: Spark overview* There are two buttons in figure [18.37](#fig-Jupyter-Kubernetes-Operators-Manager--Spark-overview): * `New Spark Job from Config`: allows a YAML file configuration that sets the fields to be submitted * `New Spark Job`: allows the fields to be set directly (figure [18.38](#fig-Jupyter-Kubernetes-Operators-Manager--Spark-creation)) Clicking either button displays a pop-up dialog for creating a new Spark instance. For example, the `New Spark Job` button displays the creation form in figure [18.38](#fig-Jupyter-Kubernetes-Operators-Manager--Spark-creation). The creation form requires the following values: 1. **Name**: Arbitrary name of the Spark instance. 2. **Image**: Container image for the driver and executor. 3. **Spark Version**: Version of Spark used by the application. 4. **Command**: Application’s command with arguments. 5. **Executor Instances**: Number of executors. 6. **PVCs** (Optional): List of Persistent Volume Claims with mountpoints where these volumes are mounted. Used to store the application and datasets, as well as packed virtual environment archives. 7. **Virtual Environment** (Optional): Configuration for the ’spark.archives’ configuration variable for the Python virtual environment. * **PVC**: Name of the Persistent Volume Claim with empty Volume which are used to store the unpacked archive. * **Archive**: Full path to the archive. * **Python Path** (Optional): Python path (default `/bin/python`). 8. **Driver Resources** (Optional): * **Memory**: Amount of memory to request for the pod. * **GPU**: List of key-value pairs such as `nvidia.com/gpu, 3`. * **Cores**: Number of cores to use for the driver process. * **Environment**: List of key-value pairs for environment variables. 9. **Executor Resources** (Optional): Same as `Driver Resources`. 10. **Spark Config** (Optional): Custom configuration parameters if necessary. They are key-value pairs, documented in the Spark documentation at [https://spark.apache.org/docs/latest/configuration.html](https://spark.apache.org/docs/latest/configuration.html). #### 18.9.7.1 Example: Calculating Pi The simplest way to run Spark involves specifying just three values: the name, the script to execute, and the Spark version. For example, the container registry image `docker.io/apache/spark:3.5.1-python3` published by the Apache Spark project can be used, and an application to calculate the value for π is located at `/opt/spark/examples/src/main/python/pi.py` inside the image. ![Jupyter Kubernetes Operators Manager: Spark creation](/base-command-manager/product-manuals/_fern-img/3027c953e8343e738ce66993098fbdff1ebfe42c25070f5cedb21904a48e1c08.webp) *Figure 18.38: Jupyter Kubernetes Operators Manager: Spark creation* The progress of execution can be monitored in the `Pods` tab (figure [18.39](#fig-Jupyter-Kubernetes-Operators-Manager--pods-tab-Spark-example-execution-status)): ![Jupyter Kubernetes Operators Manager: pods tab Spark example execution status](/base-command-manager/product-manuals/_fern-img/2731d3f30fb6cb8d8b6f80f8fc50fa48326b594e34d083ce465a720433fbf5c1.webp) *Figure 18.39: Jupyter Kubernetes Operators Manager: pods tab Spark example execution status* When the job is completed, the overview page is updated (figure [18.40](#fig-Jupyter-Kubernetes-Operators-Manager--pods-tab-Spark-example-completed-status)): ![Jupyter Kubernetes Operators Manager: pods tab Spark example completed status](/base-command-manager/product-manuals/_fern-img/837d6529ee06d6b1afca550c1780e0aa7ed72cc9bca11a5c71149eda02e978c0.webp) *Figure 18.40: Jupyter Kubernetes Operators Manager: pods tab Spark example completed status* #### 18.9.7.2 Example: Running MNIST The source files for the MNIST example are located in the `/cm/shared/examples/jupyter/spark-operator-mnist` directory **Example** ```bash $ id uid=1001(alice) gid=1001(alice) groups=1001(alice) $ ls -l /home/alice/mnist/ total 12 -rwxr-xr-x 1 alice alice 241 Jun 28 16:02 create-venv.sh drwxr-xr-x 2 alice alice 76 Jun 28 17:10 data -rw-r--r-- 1 alice alice 2719 Jun 28 16:02 mnist.py -rw-r--r-- 1 alice alice 46 Jun 28 16:02 requirements.txt $ ls -l /home/alice/mnist/data total 124952 -rw-rw-r-- 1 alice alice 18303650 Oct 1 2019 mnist_test.csv -rw-rw-r-- 1 alice alice 109640201 Oct 1 2019 mnist_train.csv -rw-r--r-- 1 alice alice 91 Jun 28 16:02 README.md ``` In the example the user is unable to train the model defined in `mnist.py` right away, as the image `docker.io/apache/spark:3.5.1-python3` does not have the required NumPy library. So, before running the Spark job, the virtual environment archive needs to be created. The safest way to create the Python environment is to use the same context as where the archive is to be used later on. That way there are no missing or conflicting libraries and paths, and the Python version matches. The `docker.io/apache/spark:3.5.1-python3` image itself can therefore be used to create the archive. The script `create-venv.sh` creates the create archive from `requirements.txt` file and packs it. In this example, a newly persistent volume (PV) is used as an intermediate storage for all the data and scripts, because using a file system where all the files are located would be a security concern. The reason for this is that once the hostPath is used for the pod, the running process UID/GID is dropped, and becomes the same as the original user ([section 4.10.1 of the *Containerization Manual*](/containerization/kubernetes#4101-kyverno)). Spark images are usually not designed with this assumption in mind, and use UID 185 to run the application ([https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#user-identity](https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#user-identity)). With these criteria in mind, the steps to run the MNIST example are: 1. Create a PVC for storing scripts and datasets (figure [18.41](#fig-Jupyter-Kubernetes-Operators-Manager--PVC-creation-for-Spark-example)). Since the resulting persistent volume is expected to be mounted to all pods (driver and executors), its mode is set to `ReadWriteMany`. ![Jupyter Kubernetes Operators Manager: PVC creation for Spark example](/base-command-manager/product-manuals/_fern-img/a527dd39451b7771b9e787e47138fc75b166b788b9a308e43f03fe521075ab3f.webp) *Figure 18.41: Jupyter Kubernetes Operators Manager: PVC creation for Spark example* 2. Migrate data from the home folder to the volume. To do this, a migration job is created (figure [18.42](#fig-Jupyter-Kubernetes-Operators-Manager--Data-migration-job-creation-for-Spark-example)), and allowed to finish (figure [18.43](#fig-Jupyter-Kubernetes-Operators-Manager--Data-migration-job-successful-finish-for-Spark-example)). ![Jupyter Kubernetes Operators Manager: Data migration job creation for Spark example](/base-command-manager/product-manuals/_fern-img/729e4a3c75e3b5bdff0075b70a5dcb0fc9ca97852fcdd25aea06c130c87a33dd.webp) *Figure 18.42: Jupyter Kubernetes Operators Manager: Data migration job creation for Spark example* ![Jupyter Kubernetes Operators Manager: Data migration job for Spark example successfully finished](/base-command-manager/product-manuals/_fern-img/6412a399a2a2db692adeaed53287b361097be6aaf0a47485c146af19c9596dc7.webp) *Figure 18.43: Jupyter Kubernetes Operators Manager: Data migration job for Spark example successfully finished* 3. Create a pod to build a packed virtual environment (figure [18.44](#fig-Jupyter-Kubernetes-Operators-Manager--pod-creation-for-virtual-environment)). For this, `mnist-data-pvc` is mounted by the user as the `/mnist` directory inside the pod. The command `/mnist/create-venv.sh /mnist/requirements.txt /mnist/venv.tar.gz` is then executed. This places the `venv.tar.gz` archive next to the other files in the volume. ![Jupyter Kubernetes Operators Manager: pod creation for virtual environment](/base-command-manager/product-manuals/_fern-img/95de57bee4fc5ab544d4acbb62c520afdb70bb4351143d855fadd3069c5d83cb.webp) *Figure 18.44: Jupyter Kubernetes Operators Manager: pod creation for virtual environment* 4. (Optional) If necessary, data can be downloaded from the volume back to the user’s home directory, to save the `venv.tar.gz` for future use (figure [18.45](#fig-Jupyter-Kubernetes-Operators-Manager--migrate-venv-to-PVC)): ![Jupyter Kubernetes Operators Manager: migrate venv to PVC](/base-command-manager/product-manuals/_fern-img/64ec5d3a2c4e6fccd9bb7ab0cde566ae09f11c9fffb287652b386bd0bea7873b.webp) *Figure 18.45: Jupyter Kubernetes Operators Manager: migrate venv to PVC* The `venv.tar.gz` file is placed in the `/home/alice/mnist` directory. ```bash $ ls -l /home/alice/mnist/ total 261516 -rwxr-xr-x 1 alice alice 241 Jun 28 19:17 create-venv.sh drwxr-xr-x 2 alice alice 99 Jun 28 19:17 data -rw-r--r-- 1 alice alice 2719 Jun 28 19:17 mnist.py -rw-r--r-- 1 alice alice 46 Jun 28 19:17 requirements.txt -rw-r--r-- 1 alice alice 237896720 Jun 28 19:17 venv.tar.gz ``` 5. Another volume is created to store the unpacked virtual environment files (figure [18.46](#fig-Jupyter-Kubernetes-Operators-Manager--create-volume-for-unpacked-virtual-environment-files)). This is mounted to all pods (driver and executor). The driver unpacks the archive, and the content is used by the executors. ![Jupyter Kubernetes Operators Manager: volume creation for unpacked virtual environment files](/base-command-manager/product-manuals/_fern-img/21556ca283867554504347f007c4d5dae9edfdd1e2e5edeb73a3913222abb538.webp) *Figure 18.46: Jupyter Kubernetes Operators Manager: volume creation for unpacked virtual environment files* 6. The Spark job can now be created (figure [18.47](#fig-Jupyter-Kubernetes-Operators-Manager--create-Spark-job)): ![Jupyter Kubernetes Operators Manager: create Spark job](/base-command-manager/product-manuals/_fern-img/6c8d0585f0d6d707bc50f83f7365e2673a897412305cc8fc7a874e94278ebc5b.webp) *Figure 18.47: Jupyter Kubernetes Operators Manager: create Spark job* The execution progress can be monitored in the `Pods` tab (figure [18.48](#fig-Jupyter-Kubernetes-Operators-Manager--monitor-Spark-instance)): ![Jupyter Kubernetes Operators Manager: monitoring the Spark instance in the Pods tab](/base-command-manager/product-manuals/_fern-img/fd6ab4c2b7df6c4abf32af6db496773ceb5ba16c762bb7b14e6c96391e94ff58.webp) *Figure 18.48: Jupyter Kubernetes Operators Manager: monitoring the Spark instance in the Pods tab* ### 18.9.8 Events Tab The `Events` tab lists all events from the user’s namespace (figure [18.49](#fig-Jupyter-Kubernetes-Operators-Manager--events-overview)): ![Jupyter Kubernetes Operators Manager: events overview](/base-command-manager/product-manuals/_fern-img/9f7f46d2f6858d06197eeca3d9287e88f9fc918c13bdcc8572730edd1e173141.webp) *Figure 18.49: Jupyter Kubernetes Operators Manager: events overview* ## 18.10 Jupyter Environment Removal Before removing Jupyter, the administrator should ensure that all kernels have been halted, and that no user is still logged onto the web interface. Stopping the `cm-jupyterhub` service with users that are still logged in, or with running kernels, has undefined behavior. To remove Jupyter, the script `cm-jupyter-setup` must be run, either in interactive mode, or with the option `--remove`. Removing Jupyter does not remove or affect Kubernetes or WLM deployments. For a more complete cleanup, the following packages must be manually removed from the nodes involved in the Jupyter deployment: `cm-jupyter`, `cm-jupyter-local`, and `cm-npm-configurable-http-proxy`.