> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Using Jupyter ## 13.1 Introduction Jupyter is a general name given to a system of software components integrated with Jupyter Notebook. [Chapter 18 of the *Administrator Manual*](/admin/jupyter) describes Jupyter for NVIDIA Base Command Manager, but is meant for cluster administrators rather than end users. For the orientation of end users, excerpts from that chapter that give a short overview of the Jupyter components are repeated in this introductory section here. How a user can use the components is described in greater detail in the remaining sections of this chapter. #### 13.1.0.1 What Is Jupyter Notebook? *Jupyter Notebook* ([https://jupyter-notebook.readthedocs.io/](https://jupyter-notebook.readthedocs.io/)), or Jupyter, is a client-server open-source application that provides a convenient way for a cluster user to write and execute *notebook documents* in an interactive environment. In Jupyter, a notebook document, or notebook, is content that can be managed by the application. Notebooks are organized in units called *cells* and can contain both executable code, as well as items that are not meant for execution. Items not meant for execution can be, for example: explanatory text, figures, formulas, or tables. Notebooks can also store the inputs and outputs of an interactive session. Notebooks can thus serve as a complete record of a user session, interleaving code with rich representations of resulting objects. These documents are encoded as JSON files and saved with the `.ipynb` extension. Since JSON is a plain text format, notebooks can be version-controlled, shared with other users and exported to other formats, such as HTML, LaTeX, PDF, and slide shows. #### 13.1.0.2 What Is A Notebook Kernel? A *notebook kernel* (often shortened to *kernel*) is a computational engine that handles the various types of requests in a notebook (e.g. code execution, code completions, inspection) and provides replies to the user ([https://jupyter.readthedocs.io/en/latest/projects/kernels.html](https://jupyter.readthedocs.io/en/latest/projects/kernels.html)). Usually kernels only allow execution of a single language. There are kernels available for many languages, of varying quality and features. #### 13.1.0.3 What Is JupyterHub? Jupyter on its own provides a single user service. *JupyterHub* ([https://jupyterhub.readthedocs.io/](https://jupyterhub.readthedocs.io/)) allows Jupyter to provide a multi-user service, and is therefore commonly installed with it. JupyterHub is an open-source project that supports a number of authentication protocols, and can be configured in order to provide access to a subset of users. #### 13.1.0.4 What Is JupyterLab? *JupyterLab* ([https://jupyterlab.readthedocs.io/](https://jupyterlab.readthedocs.io/)) is a modern and powerful interface for Jupyter. It enables users to work with notebooks and other applications, such as terminals or file browsers. It is open-source, flexible, integrated, and extensible. JupyterLab works out of the box with JupyterHub. It can be used to arrange the user interface to support a wide range of workflows in data science, scientific computing, and machine learning. JupyterLab is extensible with plugins that can customize or enhance any part of the interface. Plugins exist for themes, file editors, keyboard shortcuts, as well as for other components. #### 13.1.0.5 What Is A Jupyter Extension? Several components of the Jupyter environment can be customized in different ways with extensions. Some types of extensions are: * IPython extensions ([https://ipython.readthedocs.io/en/stable/config/extensions/#ipython-extensions](https://ipython.readthedocs.io/en/stable/config/extensions/#ipython-extensions)) * Jupyter Notebook server extensions ([https://jupyter-notebook.readthedocs.io/en/stable/extending/index.html](https://jupyter-notebook.readthedocs.io/en/stable/extending/index.html)) * JupyterLab extensions ([https://jupyterlab.readthedocs.io/en/stable/user/extensions.html](https://jupyterlab.readthedocs.io/en/stable/user/extensions.html)) Extensions are usually developed, bundled, released, installed, and enabled in different ways. Each extension provides a new functionality for a specific component. For example, JupyterLab extensions can customize or enhance any part of the JupyterLab user interface. Extensions can provide new themes, file viewers, editors and renderers for rich output in notebooks. They can also add settings, add keyboard shortcuts, or add items to the menu or command palette. #### 13.1.0.6 What Is Jupyter Enterprise Gateway? By default, Jupyter runs kernels locally, which can exhaust server resources. A resource manager can be used to deal with this issue. *Jupyter Enterprise Gateway* ([https://jupyter-enterprise-gateway.readthedocs.io/](https://jupyter-enterprise-gateway.readthedocs.io/)) (JEG) is a pluggable open-source framework that leverages local underlying resource managers, such as Slurm or Kubernetes, to distribute kernels across the compute cluster. In addition to scalability, JEG also provides an improved multi-user support, and more granular security for Jupyter. This makes Jupyter more useful for enterprise, scientific, and academic implementations. In BCM, all the technologies mentioned in these sections are combined to provide a powerful, customizable and user-friendly Jupyter Notebook web interface running on a lightweight, multi-tenant, multi-language, scalable and secure environment, ready for a wide range of enterprise scenarios. For convenience, in the following sections, *Jupyter* is generally used to collectively refer to Jupyter Notebook, JupyterHub, JupyterLab and Jupyter Enterprise Gateway. #### 13.1.0.7 BCM Jupyter Extensions For a default deployment of Jupyter, BCM installs and enables the following extensions to the Jupyter environment: * Jupyter Addons: A Jupyter Notebook server extension that performs API calls to CMDaemon and manages other server extensions; * Jupyter Kernel Creator (section [18.5](/admin/jupyter#185-jupyter-kernel-creator-extension)): A Jupyter Notebook server extension that provides a new interactive and user-friendly way to create kernels; * Jupyter VNC (section [18.7](/admin/jupyter#187-jupyter-vnc-extension)): A Jupyter Notebook server extension that enables remote desktops with VNC from notebooks; * JupyterLab Tools: A JupyterLab extension that exposes BCM server extensions functionalities to the users and shows the Cluster View section; * Jupyter WLM Magic (section [18.8](/admin/jupyter#188-jupyter-wlm-magic-extension)): An IPython extension that simplifies scheduling of workload manager jobs from the notebook; * Jupyter Kubernetes Operators Manager ([section 18.9 of the *Administrator Manual*](/admin/jupyter#189-jupyter-kubernetes-operators-manager)): An extension that integrates with Kubernetes clusters, and for which it provides basic overview and management features. The default setup for the user is described in the next sections. It may however be the case that the cluster administrator has customized Jupyter for the needs of the organization, in which case the description that follows may differ from the real life situation. For example, if the Jupyter login host is at an IP address of `10.2.75.147` then the URL the user uses to log in from with a browser (figure [13.1](#fig-JupyterHub-User-Login-Screen)) is by default: `https://10.2.75.147:8000` ![JupyterHub User Login Screen](/base-command-manager/product-manuals/_fern-img/8dd72f853527d8de794ef409bd69b9d7d11482fde06dd414cb6185b65908a7fd.webp) *Figure 13.1: JupyterHub User Login Screen* ## 13.2 Jupyter Notebook Examples The default BCM implementation of Jupyter provides a number of machine learning notebook examples that can be executed with Jupyter. The notebooks include some applications developed with TensorFlow, PyTorch, MXNet, and other frameworks. The applications can be found in the `/cm/shared/examples/jupyter/notebooks/` directory: ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/examples/jupyter/notebooks/ Keras+TensorFlow2-addition.ipynb R-iris.ipynb TensorFlow-minigo.ipynb MXNet-superresolution.ipynb Spark-pipeline.ipynb Pytorch-cartpole.ipynb Spark+XGBoost-mortgage.ipynb ``` The datasets needed to execute these notebooks can be found in the `/cm/shared/examples/jupyter/datasets/` directory: ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/examples/jupyter/datasets/ 880f8b8a6fd-mortgage-small.tar.gz kaggle-iris.csv ``` Users can copy these examples to their home directories, create or choose appropriate kernels to execute them, and interactively run them from Jupyter. In order to edit notebooks, the write permissions must be kept during the copy. It is the responsibility of users to make sure that the required modules are loaded by their Jupyter kernels. The list of frameworks and libraries required to run an example is usually available at the beginning of each notebook. ## 13.3 Jupyter Kernels In Jupyter, kernels are defined as JSON files. Any user that the cluster administrator has registered in the Linux-PAM system can list the available Jupyter kernels via the command line. The following example is run in the initial Jupyter environment: **Example** ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ jupyter kernelspec list Available kernels: python3 /cm/shared/apps/jupyter/14.0.0/share/jupyter/kernels/python3 ``` Each kernel directory contains a `kernel.json` file describing how Jupyter spawns that kernel: **Example** ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/apps/jupyter/current/share/jupyter/kernels/*/kernel.json /cm/shared/apps/jupyter/current/share/jupyter/kernels/python3/kernel.json ``` In addition to specifications for shared kernels, each user can define new personal ones in the home directory. By default, the Jupyter data directory for a user is located at `$HOME``/.local/share/jupyter`. This path can be verified with Jupyter by using the `--paths` option: **Example** ```bash [jupyterhubuser@basecm10 ~]$ jupyter --paths config: /home/jupyterhubuser/.jupyter /home/jupyterhubuser/.local/etc/jupyter /cm/local/apps/python39/etc/jupyter /cm/shared/apps/jupyter/14.0.0/etc/jupyter /etc/jupyter data: /home/jupyterhubuser/.local/share/jupyter /cm/local/apps/python39/share/jupyter /cm/shared/apps/jupyter/14.0.0/share/jupyter /usr/share/jupyter runtime: /home/jupyterhubuser/.local/share/jupyter/runtime ``` The simplest definition for a Python3 kernel designed to run on the login node is: ```bash { "argv": ["python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python" } ``` In the preceding kernel definition: * `argv`: is the command to be executed to locally spawn the kernel * `"display_name`": is the name to be displayed in the JupyterLab interface * `"language"`: is the supported programming language (`"language"`) * `"{connection_file}"` ([https://jupyter-client.readthedocs.io/en/stable/kernels.html#connection-files](https://jupyter-client.readthedocs.io/en/stable/kernels.html#connection-files)) is a placeholder, and is replaced by Jupyter with the actual path to the connection file before starting the kernel. The following kernel is Jupyter’s default Python 3 kernel distributed by BCM in the initial environment: **Example** ```bash [jupyterhubuser@basecm10 ~]$ cat /cm/shared/apps/jupyter/current/share/jupyter/kernels/python3/kernel.json "argv": [ "/cm/local/apps/python37/bin/python3.9", "-m", "ipykernel_launcher", "--InteractiveShellApp.extra_extension=cm_jupyter_wlm_magic", "--TerminalIPythonApp.extra_extension=cm_jupyter_wlm_magic", "-f", "connection_file" ], "display_name": "Python 3", "language": "python", "env": "PYTHONPATH": "/cm/shared/apps/jupyter/current/lib64/python3.9/site-packages:/cm/shared/apps/jupyter/ current/lib/python3.9/site-packages" ``` The two kernels are not very different. They differ from each other in the Python 3 binary path, the IPython extension (Jupyter WLM Magic), and the exported `PYTHONPATH` environment variable (`"env"`). ### 13.3.1 Jupyter Enterprise Gateway Kernels Jupyter is designed to run both the kernel processes, as well as the user interface (JupyterLab or Jupyter Notebook) on the same host. The kernel `{connection_file}` is therefore stored in the `~/.local/share/jupyter/runtime` directory, or in the `/run` directory. JupyterLab can delegate the task of spawning kernels to another component. During the BCM installation it is Jupyter Enterprise Gateway (JEG) that takes on this role, performing a complete lifecycle management of the kernel. That is: start, status monitoring, and termination. JEG requires an extended `kernel.json` definition to describe a particular process-proxy module to handle the kernel. A simple definition for a Python3 kernel designed to be scheduled via JEG is: ```bash { "argv": ["python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.processproxy.LocalProcessProxy", "config": {} } } } ``` In this example, the `"metadata"` entry has been added. It includes `"class_name"`, which defines a path to a Python class compatible with the JEG API; and `"config"`, an initial arguments dictionary passed to the JEG process. JEG can use several pluggable modules that are included in the BCM environment to interact with a wide range of resource managers, such as Kubernetes or Slurm. By doing so, kernels can be scheduled across compute nodes. BCM recommends that JEG kernels are created and used with the `Jupyter Kernel Creator` (section [18.5](/admin/jupyter#185-jupyter-kernel-creator-extension)) extension. ## 13.4 Jupyter Kernel Creator Extension Creating or editing kernels can be cumbersome and error-prone for users, depending on the features of the execution context desired for their notebooks. To provide a more user-friendly experience, BCM includes the *Jupyter Kernel Creator* extension in JupyterLab. This extension is accessed from the navigation pane in the JupyterLab interface, by clicking on the BCM icon. Jupyter Kernel Creator allows users to create kernels using the JupyterLab interface, without the need to directly edit JSON files. With this interface users can create kernels by customizing an available *template* according to their needs. A template can be considered to be the skeleton of a kernel, with several preconfigured options, and others options that are yet to be specified. Common customizations for templates include environment modules to be loaded, workload manager queues to be used, number and type of GPUs to acquire, and so on. Templates are usually defined by administrators according to cluster capabilities, programming languages and user requirements. Each template can provide different options for customizations. Administrators often create different templates to take advantage of different workload managers, programming languages and hardware resources. For example, an administrator may define a template for scheduling Python kernels via Kubernetes, another one for R kernels via Slurm, and yet another one for Bash kernels via Platform LSF. ### 13.4.1 BCM Predefined Kernel Templates To simplify Jupyter configuration for administrators, BCM distributes a number of pre-defined templates with Jupyter Kernel Creator. These templates can be used for default configurations of BCM workload managers, and can be customized and extended for more advanced use. Kernel templates defined by BCM can be found in the `Jupyter Kernel Creator` installation directory, under the `kerneltemplates` directory: ```bash [jupyterhubuser@basecm10 ~]$ ls /cm/shared/apps/jupyter/current/lib/python3.9/site-packages/\ cm_jupyter_kernel_creator/kerneltemplates/ filter.yaml jupyter-eg-kernel-k8s-py-spark jupyter-eg-kernel-pbspro-py39 jupyter-eg-kernel-k8s-bash jupyter-eg-kernel-k8s-r jupyter-eg-kernel-slurm-bash jupyter-eg-kernel-k8s-cmjkop-julia jupyter-eg-kernel-lsf-bash jupyter-eg-kernel-slurm-py39 jupyter-eg-kernel-k8s-cmjkop-ngc-py jupyter-eg-kernel-lsf-py39 jupyter-eg-kernel-slurm-pyxis-py39 jupyter-eg-kernel-k8s-cmjkop-py jupyter-eg-kernel-openpbs-bash jupyter-eg-kernel-uge-bash jupyter-eg-kernel-k8s-cmjkop-py-spark jupyter-eg-kernel-openpbs-py39 jupyter-eg-kernel-uge-py39 jupyter-eg-kernel-k8s-py jupyter-eg-kernel-pbspro-bash jupyter-eg-kernel-slurm-py-conda ``` #### 13.4.1.1 BCM Predefined Kernel Templates Seen By Users Users can view the available predefined kernel templates in the Jupyter web browser interface, within the `KERNEL TEMPLATES` section of the dedicated BCM extensions panel (figure [18.5](/admin/jupyter#18511-bcm-predefined-kernel-templates-seen-by-users)): ![JupyterLab BCM extensions section with kernel templates](/base-command-manager/product-manuals/_fern-img/19eea8f2206916f2281bc4b172d999373eb903c639c7f1cf72a9a58c5e4058a7.webp) *Figure 18.5: JupyterLab BCM extensions section with kernel templates* However, the templates provided by BCM are listed in the panel only if they can be used on the cluster. This means that the templates are listed only after the associated workload manager instance, or associated Kubernetes configuration (such as a Kubernetes operator), have been deployed by cluster manager utilities. For example: * After running the `cm-wlm-setup` cluster manager utility to deploy an OpenPBS workload manager, the `jupyter-eg-kernel-openpbs-bash` and `jupyter-eg-kernel-openpbs-py39` templates become available. The templates are listed as: * `Bash via OpenPBS` * `Python 3.9 via OpenPBS` and accessed via the navigation path: *menu* > *dedicated BCM extension panel* > *kernel templates section*. * After running the `cm-kubernetes-setup` cluster manager utility to deploy a Kubernetes cluster, the Kubernetes cluster instance is displayed (navigation path: *menu* > *dedicated BCM extension panel* > *Kubernetes clusters section*) Then, after running the `cm-jupyter-kernel-operator` cluster manager utility to deploy a Jupyter kernel operator package, and configuring a user ([section 6.3 of the *Containerization Manual*](/containerization/kubernetesoperators#63-the-jupyter-kernel-operator)), the templates `jupyter-eg-kernel-k8s-cmjkop-julia`, `jupyter-eg-kernel-k8s-cmjkop-py`, and `jupyter-eg-kernel-k8s-cmjkop-py-spark` become available. The templates are listed as: * `Julia on Kubernetes Operator` * `Python on Kubernete Operator` * `Python+Spark on Kubernetes Operator` and accessed via the navigation path: *menu* > *dedicated BCM extension panel* > *kernel templates section*. Users can instantiate a kernel template to create an actual kernel from the dedicated BCM extensions section using the `+` button of the template. A dialog is dynamically generated for the template being instantiated, and users are asked to fill a number of customization options defined by administrators (figure [18.6](/admin/jupyter#18511-bcm-predefined-kernel-templates-seen-by-users)): ![Jupyter kernel template customization screen](/base-command-manager/product-manuals/_fern-img/45f7592c0a838e12aa2fddcb8a9a4d37e28c2414d90d86612ce33aab0528283a.webp) *Figure 18.6: Jupyter kernel template customization screen* Once the template is completely customized, the kernel can be created. It automatically appears in the JupyterLab Launcher screen (figure [18.7](/admin/jupyter#18511-bcm-predefined-kernel-templates-seen-by-users)) and can be used to run notebooks or a console session (figure [18.7](/admin/jupyter#18511-bcm-predefined-kernel-templates-seen-by-users)): ![JupyterLab Launcher screen with new custom kernel](/base-command-manager/product-manuals/_fern-img/239b32aecaf366ec69c36b2b210bb445779ce1355c153e2e722bda733b27a561.webp) *Figure 18.7: JupyterLab Launcher screen with new custom kernel* A user who lists available Jupyter kernels via the command line now sees the newly-created kernel: ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ jupyter kernelspec list Available kernels: jupyter-eg-kernel-k8s-cmjkop-py-1gii7nm01 /home/alice/.local/share/jupyter/kernels/jupyter-eg-kernel\ -k8s-cmjkop-py-1gii7nm01 python3 /cm/shared/apps/jupyter/14.0.0/share/jupyter/kernels/python3 ``` The new kernel directory will contain the JSON definition generated by the Jupyter Kernel Creator: ```bash [jupyterhubuser@basecm10 ~]$ cat .local/share/jupyter/kernels/jupyter-eg-kernel-k8s-cmjkop-py-\ 1gii7nm01/kernel.json ``` ```bash { "language": "python", "display_name": "Python on Kubernetes Operator 221123133450", "metadata": { "process_proxy": { "class_name": "cm_jupyter_kernel_creator.eg_processproxies.k8scmjkop.KubernetesCMJupyterKernelOperator", "config": { "k8s_env_module": "kubernetes", "cmjk_template": "templates/cmjk.yaml.j2", "image": "jupyter/datascience-notebook", "image_pull_policy": "IfNotPresent", "namespace": "alice-restricted", "gpu_limit": 0 } } }, "argv": [ "python", "-m", "ipykernel_launcher", "-f", "/var/tmp/kernel-parm.json" ] } ``` Jupyter kernel names need not be unique. Users should therefore choose meaningful and distinguishable display names for their kernels. Doing so makes the JupyterLab Launcher screen easier to use. For convenience, a summary of the available kernel templates and their requirements is shown in table [18.1](/admin/jupyter#jupyter-kernel-templates): *Table 13.1: Available Jupyter kernel templates for BCM and their requirements* | **Template name** | **Requirement** | **Description** | | :-------------------------------------- | :--------------- | :------------------------------------------------------------------ | | `jupyter-eg-kernel-k8s-bash` | Kubernetes¹ | Bash via Kubernetes | | `jupyter-eg-kernel-k8s-py` | Kubernetes¹ | Python 3.9 via Kubernetes | | `jupyter-eg-kernel-k8s-py-spark` | Kubernetes² | Python 3.9 + Spark 3 via Kubernetes | | `jupyter-eg-kernel-k8s-r` | Kubernetes³ | R 3.4 via Kubernetes | | `jupyter-eg-kernel-k8s-cmjkop-ngc-py` | Kubernetes | NGC images via Jupyter Kernel Operator | | `jupyter-eg-kernel-k8s-cmjkop-julia` | Kubernetes | Jupyter official image via Jupyter Kernel Operator (using Julia) | | `jupyter-eg-kernel-k8s-cmjkop-py` | Kubernetes | Jupyter official image via Jupyter Kernel Operator (Python) | | `jupyter-eg-kernel-k8s-cmjkop-py-spark` | Kubernetes³ | Python + Spark via Jupyter Kernel Operator (using Python and Spark) | | `jupyter-eg-kernel-lsf-bash` | Platform LSF | Bash via Platform LSF | | `jupyter-eg-kernel-lsf-py39` | Platform LSF | Python 3.9 via Platform LSF | | `jupyter-eg-kernel-openpbs-bash` | Open PBS | Bash via Open PBS | | `jupyter-eg-kernel-openpbs-py39` | Open PBS | Python 3.9 via Open PBS | | `jupyter-eg-kernel-pbspro-bash` | PBS Professional | Bash via PBS Professional | | `jupyter-eg-kernel-pbspro-py39` | PBS Professional | Python 3.9 via PBS Professional | | `jupyter-eg-kernel-slurm-bash` | Slurm | Bash via Slurm | | `jupyter-eg-kernel-slurm-py-conda` | Slurm | Python and Conda 3.9 via Slurm | | `jupyter-eg-kernel-slurm-py39` | Slurm | Python 3.9 via Slurm | | `jupyter-eg-kernel-uge-bash` | GE | Bash via GE | | `jupyter-eg-kernel-uge-py39` | GE | Python 3.9 via GE | ¹ Docker image: *brightcomputing/jupyter-kernel-sample:k8s-py36-1.1.1* ² Docker image: *brightcomputing/jupyter-kernel-sample:k8s-spark-py37-1.2.1* ³ Docker image: *brightcomputing/jupyter-kernel-sample:k8s-r-1.0.0* DockerHub kernels page: [https://hub.docker.com/r/brightcomputing/jupyter-kernel-sample/tags](https://hub.docker.com/r/brightcomputing/jupyter-kernel-sample/tags) ### 13.4.2 Using Conda Kernels With Jupyter Conda is a package manager for languages. Miniconda installs a minimal Conda environment. #### 13.4.2.1 Installing Conda With Miniconda And Initializing Conda If Slurm is the workload manager that is installed and available to the Jupyter user, then Conda templates can be installed and made available to the user by installing Miniconda 3. Miniconda installation is described in the official Conda documentation at: [https://docs.conda.io/projects/miniconda/en/latest/](https://docs.conda.io/projects/miniconda/en/latest/) At the time of writing (February 2024), the “Quick command line install” for Linux at [https://docs.anaconda.com/free/miniconda/](https://docs.anaconda.com/free/miniconda/) explains how to carry out a Linux command line installation for Miniconda 3. The following is an excerpt from that text, slightly modified for BCM use: > These four commands quickly and quietly install the latest 64-bit version of the installer and then clean up after themselves. To install a different version or architecture of Miniconda for Linux, change the name of the `.sh` installer in the `wget` command. > > ```bash > [jupyterhubuser@basecm10 ~]$ mkdir -p ~/miniconda3 > [jupyterhubuser@basecm10 ~]$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \ > -O ~/miniconda3/miniconda.sh > [jupyterhubuser@basecm10 ~]$ bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 > [jupyterhubuser@basecm10 ~]$ rm -rf ~/miniconda3/miniconda.sh > ``` If the user has modules (section [2.3](/user/clusterusage#23-environment-modules)) loaded that set `$PYTHONPATH`—for example by default—then it can cause Python version incompatibilities later on when using a Conda environment that uses another version of Python. This is explained further in the section “Avoiding Python Version Incompatibilities With Conda” ([here](#13423-avoiding-python-version-incompatibilities-with-conda)). A warning is given during the Miniconda installation if an existing `$PYTHONPATH` is detected. However it is up to the user to ensure that modules are not left in place that interfere with regular Python usage within Conda, for example by checking the `~/.bash_profile` and similar files for what modules are normally loaded in the shell. After installation, the newly-installed Conda must be initialized: ```bash [jupyterhubuser@basecm10 ~]$ ~/miniconda3/bin/conda init bash ... modified /home/jupyterhubuser/.bashrc ==> For changes to take effect, close and re-open your current shell. <== ``` Users may use other shells, but initialization must be done with Bash. On restarting the shell, the base environment is indicated with the prompt: ```bash (base) [jupyterhubuser@basecm10 ~]$ ``` The Python version for the Conda environment can be seen by running `python -V`. Typically it differs from the system version. When a Miniconda installation is completed with the preceding `conda init bash` command, Conda templates become available in the kernel templates list (figure [13.5](#fig-JupyterHub-kernel-template-list-with-Conda)). ![Conda template item showing up in the menu](/base-command-manager/product-manuals/_fern-img/e7aa7adc06381c5313154632dc3b6a5506eb76366642936f6609ae32d9e5139b.webp) *Figure 13.5: Conda template item showing up in the menu* #### 13.4.2.2 Creating And Running A Conda Environment Compatible With Jupyter The Conda virtual environment by default has its own version of Python. The Conda virtual environment can be created with a minimal set of packages to run a Jupyter kernel. To make a kernel compatible with Jupyter Enterprise Gateway, the `cm-jupyter-eg-kernel-wlm` package needs to be installed in the Conda virtual environment with a particular version of Python. This can be carried out with: **Example** ```bash (base) [jupyterhubuser@basecm10 ~]$ conda config --append channels conda-forge (base) [jupyterhubuser@basecm10 ~]$ conda create -n myenv python=3.11 -y Channels: - defaults - conda-forge ... Executing transaction: done # # To activate this environment, use # # $ conda activate myenv # # To deactivate an active environment, use # # $ conda deactivate (base) [jupyterhubuser@basecm10 ~]$ module remove jupyter #do not mix Python versions (myenv) [jupyterhubuser@basecm10 ~]$ conda activate myenv (myenv) [jupyterhubuser@basecm10 ~]$ conda install -y -c file:///cm/shared/apps/jupyter/current/\ share/conda-repo cm-jupyter-eg-kernel-wlm Channels: - file:///cm/shared/apps/jupyter/current/share/conda-repo - defaults - conda-forge ... Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done ``` #### 13.4.2.3 Avoiding Python Version Incompatibilities With Conda In the preceding example, the Conda environment `myenv` is set explicitly to use Python version 3.11. Some BCM environment modules, such as the `jupyter` environment module. define the `$PYTHONPATH` environment variable. Before activating a Conda environment, unloading such BCM environment modules is recommended. For example, in the case of Jupyter, with `module unload jupyter`. This is because the value of `$PYTHONPATH` from Jupyter sets the version of Python that is used, which may conflict with that intended by Conda. Having Python modules that have a version incompatible with the one that of Conda can cause issues that are hard to debug. Besides Jupyter, software that has modules that can affect Python versions in this way are also found in the Machine Learning (ML) group. The ML group is the group that is specified by the ML selection option in [https://service.bcm.nvidia.com/packages-dashboard/](https://service.bcm.nvidia.com/packages-dashboard/). If needed, additional software can be installed into the environment from the Conda channels using its `-c|--channel` option. For example, PyTorch—which outside Conda is a package in the ML group— can be provided within Conda with: **Example** ```bash (myenv) [jupyterhubuser@basecm10 ~]$ conda install pytorch torchvision torchaudio pytorch-cuda=11.8 \ -c pytorch -c nvidia ``` The preceding PyTorch installation is based on the guidance at [https://pytorch.org/get-started/locally/](https://pytorch.org/get-started/locally/). Available environments appear in a list dropdown: ![Conda template: environment list dropdown](/base-command-manager/product-manuals/_fern-img/a6dec67d6674fd02ac2836445a8907326aef7cbf093d16061e2ea75d165e31d3.webp) *Conda template: environment list dropdown* ### 13.4.3 Using Enroot And Pyxis With Jupyter If Slurm is configured by the cluster administrator with Enroot and Pyxis support, then a template for running kernels inside Enroot containers appears in the list. Enroot must be configured by the administrator with both of the following settings enabled: * `Share raw images among users and nodes` * `Share unpacked image files among nodes` The first invocation of the command: `enroot list` must be performed by the administrator too. Before creating a kernel from a template, the image must be imported, so that it can be prepared to have the kernel run within it. The Jupyter package has a script to do this: **Example** ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ jupyter-enroot-import -h Usage: jupyter-enroot-import --container-image [--container-name ] [--as-sqsh] [--partition ] [--install-script] [jupyterhubuser@basecm10 ~]$ jupyter-enroot-import --container-image nvcr.io#nvidia/pytorch:23.12-py3 [jupyterhubuser@basecm10 ~]$ jupyter-enroot-import --container-image python:3.11 ``` The container can also be pulled as a SquashFS file: **Example** ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ jupyter-enroot-import --container-image nvcr.io#nvidia/pytorch:23.12-py3 \ --as-sqsh --container-name /home/alice/images/pytorch_23.12_py3.sqsh [jupyterhubuser@basecm10 ~]$ jupyter-enroot-import --container-image python:3.11 --as-sqsh \ --container-name /home/alice/images/python_3_11.sqsh ``` A job is submitted to pull and unpack the image. The script uses the default Slurm module that is available, and uses the default Slurm queue. If necessary, a specific module can be loaded, and a particular queue specified. The container name can also be customized. If the name is not specified, then it is chosen based on the container image name. **Example** ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ module load slurm/slurm/23.11.11 [jupyterhubuser@basecm10 ~]$ jupyter-enroot-import --container-image \ quay.io#jupyter/datascience-notebook:latest --partition defq ``` When the jobs are finished then the imported images can be displayed in the `enroot list` output: ```bash [jupyterhubuser@basecm10 ~]$ enroot list pyxis_nvcr_io_nvidia_pytorch_23_12_py3 pyxis_python_3_11 ``` The template can then be instantiated and the kernel created. If NGC containers are to be used, then a GPU resource must be added to the kernel. ![Pyxis template for Python](/base-command-manager/product-manuals/_fern-img/0ef9a554db16e2dec70cc90c7a4967aaa31f3eff0a929b5e48b8fa19c31032f2.webp) *Pyxis template for Python* Alternatively, an image based on the R language can be imported and used: **Example** ```bash [jupyterhubuser@basecm10 ~]$ module load jupyter [jupyterhubuser@basecm10 ~]$ jupyter-enroot-import --container-image quay.io#jupyter/r-notebook:latest ``` ![Pyxis template for R](/base-command-manager/product-manuals/_fern-img/e23cdebf424fbdf6505ccdc89088422232da1f84a6ae88f8a0213be15f0d723c.webp) *Pyxis template for R* ## 13.5 Jupyter VNC Extension ### 13.5.1 What Is Jupyter VNC Extension About? VNC (Virtual Network Computing) is a screen sharing service that can work in a browser. If VNC is allowed by the cluster administrator, then the Jupyter environment configured by BCM can be used to start and control remote desktops via VNC with the *Jupyter VNC* extension. ### 13.5.2 Enabling User Lingering User lingering is a systemd setting that sets a user manager for a user at boot and keeps it around after logout. This allows that user to run long-running sessions despite not being logged in. Enabling user lingering may be required for Jupyter VNC extension to run for a relatively complicated desktop such as KDE or GNOME. For each user on each machine where these environments are installed, the following command must be run: ```bash loginctl enable-linger ``` The command may also be carried out using prolog/epilog scripts in the chosen WLM. ### 13.5.3 Starting A VNC Session With The Jupyter VNC Extension Users can start a VNC session with the button added by Jupyter VNC (figure [18.21](/admin/jupyter#1873-starting-a-vnc-session-with-the-jupyter-vnc-extension)). Additional VNC parameters can be optionally specified. ![Starting Jupyter VNC session from kernel](/base-command-manager/product-manuals/_fern-img/7060da3e2998d861cdbf8131333daab962ccc0f54c7c3af696523b16a324aae0.webp) *Figure 18.21: Starting Jupyter VNC session from kernel* If VNC is available and correctly configured on the node where the kernel is running, then a new tab is automatically created by Jupyter VNC containing the new session (figure [18.22](/admin/jupyter#1873-starting-a-vnc-session-with-the-jupyter-vnc-extension)). A user can now freely interact in JupyterLab both with the notebook and with the desktop environment. ![Running Jupyter VNC session from kernel](/base-command-manager/product-manuals/_fern-img/bdd5acb8ebc7bff1e97d36d9f00e7a0e13c7873def75b335da8ec5c2c09ea8bf.webp) *Figure 18.22: Running Jupyter VNC session from kernel* To provide a user-friendly experience, Jupyter VNC also allows the graphical viewport to be resized, so that the desktop application can run full-screen (figure [18.23](/admin/jupyter#1873-starting-a-vnc-session-with-the-jupyter-vnc-extension)). ![Running Jupyter VNC session from kernel (full-screen)](/base-command-manager/product-manuals/_fern-img/0db7c42b4b7e04d33446169e91448309f7de709ef61997f8b6924761b6d8cae3.webp) *Figure 18.23: Running Jupyter VNC session from kernel (full-screen)* ### 13.5.4 Running Examples And Applications In The VNC Session With The Jupyter VNC Extension Once the VNC session is correctly started and the new JupyterLab tab has been created, Jupyter VNC automatically exports the `DISPLAY` environment variable to the running notebook (figure [18.24](/admin/jupyter#1874-running-examples-and-applications-in-the-vnc-session-with-the-jupyter-vnc-extension)). Doing so means that any application or library running in the notebook can make use of the freshly created desktop environment. An example of such a library is OpenAI Gym, a toolkit for developing and comparing reinforcement learning algorithms, that is distributed by BCM. Among the examples distributed by BCM (section [18.3](/admin/jupyter#183-jupyter-notebook-examples)), a notebook running PyTorch in the OpenAI Gym CartPole environment can be found. If executed after a VNC session has been started, a user can then observe the model being trained in real time in the graphical environment. ![Automatic configuration of DISPLAY environment variable](/base-command-manager/product-manuals/_fern-img/bb5e33735424c7738286805ae2448233bdeba4fc5625197ca9d136bd8d1f7dbc.webp) *Figure 18.24: Automatic configuration of DISPLAY environment variable* ## 13.6 Jupyter WLM Magic Extension In the Jupyter environment configured by BCM, the *Jupyter WLM Magic* extension can be used to schedule workload manager jobs from notebooks. The Jupyter WLM Magic extension is an IPython extension. It is designed to improve the capabilities of Jupyter’s default Python 3 kernel, which runs on the login node. The Jupyter WLM Magic extension should therefore not be used from kernels running on compute nodes, such as those typically created with BCM’s Jupyter Kernel Creator extension (section [18.5](/admin/jupyter#185-jupyter-kernel-creator-extension)), and submitted via Jupyter Enterprise Gateway. Indeed, compute nodes running these kernels are often incapable of starting workload manager jobs in many default WLM configurations. Jupyter WLM Magic extension makes it possible for users to programmatically submit WLM jobs, and then interact with their results. This can be done while using the Python programming language and its libraries, which are available in the notebook. Users submit jobs and check their progress from the login node. The actual computation is distributed by the underlying workload manager across compute nodes, which means that server resources are spared. Jupyter WLM Magic commands are available in the IPython kernel as *magic functions* ([https://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magic-functions](https://ipython.readthedocs.io/en/stable/interactive/tutorial.html#magic-functions)). A new line magic (`%`) and a new cell magic (`%%`) are now added in the kernel, according to the workload manager: * Platform LSF: `%lsf_job` and `%%lsf_job` * PBS Professional: `%pbspro_job` and `%%pbspro_job` * Slurm: `%slurm_job` and `%%slurm_job` * GE: `%uge_job` and `%%uge_job` A user can list the magic functions in the kernel to see if they are available, with Jupyter’s builtin command `%lsmagic` ([https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-lsmagic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-lsmagic)): **Example** ```bash In []: %lsmagic Out []: root: line: automagic:"AutoMagics" autocall:"AutoMagics" [...] slurm_job:"SLURMMagic" pbspro_job:"PBSProMagic" lsf_job:"LSFMagic" uge_job:"UGEMagic" cell: js:"DisplayMagics" javascript:"DisplayMagics" [...] slurm_job:"SLURMMagic" pbspro_job:"PBSProMagic" lsf_job:"LSFMagic" uge_job:"UGEMagic" ``` The magic functions introduced by this BCM extension share a similar syntax. For convenience, Slurm is used as an example in this section. However, the same instructions are valid for the other WLMs. Users can check which options are available for a WLM function with the line magic helper: **Example** ```bash In []: %slurm_job --help Out []: usage: %slurm_job [-h] [--module MODULE] [--module-load-cmd MODULE_LOAD_CMD] [--shell SHELL] [--submit-command SUBMIT_COMMAND] [--cancel-command CANCEL_COMMAND] [--control-command CONTROL_COMMAND] [--stdout-file STDOUT_FILE] [--stderr-file STDERR_FILE] [--preamble PREAMBLE] [--timeout TIMEOUT] [--check-condition-var CHECK_CONDITION_VAR] [--job-id-var JOB_ID_VAR] [--stdout-file-var STDOUT_FILE_VAR] [--stderr-file-var STDERR_FILE_VAR] [--dont-wait] [--write-updates WRITE_UPDATES] [--check-status-every CHECK_STATUS_EVERY] optional arguments: -h, --help show this help message and exit [...] ``` Line magic functions are typically used to set options with a global scope in the notebook. By doing so, a user will not need to specify the same option every time a job will be submitted via cell magic. For example, if two Slurm instances are deployed on the cluster and their associated environment modules are `slurm-primary` and `slurm-secondary`, a user could run the following line magic once to configure the Jupyter WLM Magic extension to always use the second deployment: **Example** ```bash In []: %slurm_job --module slurm-secondary Out []: ``` Now, jobs will always be submitted to `slurm-secondary`. This is more convenient than repeatedly defining the same module option for every cell magic upon scheduling a job: **Example** ```bash In []: %%slurm_job --module slurm-secondary Out []: In []: %%slurm_job --module slurm-secondary Out []: ``` It should be noted that line magic functions cannot be used to submit WLM jobs. Cell magic functions have to be used instead. A well-defined cell contains the WLM cell magic function provided by the extension, followed by the traditional job definition. For example, a simple MPI job running on two nodes can be submitted to Slurm by defining and running this cell: **Example** ```bash In []: %%slurm_job #SBATCH -J mpi-job-example #SBATCH -N 2 module load openmpi mpirun hostname Out []: COMPLETED STDOUT file content: /home/demo/.jupyter/wlm_magic/slurm-1.out node001 node001 node002 node002 ``` Users can take advantage of the Jupyter WLM Magic extension to store some information into Python variables about the job being submitted. The information could be the ID or the output file name, for example. Users can then later programmatically interact with them in Python. This feature is convenient when a user wants to, for example, programmatically carry out new actions depending on the job output: **Example** ```bash In []: %%slurm_job --job-id-var my_job_id --stdout-file-var my_job_out #SBATCH -J mpi-job-example #SBATCH -N 2 module load openmpi mpirun hostname Out []: COMPLETED STDOUT file content: /home/demo/.jupyter/wlm_magic/slurm-2.out node001 node001 node002 node002 In []: print(f"Job id {my_job_id} was written to {my_job_out}") print(f"Output lines: {open(my_job_out).readlines()}") Out []: Job id 2 was written to /home/demo/.jupyter/wlm_magic/slurm-2.out Output lines: ['node001\n', 'node001\n', 'node002\n', 'node002\n'] ``` Users can also exploit Python variables to define the behavior of the Jupyter WLM Magic extension. For example, they can define a Python boolean variable to submit a WLM job only if a condition is true: **Example** ```bash In []: run_job = 1 == 2 Out []: In []: %%slurm_job --check-condition-var run_job #SBATCH -J mpi-job-example #SBATCH -N 2 module load openmpi mpirun hostname Out []: Variable run_job is 'False'. Skipping submit. ```