NVIDIA Autonomous Job Recovery

5.1 Introduction

NVIDIA Mission Control can provide autonomous job recovery. This capability can be installed on a cluster managed by BCM.

Autonomous job recovery can

• monitor Kubernetes jobs and pods

• monitor Slurm jobs and scheduling

and analyze what is being monitored. If it detects processes going wrong, then it can take action to recover from what is wrong, with the aim of reducing downtime.

Autonomous job recovery can display its monitoring visually in a Grafana dashboard using Loki logging.

Autonomous job recovery can be installed and integrated into BCM if Kubernetes and Slurm are installed first.

The physical hardware requirements for a cluster running autonomous job recovery are:

• the nodes should have at least 16 GB of RAM

• the cluster should have at least 30 CPU cores

• the cluster should have at least 64 GB RAM on the head node

• the cluster should have at least 20GB of persistent storage

5.2 Prerequisites To Install Autonomous Job Recovery: Kubernetes Installation

To install autonomous job recovery, Kubernetes should already have been installed and integrated with BCM with appropriate components. Kubernetes installation and integration is typically done with the cm-kubernetes-setup utility (section 4.2 of the Containerization Manual).

5.2.1 Kubernetes Components Needed For Installation Of Autonomous Job Recovery

The Kubernetes deployment procedure should be carried out so that it includes at least the following components (figure 5.1):

Figure 5.1: Selection of Kubernetes components for installing autonomous job recovery

• Prometheus Operator Stack

• Prometheus Adapter

• Grafana Promtail

• Kubernetes Metrics Server

• Kubernetes State Metrics

• Ingress NGINX Controller

5.2.2 Settings For Kubernetes Components Used In Autonomous Job Recovery Installation

In other screens that come up with the preceding components selection, the following settings should be used:

• In the screen that asks Do you want to install Kyverno Policy Engine?, which comes just before the components selection, the answer must be no. The default answer is yes. Kyverno and the NVIDIA Mission Control Kubernetes security policies should be installed afterwards, once autonomous job recovery and autonomous hardware recovery have been installed. How to install and configure them is described in the Kubernetes Security Hardening section of the NVIDIA Mission Control Software Installation Guide:
  https://docs.nvidia.com/mission-control/docs/nmc-software-installation-guide/2.3.1/nmc-kube-security-guide.html

• In the screen asking to configure the Kubernetes StorageClass, a local path storage class should be enabled, with the default NFS storage enabled (figure 5.2):

Figure 5.2: Setting the storage class

• In the screen that asks about Loki access (figure 5.3):

  • Loki should be added as the data source for Grafana

  • the Loki API should be exposed for Ingress

Figure 5.3: Setting up Loki as data source for Grafana, and Loki access via Ingress

• In the screen that configures logging for Grafana (figure 5.4), Grafana Promtail is recommended to have its collection disabled for /var/log from the Kubernetes worker nodes.

Figure 5.4: Log collection configuration for Grafana Promtail

5.3 Prerequisites To Install Autonomous Job Recovery: Slurm Installation

To install autonomous job recovery, Slurm should already have been installed and integrated with BCM. Slurm installation and integration is typically done with the cm-wlm-setup utility (section 7.3.2 of the Administrator Manual).

If Slurm is not to be run during regular use, then Slurm should still be installed before autonomous job recovery is installed, despite autonomous job recovery not being applied to Slurm during regular use. This is because autonomous job recovery installation makes use of Slurm configuration.

5.4 Autonomous Job Recovery Installation With cm-mission-control-setup

If Kubernetes has been installed (section 5.2), and Slurm has been installed (section 5.3), then autonomous job recovery can be installed with the cm-mission-control-setup utility. The cm-mission-control-setup utility is part of the cm-setup package.

5.4.1 Credentials Required To Install Autonomous Job Recovery When Running cm-mission-control-setup

The following credentials are required to install autonomous job recovery:

• Ingress credentials for Loki API access. This would have been set up during the cm-kubernetes-setup session.

• MySQL root password for the main CMDaemon database. By default this is the same as the root user password of the operating system.

• NVCR personal token for operator and container images access. The personal token is a string with the prefix: nvapi-

5.4.2 Running cm-mission-control-setup

The cm-mission-control-setup script should be run without options on the active head node. It brings up a TUI dialog (figure 5.5):

Figure 5.5: Selection from cm-mission-control-setup of autonomous job recovery

If there is more than one Kubernetes instance on the cluster, then a screen appears asking for one of them to be selected for the installation of autonomous job recovery.

If there is more than one Slurm WLM instance on the cluster, then a screen appears asking for one of them to be selected for the installation of autonomous job recovery.

Next, a screen appears asking for the MySQL access credentials.

Figure 5.6: Autonomous job recovery deployment configuration: MySQL access

Next, a screen appears asking for the Helm repository credentials.

Figure 5.7: Autonomous job recovery deployment configuration: Helm repository access

Next, a screen appears asking for the credentials for Loki API access via Ingress.

Figure 5.8: Autonomous job recovery deployment configuration: Loki API access via Ingress

The configuration settings can then be saved and deployment of autonomous job recovery is started.

Deployment can take up to 30 minutes.

5.5 Post-installation Checks

5.5.1 Grafana Access

Grafana for the Kubernetes Prometheus stack is exposed by default at:

https:// <cluster IP address>/grafana/login

as set by the Kubernetes Ingress NGINX Controller.

The Grafana administrator account is admin. Its password is generated automatically during deployment and stored in the Kubernetes secret kube-prometheus-stack-grafana, in the prometheus namespace. The password is retrievable with:

Example

root@basecm11:~# kubectl get secret kube-prometheus-stack-grafana -n prometheus \
-o jsonpath="{.data.admin-password}" | base64 -d

This password should be changed after installation, since anyone with read access to the prometheus namespace can retrieve it. The password can be changed from the Grafana user menu after logging in.

5.5.2 Autonomous Job Recovery Failure When The Slurm Daemon Spool Directory Is Not Cleaned Up

The slurmd spool directory is normally cleaned up automatically when a compute node reboots. The slurmd spool directory path is defined in the Slurm configuration file slurm.conf by the parameter SlurmdSpoolDir. SlurmdSpoolDir has a default value of /var/spool/slurmd.

A clean slurmd spool is needed after rebooting for Slurm to function correctly. If the spool directory contents persist after a reboot, then it means that a job may not resume properly.

The spool directory contents can persist in, for example, the following use case:

• when autonomous job recovery decides to reboot a node where a job is running, and

• if the job is submitted with the --requeue command line option, and

• it uses PMIX

In that use case, Slurm does not clean up PMIX temporary files such as stepd.slurm.pmix.$SLURM_JOBID.0 from the default spool directory. So when the job starts on the rebooted node, slurmstepd cannot start the job again because the file exists.