Running with Kubernetes#

Added in version 2.0.

Installing Docker#

First you will need to set up the repository.

Update the apt package index with the command below:

sudo apt-get update

Install packages to allow apt to use a repository over HTTPS:

1sudo apt-get install -y \
2    apt-transport-https \
3    ca-certificates \
4    curl \
5    gnupg-agent \
6    software-properties-common

Next you will need to add Docker’s official GPG key with the command below:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Verify that you now have the key with the fingerprint 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88, by searching for the last 8 characters of the fingerprint:

1sudo apt-key fingerprint 0EBFCD88
2
3pub   rsa4096 2017-02-22 [SCEA]
4    9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
5uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
6sub   rsa4096 2017-02-22 [S]

Use the following command to set up the stable repository:

1sudo add-apt-repository \
2"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
3$(lsb_release -cs) \
4stable"

Install Docker Engine - Community Update the apt package index:

sudo apt-get update

Install Docker Engine:

Please refer to Install Docker Engine on Ubuntu | Docker Documentation for a current installation procedure for Ubuntu.

Verify that Docker Engine - Community is installed correctly by running the hello-world image:

sudo docker run hello-world

More information on how to install Docker can be found in the Installing Docker section.

Installing Kubernetes#

Make sure Docker has been started and enabled before beginning installation:

sudo systemctl start docker && sudo systemctl enable docker

Execute the following to add apt keys:

1sudo apt-get update && sudo apt-get install -y apt-transport-https curl
2curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
3sudo mkdir -p  /etc/apt/sources.list.d/

Create kubernetes.list:

1cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
2deb https://apt.kubernetes.io/ kubernetes-xenial main
3EOF

Now execute the below to install kubelet, kubeadm and kubectl:

1sudo apt-get update
2sudo apt-get install -y -q kubelet=1.21.1-00 kubectl=1.21.1-00 kubeadm=1.21.1-00
3sudo apt-mark hold kubelet kubeadm kubectl

Reload the system daemon:

sudo systemctl daemon-reload

Disable swap#

1sudo swapoff -a
2sudo nano /etc/fstab

Note

Add a # before all the lines that start with /swap. # is a comment, and the result should look something like this:

1UUID=e879fda9-4306-4b5b-8512-bba726093f1d / ext4 defaults 0 0
2UUID=DCD4-535C /boot/efi vfat defaults 0 0
3#/swap.img       none    swap    sw      0       0

Initializing the Kubernetes cluster to run as a control-plane node#

Execute the following command:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16

Output:

 1Your Kubernetes control-plane has initialized successfully!
 2
 3To start using your cluster, you need to run the following as a regular user:
 4
 5    mkdir -p $HOME/.kube
 6    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 7    sudo chown $(id -u):$(id -g) $HOME/.kube/config
 8
 9Alternatively, if you are the root user, you can run:
10
11    export KUBECONFIG=/etc/kubernetes/admin.conf
12
13You should now deploy a pod network to the cluster.
14Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
15https://kubernetes.io/docs/concepts/cluster-administration/addons/
16
17Then you can join any number of worker nodes by running the following on each as root:
18
19kubeadm join <your-host-IP>:6443 --token 489oi5.sm34l9uh7dk4z6cm \
20        --discovery-token-ca-cert-hash sha256:17165b6c4a4b95d73a3a2a83749a957a10161ae34d2dfd02cd730597579b4b34

Following the instructions in the output, execute the commands as shown below:

1mkdir -p $HOME/.kube
2sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
3sudo chown $(id -u):$(id -g) $HOME/.kube/config

With the following command, you install a pod-network add-on to the control plane node. We are using calico as the pod-network add-on here:

kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml

You can execute the below commands to ensure that all pods are up and running:

kubectl get pods --all-namespaces

Output:

 1NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
 2kube-system   calico-kube-controllers-65b8787765-bjc8h   1/1     Running   0          2m8s
 3kube-system   calico-node-c2tmk                          1/1     Running   0          2m8s
 4kube-system   coredns-5c98db65d4-d4kgh                   1/1     Running   0          9m8s
 5kube-system   coredns-5c98db65d4-h6x8m                   1/1     Running   0          9m8s
 6kube-system   etcd-#yourhost                             1/1     Running   0          8m25s
 7kube-system   kube-apiserver-#yourhost                   1/1     Running   0          8m7s
 8kube-system   kube-controller-manager-#yourhost          1/1     Running   0          8m3s
 9kube-system   kube-proxy-6sh42                           1/1     Running   0          9m7s
10kube-system   kube-scheduler-#yourhost                   1/1     Running   0          8m26s

The get nodes command shows that the control-plane node is up and ready:

kubectl get nodes

Output:

1NAME             STATUS   ROLES                  AGE   VERSION
2#yourhost        Ready    control-plane,master   10m   v1.21.1

Since we are using a single-node Kubernetes cluster, the cluster will not schedule pods on the control plane node by default. To schedule pods on the control plane node, we have to remove the taint by executing the following command:

kubectl taint nodes --all node-role.kubernetes.io/master-

Refer to https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ for more information.

Installing Helm#

Execute the following command to download and install Helm 3.5.4:

1wget https://get.helm.sh/helm-v3.5.4-linux-amd64.tar.gz
2tar -zxvf helm-v3.5.4-linux-amd64.tar.gz
3sudo mv linux-amd64/helm /usr/local/bin/helm
4rm -rf helm-v3.5.4-linux-amd64.tar.gz linux-amd64/

Refer to helm/helm and https://helm.sh/docs/using_helm/#installing-helm for more information.

NVIDIA Network Operator#

Added in version 2.0.

Prerequisites#

Note

If Mellanox NICs are not connected to your nodes, please skip this step and proceed to NVIDIA GPU Operator.

The below instructions assume that Mellanox NICs are connected to your machines.

Execute the below command to verify Mellanox NICs are enabled on your machines:

lspci | grep -i "Mellanox"

Output:

10c:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
20c:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

Execute the below command to know which Mellanox Device is Active:

Note

Use the Device whichever shows as Link Detected: yes in further steps. Below command works only if you add the NICs before installing the Operating System.

for device in `sudo lshw -class network -short | grep -i ConnectX | awk '{print $2}' | egrep -v 'Device|path' | sed '/^$/d'`;do echo -n $device; sudo ethtool $device | grep -i "Link detected"; done

Output:

1ens160f0        Link detected: yes
2ens160f1        Link detected: no

Create the custom network operator values.yaml.

nano network-operator-values.yaml

Update the active Mellanox device from the above command.

 1deployCR: true
 2ofedDriver:
 3  deploy: true
 4  nvPeerDriver:
 5  deploy: true
 6  rdmaSharedDevicePlugin:
 7  deploy: true
 8  resources:
 9    - name: rdma_shared_device_a
10      vendors: [15b3]
11      devices: [ens160f0]
12operator:
13  repository: nvcr.io/nvaie
14  image: network-operator-4-0
15imagePullSecrets: [ngc-secrect]

For more information about custom network operator values.yaml, please refer Network Operator.

Add the NVIDIA repo:

Note

Installing Helm is required to install GPU Operator.

helm repo add mellanox https://helm.ngc.nvidia.com/nvaie

Update the Helm repo:

helm repo update

Install NVIDIA Network Operator#

Execute the commands below:

1kubectl label nodes --all node-role.kubernetes.io/master- --overwrite
2helm install -f ./network-operator-values.yaml -n network-operator --create-namespace --wait network-operator nvaie/network-operator-3-0

Validating the State of Network Operator#

Please note that the installation of the Network Operator can take a couple of minutes. How long the installation will take depends on your internet speed.

kubectl get pods --all-namespaces | egrep 'network-operator|nvidia-network-operator-resources'
 1NAMESPACE                           NAME                                                              READY   STATUS      RESTARTS   AGE
 2network-operator                    network-operator-547cb8d999-mn2h9                                 1/1     Running            0          17m
 3network-operator                    network-operator-node-feature-discovery-master-596fb8b7cb-qrmvv   1/1     Running            0          17m
 4network-operator                    network-operator-node-feature-discovery-worker-qt5xt              1/1     Running            0          17m
 5nvidia-network-operator-resources   cni-plugins-ds-dl5vl                                              1/1     Running            0          17m
 6nvidia-network-operator-resources   kube-multus-ds-w82rv                                              1/1     Running            0          17m
 7nvidia-network-operator-resources   mofed-ubuntu20.04-ds-xfpzl                                        1/1     Running            0          17m
 8nvidia-network-operator-resources   rdma-shared-dp-ds-2hgb6                                           1/1     Running            0          17m
 9nvidia-network-operator-resources   sriov-device-plugin-ch7bz                                         1/1     Running            0          10m
10nvidia-network-operator-resources   whereabouts-56ngr                                                 1/1     Running            0          10m

Please refer to the Network Operator page for more information.

NVIDIA GPU Operator#

Added in version 2.0.

NVIDIA AI Enterprise customers have access to a pre-configured GPU Operator within the NVIDIA NGC Catalog. The GPU Operator is pre-configured to simplify the provisioning experience with NVIDIA AI Enterprise deployments.

The pre-configured GPU Operator differs from the GPU Operator in the public NGC catalog. The differences are:

  • It is configured to use a prebuilt vGPU driver image (Only available to NVIDIA AI Enterprise customers).

  • It is configured to use the NVIDIA License System (NLS).

Install GPU Operator#

Note

The GPU Operator with NVIDIA AI Enterprise requires some tasks to be completed prior to installation. Refer to the document NVIDIA AI Enterprise for instructions prior to running the below commands.

License GPU Operator for CLS#

Add the NVIDIA AI Enterprise Helm repository, where api-key is the NGC API key for accessing the NVIDIA Enterprise Collection that you generated.

helm repo add nvaie https://helm.ngc.nvidia.com/nvaie --username='$oauthtoken' --password=api-key && helm repo update
  1. Copy the NLS license token in the file named client_configuration_token.tok.

  2. Create an empty gridd.conf file using the command below.

    touch gridd.conf
    
  3. Create Configmap for the NLS Licensing using the command below.

    kubectl create configmap licensing-config -n gpu-operator --from-file=./gridd.conf --from-file=./client_configuration_token.tok
    
  4. Create K8s Secret to Access NGC registry.

    kubectl create secret docker-registry ngc-secret --docker-server="nvcr.io/nvaie" --docker-username='$oauthtoken' --docker-password=’<YOUR API KEY>’ --docker-email=
  5. Install the GPU Operator with the command below.

    helm install --wait --generate-name nvaie/gpu-operator -n gpu-operator
    

License GPU Operator for DLS#

Add the NVIDIA AI Enterprise Helm repository, where api-key is the NGC API key for accessing the NVIDIA Enterprise Collection that you generated.

helm repo add nvidia https://nvidia.github.io/gpu-operator \
   && helm repo update

Prior to GPU Operator v1.9, the operator was installed in the default namespace while all operands were installed in the gpu-operator-resources namespace.

Starting with GPU Operator v1.9, both the operator and operands get installed in the same namespace. The namespace is configurable and is determined during installation. For example, to install the GPU Operator in the gpu-operator namespace.

1helm install --wait --generate-name \
2    -n gpu-operator --create-namespace
3    nvidia/gpu-operator

If a namespace is not specified during installation, all GPU Operator components will be installed in the default namespace.

GPU Operator with RDMA (Optional)#

Prerequisites#

After NVIDIA Network Operator installation is completed, execute the below command to install the GPU Operator to load nv_peer_mem modules.

helm install --wait gpu-operator nvaie/gpu-operator -n gpu-operator --set driver.rdma.enabled=true

Validating the Network Operator with GPUDirect RDMA#

Execute the below command to list the Mellanox NIC’s with the status:

kubectl exec -it $(kubectl get pods -n nvidia-network-operator-resources | grep mofed | awk '{print $1}') -n nvidia-network-operator-resources -- ibdev2netdev

Output:

1mlx5_0 port 1 ==> ens192f0 (Up)
2mlx5_1 port 1 ==> ens192f1 (Down)

Edit the networkdefinition.yaml.

1nano networkdefinition.yaml

Create network definition for IPAM and replace the ens192f0 with active Mellanox device for master.

 1apiVersion: k8s.cni.cncf.io/v1
 2kind: NetworkAttachmentDefinition
 3metadata:
 4annotations:
 5    k8s.v1.cni.cncf.io/resourceName: rdma/rdma_shared_device_a
 6name: rdma-net-ipam
 7namespace: default
 8spec:
 9config: |-
10    {
11        "cniVersion": "0.3.1",
12        "name": "rdma-net-ipam",
13        "plugins": [
14            {
15                "ipam": {
16                    "datastore": "kubernetes",
17                    "kubernetes": {
18                        "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
19                    },
20                    "log_file": "/tmp/whereabouts.log",
21                    "log_level": "debug",
22                    "range": "192.168.111.0/24",
23                    "type": "whereabouts"
24                },
25                "type": "macvlan",
26                "master": "ens192f0",
27                "vlan": 111
28            },
29            {
30                "mtu": 1500,
31                "type": "tuning"
32            }
33        ]
34    }
35EOF

Note

If you do not have VLAN-based networking on the high-performance side, please set “vlan”: 0

Validate the state of the GPU Operator#

Please note that the installation of the GPU Operator can take a couple of minutes. How long the installation will take depends on your internet speed.

kubectl get pods --all-namespaces | grep -v kube-system

Results:

 1NAMESPACE                NAME                                                              READY   STATUS      RESTARTS   AGE
 2default                  gpu-operator-1622656274-node-feature-discovery-master-5cddq96gq   1/1     Running     0          2m39s
 3default                  gpu-operator-1622656274-node-feature-discovery-worker-wr88v       1/1     Running     0          2m39s
 4default                  gpu-operator-7db468cfdf-mdrdp                                     1/1     Running     0          2m39s
 5gpu-operator-resources   gpu-feature-discovery-g425f                                       1/1     Running     0          2m20s
 6gpu-operator-resources   nvidia-container-toolkit-daemonset-mcmxj                          1/1     Running     0          2m20s
 7gpu-operator-resources   nvidia-cuda-validator-s6x2p                                       0/1     Completed   0          48s
 8gpu-operator-resources   nvidia-dcgm-exporter-wtxnx                                        1/1     Running     0          2m20s
 9gpu-operator-resources   nvidia-dcgm-jbz94                                                 1/1     Running     0          2m20s
10gpu-operator-resources   nvidia-device-plugin-daemonset-hzzdt                              1/1     Running     0          2m20s
11gpu-operator-resources   nvidia-device-plugin-validator-9nkxq                              0/1     Completed   0          17s
12gpu-operator-resources   nvidia-driver-daemonset-kt8g5                                     1/1     Running     0          2m20s
13gpu-operator-resources   nvidia-operator-validator-cw4j5                                   1/1     Running     0          2m20s

Please refer to the GPU Operator page on NGC for more information.