Created on Jan 20, 2020
Introduction
This document describes how to install Kubernetes(K8s) cluster with kubespray.
This guide assumes the following:
Hardware
All servers used during deployment have the same hardware specification and are equipped with a Mellanox ConnectX network card.Software
Ubuntu Server 18.04 operating system is installed on all servers with OpenSSH server packages.Network
DHCP server and DNS server is part of the IT infrastructure.
For each server, the server's name and IP address are reserved on the DHCP server.
All servers used in our deployment are connected to an Ethernet switch by single 100GbE Ethernet copper cable and have Internet access.
This deployment is HW independent, and we use NVIDIA gears as an example.
References
Solution Overview
Equipment
For K8s cluster deployment, you should use the same type of servers.
For additional info please refer to Kubespray Requirements.
Solution Logical Design
The MGMT node is only used to deploy a K8s cluster through Node1-Node5.
Network Configuration
Server names with network configurations are provided in the below table:
Host Configuration
Ubuntu Server 18.04 operating system is installed on all servers with OpenSSH server packages.
Please update/upgrade Ubuntu software packages and install the latest HWE kernel by running the below commands:
Server console
# apt-get update
# apt-get -y install linux-image-generic-hwe-18.04
# reboot
Deployment Guide
K8s Cluster Deployment and Configuration
SSH Private Key and SSH Passwordless Login
Please login to the Management server as root user and create SSH private key for configuring the password-less authentication on your computer by running the following command:
# ssh-keygen
Copy your SSH private key , such as ~/.ssh/id_rsa , to all nodes in your deployment by running the following command:
# ssh-copy-id root@nodename
Kubespray Configuration
Please install dependencies for running Kubespray with Ansible on the Management server:
# cd ~
# apt -y install python3-pip
# git clone https://github.com/kubernetes-sigs/kubespray.git
# cd kubespray
# pip3 install -r requirements.txt
Create new cluster configuration:
# cp -rfp inventory/sample inventory/mycluster
# declare -a IPS=(192.168.222.111 192.168.222.101 192.168.222.102 192.168.222.103 192.168.222.104)
# CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
Review and change the host configuration file - inventory/mycluster/hosts.yaml.
Example:
inventory/mycluster/hosts.yaml
all:
hosts:
node1:
ansible_host: 192.168.222.111
ip: 192.168.222.111
access_ip: 192.168.222.111
node2:
ansible_host: 192.168.222.101
ip: 192.168.222.101
access_ip: 192.168.222.101
node3:
ansible_host: 192.168.222.102
ip: 192.168.222.102
access_ip: 192.168.222.102
node4:
ansible_host: 192.168.222.103
ip: 192.168.222.103
access_ip: 192.168.222.103
node5:
ansible_host: 192.168.222.104
ip: 192.168.222.104
access_ip: 192.168.222.104
children:
kube-master:
hosts:
node1:
kube-node:
hosts:
node2:
node3:
node4:
node5:
etcd:
hosts:
node1:
k8s-cluster:
children:
kube-master:
kube-node:
calico-rr:
hosts:
Review and change cluster installation parameters under inventory/mycluster/group_vars:
# cat inventory/mycluster/group_vars/all/all.yml
# cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
The default Kubernetes CNI can be changed by setting the desired kube_network_plugin value parameter in inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml.
Install K8s Cluster Using Ansible Playbook
Deploy K8s cluster with Kubespray Ansible Playbook:
# ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml
The execution time for this step may take a while to finalize.
Example of a successful completion of the playbooks:
...
PLAY RECAP *********************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0
node1 : ok=611 changed=137 unreachable=0 failed=0
node2 : ok=442 changed=92 unreachable=0 failed=0
node3 : ok=404 changed=88 unreachable=0 failed=0
node4 : ok=404 changed=88 unreachable=0 failed=0
node5 : ok=404 changed=88 unreachable=0 failed=0
Wednesday 15 January 2020 10:39:14 +0200 (0:00:00.164) 0:16:20.903 *****
===============================================================================
container-engine/docker : ensure docker packages are installed --------------------------------------------------------------------------------------------------------------------- 65.67s
download : download_file | Download item ------------------------------------------------------------------------------------------------------------------------------------------- 53.43s
download : download_file | Download item ------------------------------------------------------------------------------------------------------------------------------------------- 50.71s
kubernetes/master : kubeadm | Initialize first master ------------------------------------------------------------------------------------------------------------------------------ 45.96s
kubernetes/kubeadm : Join to cluster ----------------------------------------------------------------------------------------------------------------------------------------------- 42.65s
download : download_file | Download item ------------------------------------------------------------------------------------------------------------------------------------------- 28.74s
download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 27.37s
download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 22.59s
kubernetes/master : Master | wait for kube-scheduler ------------------------------------------------------------------------------------------------------------------------------- 22.41s
bootstrap-os : Install python ------------------------------------------------------------------------------------------------------------------------------------------------------ 18.72s
kubernetes/preinstall : Install packages requirements ------------------------------------------------------------------------------------------------------------------------------ 15.50s
download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 14.98s
download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 12.85s
download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 12.33s
download : download_container | Download image if required ------------------------------------------------------------------------------------------------------------------------- 12.08s
download : download_container | Download image if required -------------------------------------------------------------------------------------------------------------------------- 9.86s
download : download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------- 9.61s
etcd : wait for etcd up ------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.91s
container-engine/docker : ensure docker-ce repository is enabled -------------------------------------------------------------------------------------------------------------------- 8.81s
bootstrap-os : Install dbus for the hostname module --------------------------------------------------------------------------------------------------------------------------------- 7.98s
Deployment Verification
Verify that the Kubernetes cluster is running by executing the following command on the K8s Master Node:
root@node1:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready master 4h v1.16.3 192.168.222.111 <none> Ubuntu 18.04.3 LTS 5.0.0-37-generic docker://18.9.7
node2 Ready <none> 4h v1.16.3 192.168.222.101 <none> Ubuntu 18.04.3 LTS 5.0.0-37-generic docker://18.9.7
node3 Ready <none> 4h v1.16.3 192.168.222.102 <none> Ubuntu 18.04.3 LTS 5.0.0-37-generic docker://18.9.7
node4 Ready <none> 4h v1.16.3 192.168.222.103 <none> Ubuntu 18.04.3 LTS 5.0.0-37-generic docker://18.9.7
node5 Ready <none> 4h v1.16.3 192.168.222.104 <none> Ubuntu 18.04.3 LTS 5.0.0-37-generic docker://18.9.7
Done!
Related Documents