Kubernetes Cluster on Pi
With Kubernetes Cluster on Raspberry Pi, you may orchestrate and manage your Docker containers with full control
Kubernetes Cluster
(Total Setup Time: 70 mins)
Similar to the previous MicroK8s cluster setup, I am using Ubuntu Server 20.04 LTS (64-bit) as my OS. Having a Kubernetes Cluster on Raspberry Pi, you will have more control over how the cluster configured.
After burning the image onto my 64GB SD card, an empty file named ssh is created at d:/boot for a headless setup.
Powering Up
(5 mins)
After powering up the Pi, update the OS to the latest by:
sudo apt update
sudo apt upgrade
Change its hostname by running:
sudo nano /etc/hostname
sudo nano /etc/hosts
Enable memory cgroup, by opening the file:
sudo nano /boot/firmware/cmdline.txt
Next, add the following at the end of the line:
cgroup=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1
Add the following /boot/firmware/usercfg.txt to disable WiFi and Bluetooth:
dtoverlay=disable-wifi
dtoverlay=disable-bt
For iptables to see bridged traffic, perform these:
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
Perform a reboot:
sudo reboot
Installing Docker
(5 mins)
First, referencing the previous post on external storage, storage setup are summarized in these steps:
# Get UUID of external HDD
sudo blkid
# Create the required folder
sudo mkdir /mnt/hdd
sudo nano /etc/fstab
# Add line to /etc/fstab
UUID=b89ca96d-5ff9-403e-8c7c-82a1a49e5f0d /mnt/hdd ext4 defaults 0 1
# Test mount without reboot
sudo mount -a
Second, from the previous preparation section on docker and gitea, these commands set docker up:
# Install docker-compose
sudo apt install docker-compose
# Check version
docker-compose version
# Add ubuntu user to docker group
sudo usermod -aG docker ubuntu
sudo mkdir -p /mnt/hdd/master1k8s/docker/images
Third, add the –data-root option to the ExecStart:
sudo nano /lib/systemd/system/docker.service
Fourth, add the –data-root option to the ExecStart, similar to this:
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --data-root=/mnt/hdd/master1k8s/docker/images
Last, to setup docker daemon, run this command:
# Run su commnad as root
sudo su -
# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
Installing kudeadm
(25 mins)
Add Kubernetes repository:
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
Next, install kubeadm and perform a reboot:
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo reboot
Creating Kubernetes Cluster
(20 mins)
Create the kubernetes cluster by running:
# Pass the container runtime sockets:
sudo kubeadm init --control-plane-endpoint master1k8s --service-cidr 10.96.0.0/12
# Reset the cluster creation when things went wrong and try init again
sudo kubeadm reset
[init] Using Kubernetes version: v1.18.5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master1k8s kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master1k8s] and IPs [10.96.0.1 192.168.100.51]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master1k8s localhost] and IPs [192.168.100.51 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master1k8s localhost] and IPs [192.168.100.51 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0710 11:09:46.065710 24688 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0710 11:09:46.069293 24688 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 31.009879 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master1k8s as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master1k8s as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: cnifgh.ptp6is3fyb60to81
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join master1k8s:6443 --token sxddbo.ouonf7ehvkkso3b2 \
--discovery-token-ca-cert-hash sha256:130451a73b3d4ff65fc4dbbfba4e78b648c7b75e90621ebc1fdba789135957b5 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join master1k8s:6443 --token sxddbo.ouonf7ehvkkso3b2 \
--discovery-token-ca-cert-hash sha256:130451a73b3d4ff65fc4dbbfba4e78b648c7b75e90621ebc1fdba789135957b5
If you plan to add nodes as in my case, please keep a copy of the above kubeadm init
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
The Kubernetes Cluster on Raspberry Pi status is verified with:
kubectl cluster-info
To allow the control-plane node to schedule pods, this command can be run (by default, for security reasons the kubernetes cluster will not schedule pods on the control-plane node):
kubectl taint nodes --all node-role.kubernetes.io/master-
Installing Networking Addons
(5 mins)
You may choose from any of the networking addons listing. For me, I use Calio:
curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml
watch kubectl get pods --all-namespaces
kubectl get nodes -o wide
Without installing the networking addons, the nodes status will be NotReady and pods status will be Pending.
Adding Leaf Nodes
(10 mins)
Firstly, install kubectl, kubelet and kubeadm, following the same steps as the above section for the master node.
Secondly, install docker:
sudo apt install docker.io
sudo usermod -aG docker ubuntu
su - ubuntu
sudo systemctl enable docker.service
Thirdly, add hostname of the master to /etc/hosts:
192.168.100.51 master1k8s
Lastly, join to the newly created kubernetes cluster:
sudo kubeadm join master1k8s:6443 --token sxddbo.ouonf7ehvkkso3b2 \
--discovery-token-ca-cert-hash sha256:130451a73b3d4ff65fc4dbbfba4e78b648c7b75e90621ebc1fdba789135957b5
Troubleshooting
[ERROR FileExisting-conntrack]: conntrack not found in system path
From the troubeshooting guide, install conntrack by:
sudo apt -y install conntrack
[WARNING FileExisting-socat]: socat not found in system path
Install socat by:
sudo apt -y install socat