I have been thinking of making a homelab for years, but living in a 700 sqft condo I don’t have the room for the basement rack of my dreams. A few months ago with the launch of the Raspberry Pi Model 4 B with 4GB of ram I started looking into building a cluster of them for tooling around at home. I found the Pi Dramble which give me the push to build it out for testing Kubernetes and eventually ESXi on ARM. The PoE HAT makes it cost a bit more, but makes it look nice and clean with one power plug needed.
4x Raspberry Pi 4 Model B/4GB - $55x4 = $220
4x Raspberry Pi PoE HAT - $21x4 = $84
4x Samsung 32GB EVO Plus Class 10 Micro SDHC - $13x4 = $52
1x Raspberry Pi 4 Heatsink Pack - $8
1x NETGEAR 5-Port Gigabit Ethernet PoE Switch - $45
1x Acrylic Dog Bone Stacking Case - $19
1x 10 Pack of 6” Cat6 - $14
Total Price as of 2020-02-27: $442
Open all the presents up, affix the heatsinks to the Pi, screw spacers onto the Pi with the PoE HAT, peel the paper backing off the acrylic case, and build the stack. Easy as PI 😂
Flashing SD Cards
For the OS on I went with Raspbian Buster Lite, no need for a GUI when I’m driving everything from my other computers and it means more headroom for workloads.
Balena Etcher is awesome for flashing SD cards with an image from ZIP or ISO, highly recommended. It takes about 3 min per card with the super speed cards above.
Be sure to enable SSH on the images by creating an empty file in the boot directory, on Windows by making a empty text file in the boot directory and naming it ssh with no extension or from a Linux variant by:
$ touch /Volumes/boot/ssh
Grab your favorite homelabing drink and repeat for the rest of the SD cards.
Slot in the SD cards, attach network cables, and power up the switch. After a minute or so the Pis should be powered up. Find the IPs however you like, looking on your router, nmap, or your IP scanner of choice. Once you find them you should be able to SSH into the Raspberry Pis as follows:
$ ssh pi@[IP Address]
Default password for Raspbian is: raspberry
Once connected to a Pi start by giving yourself a static IP:
$ sudo nano -w /etc/dhcpcd.conf
Find the interface eth0 area and edit in your static IP config for me on my raspberrypi-1 node that is:
static ip_address=10.10.10.51/24 static routers=10.10.10.254 static domain_name_servers=10.10.10.254
Next, run the Raspberry Pi setup tool. Change the login password from 1, Set your hostname from 2, localization from 4, and overclock if you are feeling like it.
$ sudo raspi-config
Once you select Finish the Pi will reboot and have your config ready to go.
Installing the prerequisites
From this point there is some replicated setup to do on all of the nodes. The easiest way is to do this is use something like Terminator to broadcast to a group.
One of the first things you need is to install a Container Runtime. I’m using Docker.
$ curl -sSL get.docker.com | sh && sudo usermod pi -aG docker
Then disable swap. Why? See the Github Issue
$ sudo dphys-swapfile swapoff && sudo dphys-swapfile uninstall && sudo update-rc.d dphys-swapfile remove
Next, add cgroup settings to the boot line, this allows resource limiting and allocation.
$ sudo cp /boot/cmdline.txt /boot/cmdline_backup.txt $ orig="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_enable=memory" $ echo $orig | sudo tee /boot/cmdline.txt
And finally for this part, add the Kubernetes repo, update, and install kubeadm which will pull all the packages.
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - $ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list $ sudo apt-get update -q && sudo apt-get install -qy kubeadm
After this, reboot and we’ll start initializing Kubernetes.
$ sudo reboot
Initialize the Kubernetes master
It’s time to setup Kubernetes. First on the list is to spin up the cluster with kubeadm. Because this is a testing cluster and I like to chaos monkey pull cables to see failover states quickly I’m going to make a fast failover config for the bringup.
$ nano kubeadm_conf.yaml
And copy the following config into the file. This will set the pod-eviction-timeout to 10s instead of 5min and node-monitor-grace-period to 10s instead of 40s.
apiVersion: kubeadm.k8s.io/v1beta2 kind: MasterConfiguration controllerManagerExtraArgs: pod-eviction-timeout: 10s node-monitor-grace-period: 10s
Save and run:
$ sudo kubeadm init --config kubeadm_conf.yaml
After a few minutes you’ll see something like the following:
... Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join --token TOKEN 10.10.10.51:6443 --discovery-token-ca-cert-hash HASH
Follow the instructions to setup your user folders:
$ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config
Verify that your node is up and running
$ kubectl get nodes NAME STATUS ROLES AGE VERSION raspberrypi-1 NotReady master 5m v1.17.3
NotReady is fine for now, it won’t show as ready until we install a container network.
Setting up the Worker Nodes
Assuming you did the pre-reqs above on the worker nodes, spinning them up is one line taken from when the master was initialized:
$ sudo kubeadm join --token TOKEN 10.10.10.51:6443 --discovery-token-ca-cert-hash HASH
Repeat for each node and they are good to go.
Setup your Container Network
A container network is needed for the nodes to communicate with each other. There are a bunch of solutions out there, such as NSX-T. weave-net is the one I’ve seen most recommended for easy setups. Run the following on the master node to install it.
$ kubectl apply -f “https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d ‘\n’)
With this done, your cluster should be up and running. Verify by getting your nodes again:
$ kubectl get nodes NAME STATUS ROLES AGE VERSION raspberrypi-1 Ready master 15m v1.17.3 raspberrypi-2 Ready <none> 13m v1.17.3 raspberrypi-3 Ready <none> 11m v1.17.3 raspberrypi-4 Ready <none> 9m v1.17.3
Now you are ready to go! Find some projects to do on Kubernetes, an autoscaling 3 tier app with a loadbalancer for instance.
- Add a USB 3 NVME drive to the master node to provide Persistant Volume storage from a NFS Share
- Get more USB 3 drives and test Gluster
- Setup MetalLB
- Get another set of SD Cards for ESXi on ARM
- Maybe get a computer capable of nested VCF, install nested VCF on it, add in Pis as a NFS backed WLD