Self Hosted On-Premise Kubernetes
- Motivation for Kubernetes
- Container Runtime
- Install kubeadm, kubelet, and kubectl
- Initialize the Control Plane Node
- Deploying a Static Site
Kubernetes (K8s) is an open-source system designed to manage, scale, and deploy containerized applications.
The motivation is scale.
When designed appropriately, an application can have its computation load distributed across multiple machines.
The promise of Kubernetes is to have many containerized applications running on various machines, ranging from cloud infrastructure to local computers (or a hybrid of both), with trivial effort required to update and deploy new applications.
It is a universal abstraction layer that can use any Infrastructure as a Service (IaaS) provider, allowing a developer to create applications using infrastructure-as-code concepts through a standardized API.
Most of the available documentation focuses on deploying applications to existing Kubernetes clusters, relying on Minikube (a single-node cluster running a virtual machine) for local development and testing.
Additionally, the popular way to use Kubernetes is through managed solutions like Google Kubernetes Engine or DigitalOcean Kubernetes.
This arrangement poses two primary disadvantages:
Diagnosing errors that occur between Minikube and the managed cloud Kubernetes service is now abstracted away from you. You have no guarantee that the Minikube behavior accurately reflects the production environment.
Minikube is not intended to be production facing, so using it as the self-hosted cluster for a highly available product is not advised.
I aimed to explore how complex it is to build, run, and maintain a Kubernetes cluster for myself.
I will be following the reference documentation for
My aim is to transition my static website, currently hosted on a single server by nginx in a non-containerized fashion, into something managed by Kubernetes.
I will document my checks, configurations, and issues as I go about this process.
Note: Confusion will be indicated with this visual blockquote. This indicates a step that required backtracking/revision from the reference documentation.
All listed tokens are no longer active as of the publishing of this blog post.
I have three virtual machines that I will be using. I have named them
The control-plane node will be
beryllium are worker nodes, making this a single control-plane cluster of two worker nodes.
All three of these VMs are running Ubuntu 18.04 and have been configured according to my server quality of life specification.
All swap partitions have been disabled in the
/etc/fstab file, verified by checking
swapon --show and seeing no results.
The product uuid is found running
sudo cat /sys/class/dmi/id/product_uuid.
The IPV4 addresses are private to the local network, while IPV6 addresses are public.
|Hostname||VCPUs||Disk||RAM||MAC address||IPV4 Address||IPV6 Address||UUID|
All three virtual machines have the following steps applied to them.
The security groups are configured to the required ports documentation.
As a form of sanity checking, verify that the ports work using
netcat. For example
nc -l 2379while on another server
echo "Testing Port 2379" | nc <ip> 2379
Although the documentation states that the ports for the worker and control plane nodes can be different, I combined them into one group of rules for simplicity.
|Direction||Ether Type||IP Protocol||Port Range||Remote IP Prefix||Remote Security Group|
|Ingress||IPv4||TCP||2379 - 2380||-||kube-node|
|Ingress||IPv6||TCP||2379 - 2380||-||kube-node|
|Ingress||IPv4||TCP||10250 - 10252||-||kube-node|
|Ingress||IPv6||TCP||10250 - 10252||-||kube-node|
|Ingress||IPv4||TCP||30000 - 32767||-||kube-node|
|Ingress||IPv6||TCP||30000 - 32767||-||kube-node|
Load the module
br_netfilter and ensure that the following sysctl variables are set.
As this module must be loaded, ensure that it is loaded after boot.
The lines for ip forwarding were not part of the original documentation, but required.
The documentation specified three types of container runtimes to choose from.
I have decided to use the docker.io runtime, although the containerd or CRI-O runtimes can also be used.
This must be installed on all nodes.
I had a lot of difficulty getting CRI-O to work, running into cri-o issue#3301 in my first attempt at setting up a cluster. My second attempt started from fresh servers using the default docker runtime.
When inspecting the output of
systemctl status docker, kernel does not support swap memory limit, cgroup rt period, and cgroup rt runtime warnings appeared:
All my virtual machines are currently running
Linux 4.15.0-101-generic as the kernel.
I am ignoring these errors for the time being.
These three packages must be installed on all of the machines.
kubeadm: the command to bootstrap the cluster.
kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.
kubectl: the command line util to talk to your cluster.
These commands only need to be done on the control plane node
- I created an
kube.udia.cato my IPv6 address for the cluster control plane node.
- I have chosen the default pod network add-on Calico. I will add the flag
--pod-network-cidr=192.168.0.0/16, as it is not currently in use within my network.
- I am relying on kubeadm to automatically detect my container runtime, as I have left everything as default.
- I will be using IPv6 addressing, and therefore will specify the
- I have run
kubeadm config images pullprior to
kubeadm initto verify connectivity to the gcr.io container image registry.
After a few moments, I was presented with a successful control-plane initialization message.
I ran the above steps for my regular user. As a non-sudo user, I installed the Calico pod network.
Afterwards, I logged into my two worker virtual machines and ran the join command (as root).
I have verified that my cluster is operational.
I will be using my existing site to deploy to my new cluster.
hugo, a directory containing the public facing production site is available at
I created a
Dockerfile at the root directory.
docker build . -t udia/udia.ca:blog to create a docker image.
This docker image contains the base nginx web server as well as the compiled public facing website.
Verify that this works by running the image locally.
localhost:8080 should serve your static site.
Now that a docker image has been created, we need to upload this image to a registry.
I will be using Dockerhub, but any registry would work.
Visiting https://hub.docker.com/repository/docker/udia/udia.ca, I see that a new image has been uploaded.
A Kubernetes pod is a group of one or more containers (e.g. Docker), with shared network, storage, and a specification for how to run the containers.
Create a file named
blog-deployment.yml that will be used to create a pod on the cluster.
A deployment can be created by running the following command.
Expose this deployment as a service by running the following command:
You now have the a static site hosted using Kubernetes.
It was very involved trying to setup Kubernetes to do such a simple task.
Choosing the incorrect container runtime or pod network addon will cause additional headaches that require specialized knowledge to fix.
Minor deviations from the default installation path are punishingly difficult to resolve.
There is sparse unofficial documentation and outdated blog posts outlining how to setup Kubernetes to use Let’s Encrypt for SSL certificates.
The approach that I have defined needs multiple virtual machines and the docker registry to deploy, making it a more complicated/over-engineered solution for a simple static site.
For the time being, I will remain with a single VM running nginx to handle my static site needs.
If horizontal scaling is required, I will likely use a DNS load balancer to multiple VMs all hosting the same content.
Although I do see value in the promise of Kubernetes, for my use case, it is not a solution I currently feel comfortable depending on.