kubernetes

excerpt

Of the complete kubernetes site. Kubernetes is a cluster of nodes, or at least 1 node. The control plane manages the nodes and the pods/workloads that run on those nodes.

Typically all control plane components are started on a separate machine, although all components could be running on any machine in the cluster.

The master or control plane consists of these components :

• kube API server : this is the interface that kubectl talks to, you can also use a client api lib if needed.
• etcd : HA key value store for all cluster data
• kube-scheduler : watches for new pods and selects a node for them to run on.
• kube-controller-manager : controller processes
• cloud-controller-manager : cloud specific logic

The nodes always have these components:

• container runtime, like docker.
• kubelet : makes sure containers are running in a Pod
• kube-proxy : handles all networking for the node

configmap

This is a dynamic external configuration of your app. For instance if you change the db url in the frontend, you would alter configmap.

NOT credentials.. for that you use 'secrets'.

secrets

This is like configmap, but contains passwords and keys.

volumes

k8s does not manage data persistence, you should do that explicitly with Volumes. Volumes CAN be local, or any external storage as well.

Volumes should be regarded as hard disks plugged into the cluster.

work instruction

You can create a volume on the website, 20GB costs 2 euro/month so 24 euro a year. Network Netherlands occupies 1.1GB , Europe at least 50GB probably more !

But do not create a volume through the interface, create it with .yml files !!

Follow this guide exactly, if you don't have the personal key do that bit as well :

visit

Deleting the persistent volume claim can help in case of trouble, but remove the pod that references it first !!

However these volumes are really for linodes, and kubernetes has an automated way of getting volumes. Much like node balancers they do create billable resources so pay attention.

Create a Persistent Volume Claim (pvc) like this :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: network-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: linode-block-storage-retain

10 GB is actually the smallest option, so stick with that.

Run it like always :

kubectl apply -f pvc.yml
kubectl get pvc
NAME          STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
network-pvc   Bound    pvc-da98655485c441df   10Gi       RWO            linode-block-storage-retain   3m22s

If you look on the linode site, a volume was created with 10Gb and in Frankfurt, so it logically took storage where the pod is located as well. I don't know if 10 Gb cost less, we will see in the account info.

Attaching a pod

Probably it is not possible to attach a pod to a volume, but let's just create a new one for the network and backend.

apiVersion: v1
kind: Pod
metadata:
  name: backend-pod
  labels:
    app: backend-pod
spec:
  containers:
    - name: owncloud
      image: owncloud/server
      ports:
        - containerPort: 8080
      volumeMounts:
      - mountPath: "/mnt/data/files"
        name: network-pvc
  volumes:
    - name: pvc-example
      persistentVolumeClaim:
        claimName: network-pvc

StatefullSet

Apps having state (databases) need to use statefullset's . Much harder to setup than deployments

debugging

First get the pod information :

$ kubectl get pods
NAME                                             READY   STATUS    RESTARTS   AGE
doc-deployment-6864c8ff4-2wfj9                   1/1     Running   0          26h
doc-deployment-6864c8ff4-qkg7r                   1/1     Running   0          26h
doc-pod                                          1/1     Running   0          26h
docker-hub                                       1/1     Running   0          34h
nginx-ingress-controller-64b5588659-th6wm        1/1     Running   0          37h
nginx-ingress-default-backend-778694598b-wkndb   1/1     Running   0          37h

So two pods are running for doc-deployment, if not see the section below this for debugging. You can get logs from each of them with :

$ kubectl logs doc-deployment-6864c8ff4-2wfj9
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.2.0.52. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.2.0.52. Set the 'ServerName' directive globally to suppress this message
[Tue Dec 22 07:08:18.191124 2020] [mpm_event:notice] [pid 1:tid 139987641517184] AH00489: Apache/2.4.46 (Unix) configured -- resuming normal operations
[Tue Dec 22 07:08:18.191519 2020] [core:notice] [pid 1:tid 139987641517184] AH00094: Command line: 'httpd -D FOREGROUND'
10.2.0.35 - - [22/Dec/2020:08:18:26 +0000] "GET /index.html HTTP/1.1" 200 14090
10.2.0.35 - - [22/Dec/2020:08:36:29 +0000] "GET /klopt/index.html HTTP/1.1" 200 18266
10.2.0.35 - - [22/Dec/2020:08:37:23 +0000] "GET /virtualization/index.html HTTP/1.1" 200 7626
10.2.0.35 - - [22/Dec/2020:08:37:25 +0000] "GET /_static/css/fonts/lato-normal-italic.woff2?4eb103b4d12be57cb1d040ed5e162e9d HTTP/1.1" 200 195704

Or even better, log in to the pod ! Use exec -it :

$ kubectl exec -i -t doc-deployment-6864c8ff4-2wfj9 -- /bin/ls
bin  build  cgi-bin  conf  error  htdocs  icons  include  logs  modules
$ kubectl exec -it doc-deployment-6864c8ff4-2wfj9 -- /bin/bash
# ls 
# bin  build  cgi-bin  conf  error  htdocs  icons  include  logs  modules
# find . | grep kubernetes 
./htdocs/_raw_sources/virtualization/kubernetes.rst.txt
./htdocs/_raw_sources/.doctrees/virtualization/kubernetes.doctree
./htdocs/virtualization/kubernetes.html
./htdocs/_sources/virtualization/kubernetes.rst.txt
./htdocs/_sources/.doctrees/virtualization/kubernetes.doctree

• -it could be read as interactive terminal, but in fact it means :
• -i : --stdin=false: Pass stdin to the container
• -t : --tty=false: Stdin is a TTY

Pod not started

When you don't get the 'running' state but something like :

NAME                                             READY   STATUS             RESTARTS   AGE
backend-deployment-5c6b8c7644-h6bzl              0/1     CrashLoopBackOff   98         7h58m
backend-pod                                      0/1     CrashLoopBackOff   98         7h58m

There is nothing to get logs from or log into, so you need to get some more info from the pod with 'describe'. View the section with the container and look for clues, in this case the "Message" line :

Containers:
  backend:
    Container ID:   docker://3a483d848ae1294919fb35076fe02f99f2dede27aced384ca8d093ba83a0a25e
    Image:          kees/klopt:backend
    Image ID:       docker-pullable://kees/klopt@sha256:9ebebc7d7cda50ccd72cfbea655188e1697ffc88977318e4e099f1f2bbae467f
    Port:           443/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: container_linux.go:349: starting container process caused "exec: "backend": executable file not found in $PATH": unknown
      Exit Code:    127
      Started:      Fri, 08 Jan 2021 07:43:48 +0100
      Finished:     Fri, 08 Jan 2021 07:43:48 +0100
    Ready:          False
    Restart Count:  97

You can see that it keeps on restarting because it cannot find the 'backend' executable. To debug this you can start the container by hand :

docker run -it kees/klopt:backend ash
/ # backend
ash: backend: not found
/ # ./backend
opening...

So it seems we need to explicitly use this in Dockerfile :

CMD ["./backend"]

To not go through the complete Jenkins pipeline, test it with :

docker build -t kees/klopt:backend .
docker push kees/klopt:backend
kubectl apply -f deploy.yml

This does take a long time, because the steps of compiling and installing the run image are in the same Dockerfile. Also you need to force the deployment to reload or you will get : deployment.apps/backend-deployment unchanged

Easiest is to set replicas to 0 and back again and redeploy twice, actually it's the only way because when you remove a deployment it will be automatically recreated by the scheduler !.

On the command line this can be done with :

kubectl scale deployment/backend-deployment --replicas=0
kubectl scale deployment/backend-deployment --replicas=2

After doing this 'fix' it was still not working but now we do get a log result :

kubectl logs backend-deployment-5c6b8c7644-9l6l8
opening 
opening 
opening 
Required files are missing, cannot continue.  Have all the pre-processing steps been run?
Loop ... 0
Init ... 
Accepted !!

This means the backend is actually starting but without a network, so it exits and that also causes a pod restart (as it should !!) Note that you still cannot log into this pod/deployment because it keeps restarting. If you want to actually step in and investigate, you will need a task that keeps running. So this would work in Dockerfile :

# bash images 
CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait"
# alpine : 
CMD exec /bin/sh -c "trap : TERM INT; sleep 9999999999d & wait"

But it means reconstructing the image, you could also create a debugging image like that just to 'look around' in the kubernetes pod ! I opt for building it into the solver process. We want to make backend to just print diagnostics but keep running at all costs, that way we can use it to login to the pod and also copy data to it as well. I implemented that with adding sleep(100000); just before the most common exit() functions.

config files

Most config files have the format :

---
version:
metadata:
spec:
[status:]

When I say file I mean separate configurations, because you can add more configurations together into one physical file. Status is not something you provide, but is kubernetes managed so it won't appear in your config files. But if you run :

kubectl edit deployments

You will see the status is present. etcd will maintain it.

version

Mostly v1 or apps/v1 but it can vary for any 'kind' of configuration.

See this page for an overview :

visit

It contains a list of values you should use, but also what each setting means. It states that Deployment should use apiversion : extensions/v1beta1

However, it already says there that it will be deprecated and also all examples use app/v1 so i will stick with the examples !

metadata

This is data about the direct parent, as you can see in example files you can have metadata sections repeat multiple times on multiple levels. Mostly this section contains a name and the labels. Labels is a generic dictionary that can be used for referencing etc.

spec

The specification of the direct parent. Same as metadata this can repeat for subitems throughout the configurations. The specification and status relate to eachother so that kubernetes always tries to have the status be the same as the spec, and take action if it is not.

linode

I use kubernetes on linode for now, here is how to setup.

First create a node, you pay for these per node so 1 is ok until we really get some more load.

You can login to the node with the 'Launch Console' button on the linode website : visit

However, it has an unknown root password. You will have to stop the node to set the root password.

• First do "Running" -> "Power Off".
• Second choose "Settings" -> "Reset root password"
• Restart the node again.

access to the private repository of docker hub

The steps that worked for me :

• on any machine, login to docker
• create a secret from the docker credentials
• run with the specified secret.

docker login
cat ~/.docker/config.json # see whats in it
cat ~/.docker/config.json | base64 -w 0 > secret.yaml

Now create the secret.yaml file so that it contains :

apiVersion: v1
kind: Secret
metadata:
    name: klopt-secret
data:
    .dockerconfigjson: [visit](the string from the base64 command)
type: kubernetes.io/dockerconfigjson

Kubernetes objects (such as a Secret) can be applied with this command:

kubectl apply -f secret.yaml
# examine it with
kubectl describe secrets
# or list secrets with
kubectl get secrets

To apply the secret to a pod/deployment we need this entry added to the container.

...
containers:
- name: doc
  ports:
  - containerPort: 80
  envFrom:
  - secretRef:
      name: klopt-secret

Note that this can only be done for the pod, and not directly to a deployment. That is because a deployment is just an application of a pod, and the pod get's pulled NOT a deployment. The deployment file for doc comes down to this :

apiVersion: v1
kind: Pod
metadata:
  name: docker-hub
spec:
  containers:
  - name: doc
    image: kees/klopt:doc
  imagePullSecrets:
  - name: klopt-secret
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: doc-deployment
  labels:
    app: doc
spec:
  replicas: 3
  selector:
    matchLabels:
      app: doc
  template:
    metadata:
      labels:
        app: doc
    spec:
      containers:
      - name: doc
        image: kees/klopt:doc
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: doc-service
  annotations:
    service.beta.kubernetes.io/linode-loadbalancer-throttle: "4"
  labels:
    app: doc
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: doc
  sessionAffinity: None

Making it a three-stage process : get the image for the pod, create a deployment for 3 of them and a service to load balance these 3.

yaml specs

See here for the various specs for the .yaml files.

visit

http / ssl

The linode GUI gives you options to add port 443 to your node balancer(s). When you go to "Dashboard" -> "NodeBalancers" and choose your nodes Configurations you will see port 80 by default.

Add another configuration will let you add 443 as well. Fill in records, and cut and paste klopt.org certificate AND KEY. into the forms. I choose port 443 for the node but it looks like any value would do (the http config has a random one).

Preparing the docker image for use with port 443. We should first test it locally, in any case expose PORT 443 :

FROM httpd:latest
MAINTAINER klopt.org
COPY ./build/html/ /usr/local/apache2/htdocs/
#CMD [“/usr/sbin/apache2ctl start”, “-D”, “FOREGROUND”]
EXPOSE 80
EXPOSE 443

certificates

klopt.org

This is how i did it according to the NANA Kubernetes video tutorial.

• create a secret based on the key/pub files
• add subdomain backends as deployments + service
• create an ingress using that secret by reference for these services

secret

Create a file like this called secret.yml :

apiVersion: v1
kind: Secret
  metadata:
    name: klopt-tls
    namespace: default
data:
  tls.crt: [visit](base64 encoded cert)
  tls.key: [visit](base64 encoded key)
type: kubernetes.io/tls

The base64 encoded sections can be generated like this :

ssh klopt.org -p 333
sudo su
locate www_sslcertificaten_nl_2019_2021
cat /etc/ssl/certs/www_sslcertificaten_nl_2019_2021.pem | base64 -w 0 > cert
# same for key, the values should be substituted in the yaml above

Then import it :

kubectl apply -f secret.yml 
kubectl describe secret klopt-tls
Name:         klopt-tls
Namespace:    default
Labels:       [visit](none)
Annotations:  [visit](none)

Type:  kubernetes.io/tls

Data
====
tls.crt:  2285 bytes
tls.key:  1704 bytes

Use the name "klopt-tls" for references later.

cert-manager

This is the tool to manage certificates, both letsencrypt and regular ones. A short glance over the concepts :

• Issuer : is what it sounds like, we only need that when using letsencrypt because we already have an issued crt for klopt.org
• Certificate : that
• CertificateRequest : a request for a Certificate from an Issuer
• ACME Orders :

Explanation :

visit

troubleshooting

logging

minikube does not start

On ubuntu i got this message after starting up minikube.

X minikube is unable to connect to the VM: dial tcp 192.168.99.102:22: i/o timeo

    This is likely due to one of two reasons:

    - VPN or firewall interference
    - virtualbox network configuration issue

    Suggested workarounds:

    - Disable your local VPN or firewall software
    - Configure your local VPN or firewall to allow access to 192.168.99.102
    - Restart or reinstall virtualbox
    - Use an alternative --vm-driver
    - Use --force to override this connectivity check

X Exiting due to GUEST_PROVISION: Failed to validate network: dial tcp 192.168.9

Note that deleting and rebuilding the VM on virtualbox works but ONLY LIKE THIS:

minikube delete
minikube start

If you just remove it from the Virtualbox gui it will end up in the same error again.!!

permission denied

After trying :

docker login

This message appears.

Got permission denied while trying to connect to the Docker daemon socket

This only happens when you do this as non-root on a linode kubernetes node. As root it works !. For now i just ran as root, but here is a workaround that is supposed to work (untried)

adduser klopt
su - klopt
sudo groupadd docker
sudo usermod -aG docker klopt

See here for more info : visit

kubectl Authentication required

These errors have to do with forgetting the KUBECONFIG env var.

error: no matches for kind "Secret" in version "v1"

Do NOT forget to set the environment variable KUBECONFIG

You can also just copy the config file to ~/.kube/config then the ENV variable is not needed.

export KUBECONFIG=klopt-kubeconfig.yaml

ClusterIP change

The Service "doc-service" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when [type]{.title-ref} is 'ClusterIP'

Caused by the fact that the deploy.yml was changed and re-applied with different type (LoadBalancer -> ClusterIP). Use force to run anyways :

kubectl apply -f test-service.yaml --force.

If this does not help, you can edit the configurations with kubectl like this :

kubectl edit srv doc-service

You will get the complete config as yml, change the type from loadBalancer to ClusterIP, and the error will go away. You might have to re-apply the file after the change.

Switch between clusters

When using more than 1 cluster, like testing with minikube you may want to switch back and forth. To do this you need to switch between contexts. Inside the ~/.kube/config file all known configurations are printed, to list them do :

kubectl config get-contexts
CURRENT   NAME           CLUSTER    AUTHINFO         NAMESPACE
*         lke15110-ctx   lke15110   lke15110-admin   default
          minikube       minikube   minikube         default

The * shows which contexts is currently used, shown in ~/.kube/config as current-context. To switch :

kubectl config use-context minikube
get svc # only minikube services will be shown

Installing a test cluster

Installation of a demo cluster can be done by hand, maybe there will be a work instruction. But here is a short description of a working installation using vagrant and ansible.

Vagrantfile

The example I used was : visit

The image there was ubuntu, so here is my vagrantfile :

# -*- mode: ruby -*-
# vi: set ft=ruby :

IMAGE_NAME = "generic/ubuntu1604"

# master has to be provisioned first !!
ENV['VAGRANT_NO_PARALLEL'] = 'yes'
# number of nodes 
N=2

# you're doing.
Vagrant.configure("2") do |config|
  config.vm.box = IMAGE_NAME

  config.vm.provider "libvirt" do |vb|
    vb.memory = "2048"
  end

  config.vm.define "master" do |master|
    master.vm.network "private_network", ip: "192.168.55.10"
    master.vm.hostname = "master"

    master.vm.provision "ansible" do |ansible|
      ansible.playbook = "master.yml"
      ansible.compatibility_mode = "2.0"
      ansible.extra_vars = {
        node_ip: "192.168.55.10",
      }
    end
  end

  (1..N).each do |i|
    config.vm.define "node-#{i}" do |node|
      node.vm.network "private_network", ip: "192.168.55.#{i + 19}"
      node.vm.hostname = "node-#{i}"
      node.vm.provision "ansible" do |ansible|
        ansible.playbook = "node.yml"
        ansible.compatibility_mode = "2.0"
        ansible.extra_vars = {
          node_ip: "192.168.55.#{i + 19}",
        }
      end
    end
  end
end

There are two flavours, 1 master and N nodes that should be provisioned sequentially because master generates a file that is used by the nodes to join the cluster !!.

You can run this by simple :

vagrant up 

However you will need some more files, for instance the ansible files :

master.yml

This will need some explanation, this will be done later.

---
- hosts: all
  remote_user: vagrant
  become: true
  tasks:
  - name: Install packages that allow apt to be used over HTTPS
    apt:
      name: "{{ packages }}" 
      state: present
      update_cache: yes
    vars:
      packages:
      - apt-transport-https
      - ca-certificates
      - curl
      - gnupg-agent
      - software-properties-common

  - name: Add an apt signing key for Docker
    apt_key:
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present

  - name: Add apt repository for stable version
    apt_repository:
      repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
      state: present

  # cgroupdriver must be systemd, see 
  # https://kubernetes.io/docs/setup/production-environment/container-runtimes/
  - name: Create /etc/docker
    file:
      path: /etc/docker
      state: directory
      mode: '0755'

  - name: Fill the daemon file
    copy:
      dest: /etc/docker/daemon.json
      src: daemon.json

  - name: Install docker and its dependecies
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - containerd.io=1.2.13-2
      - docker-ce=5:19.03.11~3-0~ubuntu-xenial
      - docker-ce-cli=5:19.03.11~3-0~ubuntu-xenial
    notify:
      - docker status

  - name: Add vagrant user to docker group
    user:
      name: vagrant
      group: docker

  - name: Remove swapfile from /etc/fstab
    mount:
      name: "{{ item }}"
      fstype: swap
      state: absent
    with_items:
      - swap
      - none

 - name: Disable swap
    command: swapoff -a
    when: ansible_swaptotal_mb > 0

  - name: Add an apt signing key for Kubernetes
    apt_key:
      url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
      state: present

  - name: Adding apt repository for Kubernetes
    apt_repository:
      repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
      state: present
      filename: kubernetes.list

  - name: Install Kubernetes binaries
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
        - kubelet
        - kubeadm
        - kubectl

  - name: Configure node ip
    lineinfile:
      path: /etc/default/kubelet
      line: KUBELET_EXTRA_ARGS=--node-ip={{ node_ip }}
      create: yes

  - name: Restart kubelet
    service:
      name: kubelet
      daemon_reload: yes
      state: restarted

  - name: Initialize the Kubernetes cluster using kubeadm
    command: kubeadm init --apiserver-advertise-address="192.168.55.10" --apiserver-cert-extra-sans="192.168.55.10"  --node-name master --pod-network-cidr=192.168.0.0/16
    ignore_errors: yes

  - name: Create a directory if it does not exist
    file:
      path: /home/vagrant/.kube
      state: directory
      mode: '0755'


  - name: Setup kubeconfig for vagrant user
    command: "{{ item }}"
    with_items:
     - cp /etc/kubernetes/admin.conf /home/vagrant/.kube/config
     - chown vagrant:vagrant /home/vagrant/.kube/config

  - name: Install flannel pod network
    become: false
    command: kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    ignore_errors: yes

  - name: Generate join command
    command: kubeadm token create --print-join-command
    register: join_command
    ignore_errors: yes

  - name: Copy join command to local file
    become: false
    local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="./join-command"

  handlers:
    - name: docker status
      service: name=docker state=started

In short :

• Install needed packages
• Add docker repository + key
• Put daemon.json file in docker directory to make cgroupdriver=systemd
• Install some older versions of docker, since kubeadm only works with those.
• Remove swap, kubernetes nodes don't run with swap.
• Configure ip and restart.
• Initialize the cluster and install config
• Install the network (Flannel)
• Generate a file with the join commands the nodes will need
• Restart docker.

access to the cluster

Almost all examples tell you to do a simple command like kubectl get nodes. But they all do it from the master node. To get things started log in to the master node and look inside ~/.kube/config. You will see a layout similar to, with the keys cut out :

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJU.......
    server: https://192.168.55.10:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: LS0tLS1CRUdJT.......
        client-key-data: LS0tLS1CRUdJT......

To use these values from outside the cluster copy these to the ~/.kube/config on the host system. It should be in this format, so you can't just concatenate the files.

apiVersion: v1
clusters:
- cluster: ....
  name: kubernetes
- cluster: ....
  name: lke15110

contexts:
- context: .. etc

Here is a trick to do this in newer versions of kubectl.

cp ~/.kube/config firstfile
cp [visit](newconfigname) secondfile
# you can export, but we use this oneliner, note that this overwrites your config !!!
KUBECONFIG=firstfile:secondfile kubectl config view --flatten > ~/.kube/config\

Actually, since i kept the linode file in ~/Downloads, this is the actual command when you name you config file myconf.yaml :

KUBECONFIG=./myconf.yaml:~/Downloads/klopt-kubeconfig.yaml kubectl config view --flatten > ~/.kube/config
chmod 600 ~/.kube/config

The chmod is needed to prevent warnings like this :

WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/kees/.kube/config

Put this in some form in the ansible installer (copy from join-command)

Possibly you will have to switch context from linode to dev, see what the current context is with :

kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
          kubernetes-admin@kubernetes   kubernetes   kubernetes-admin   
*         lke15110-ctx                  lke15110     lke15110-admin     default

If needed switch with

kubectl config use-context kubernetes

Now try it again :

kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    [visit](none)        443/TCP   79m
kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   81m   v1.20.2
node-1   Ready    [visit](none)                 76m   v1.20.2
node-2   Ready    [visit](none)                 71m   v1.20.2

In the next chapters we try to add prometheus + grafana to this cluster.

prometheus

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

NAME: prometheus
LAST DEPLOYED: Sun Feb  7 12:38:22 2021
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

Now see what pods have been created :

kubectl get pods -nmonitoring
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          10m
prometheus-grafana-858b5899bf-qgfm5                      2/2     Running   0          10m
prometheus-kube-prometheus-operator-fd64ffb66-lb9cs      1/1     Running   0          10m
prometheus-kube-state-metrics-c65b87574-lmrbj            1/1     Running   0          10m
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   1          10m
prometheus-prometheus-node-exporter-qcmh9                1/1     Running   0          10m

Now port-forward the prometheus pod to 9090 :

kubectl port-forward -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090

And the grafana pod to 3000:

kubectl port-forward -n monitoring prometheus-grafana-858b5899bf-qgfm5 3000

Now view grafana on visit and login with admin:prom-operator

get password

It is not likely this default password will change but here is a short guide on how to obtain it :

kubectl get secret -n monitoring prometheus-grafana -o yaml

Find the admin-password and admin-usr, these are the base64 encoded strings, you can decode with base64 -d.

problems encountered

There are a number of problems solved inside this implementation.

• The installation use the wrong (previous) join-command file
• Docker was run as cgroupd and that should be systemd according to warning
• Changed the network to Flannel since the one given was not reachable.
• Altered docker versions, kubernetes complains about supported versions.

unable to build kubernetes objects

When installing prometheus-community/kube-prometheus-stack, some older configuration seems to be in the way.

The command that fails is :

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec, ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector" in com.coreos.monitoring.v1.Alertmanager.spec]

On the github page it says to cleanup the crd's that are created :

visit

And it works but be aware :

Don't do this on production !!

This could break more than you want !

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com

ingress

When installing the ingress controller a lot of information is printed. It looks like useful information so here is a print of it :

WARNING: This chart is deprecated
NAME: nginx-ingress
LAST DEPLOYED: Fri Feb 26 10:34:49 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
*******************************************************************************************************
* DEPRECATED, please use https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx *
*******************************************************************************************************


The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: [visit](base64 encoded cert)
    tls.key: [visit](base64 encoded key)
  type: kubernetes.io/tls

The deprecation warning did not cause the install to fail, so just use the given url instead if you install a new controller.