Upgrade Kubernetes Cluster to v1.16 using Kubeadm

Kubernetes has released latest version 1.16 on Oct 22, 2019. Refer release notes v1.16

In this article we will see how safely upgrade step by step using Kubeadm

 

Before you begin

  1. Backup Kubernetes cluster using Heptio Velero explained in my previous blog
  2. Backup app-level state stored in database, kubeadm-upgrade doesn’t touch workloads

Upgrade Kubernetes Control Plane node (Master)

Common steps followed for upgrade like drain, plan and apply

  1. Install latest kubeadm v1.16
# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
apt-mark hold kubeadm

 

2. Verify latest version

kubeadm version

3. Drain the control plane node

kubectl drain $MASTER --ignore-daemonsets

4. Plan for upgrading control plane node

sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.15.2
[upgrade/versions] kubeadm version: v1.16.0

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     1 x v1.15.2   v1.16.0

Upgrade to the latest version in the v1.16 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.15.2   v1.16.0
Controller Manager   v1.15.2   v1.16.0
Scheduler            v1.15.2   v1.16.0
Kube Proxy           v1.15.2   v1.16.0
CoreDNS              1.3.1     1.6.2
Etcd                 3.3.10    3.3.15

Note: It will upgrade appropriate etcd version

5. Apply upgrade

sudo kubeadm upgrade apply v1.16.0
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.16.0". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

6. Start Scheduler on control plane node

kubectl uncordon $MASTER

Upgrade Kubelet and Kubectl

# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
apt-mark hold kubelet kubectl

Restart kubelet

sudo systemctl restart kubelet

Upgrade Worker Nodes

The upgrade procedure on worker nodes should be executed one node at a time or few nodes at a time, without compromising the minimum required capacity for running your workloads

  • Upgrade Kubeadm as we did on Master Node
# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
apt-mark hold kubeadm
  • Drain node for maintenance by marking it unschedulable and evicting the workloads
kubectl drain $NODE1 --ignore-daemonsets

You should see output similar to this:

node1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-dj7d7, kube-system/weave-net-z65qx
node1 drained
  •   Upgrade Kubeadm on worker node

sudo kubeadm upgrade node

Upgrade the kubelet and kubectl on all worker nodes:

# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
apt-mark hold kubelet kubectl
  • Restart kubelet service
sudo systemctl restart kubelet
  • Uncordon the node
kubectl uncordon $NODE1

Repeat the same process on all worker nodes one by one and verify the status of cluster

kubectl get nodes

The STATUS column should show Ready for all your nodes, and the version number should be updated

 

Recovering from failure state

If kubeadm upgrade fails and does not roll back, for example because of an unexpected shutdown during execution, you can run kubeadm upgrade again. This command is idempotent and eventually makes sure that the actual state is the desired state you declare.

To recover from a bad state, you can also run kubeadm upgrade --force without changing the version that your cluster is running

How it works

kubeadm upgrade apply does the following:

  • Checks that your cluster is in an upgradeable state:
    • The API server is reachable
    • All nodes are in the Ready state
    • The control plane is healthy
  • Enforces the version skew policies.
  • Makes sure the control plane images are available or available to pull to the machine.
  • Upgrades the control plane components or rollbacks if any of them fails to come up.
  • Applies the new kube-dns and kube-proxy manifests and makes sure that all necessary RBAC rules are created.
  • Creates new certificate and key files of the API server and backs up old files if they’re about to expire in 180 days.

kubeadm upgrade node does the following on additional control plane nodes:

  • Fetches the kubeadm ClusterConfiguration from the cluster.
  • Optionally backups the kube-apiserver certificate.
  • Upgrades the static Pod manifests for the control plane components.
  • Upgrades the kubelet configuration for this node.

kubeadm upgrade node does the following on worker nodes:

  • Fetches the kubeadm ClusterConfiguration from the cluster.
  • Upgrades the kubelet configuration for this node