Upgrade Kubernetes on RHEL 8

Upgrading Kubernetes on Red Hat 8 can be complex and depends heavily on how your cluster was initially deployed and configured. Here’s a general guide outlining the steps and considerations, broken down by common deployment methods:

Before You Start (Critical Planning & Preparation):

1. Read the Official Kubernetes Upgrade Documentation: This is absolutely essential! The official documentation is the most reliable source and contains critical information specific to the versions you’re upgrading from and to.

   • https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ (If you used kubeadm)
   • https://kubernetes.io/docs/tasks/administer-cluster/ (General Administration Tasks)

2. Determine Your Current and Target Versions: Use kubectl version on your master node to find your current Kubernetes version. Decide which version you want to upgrade to. Upgrade one minor version at a time! For example, if you’re on 1.23, upgrade to 1.24 before attempting 1.25. Skipping versions is generally unsupported and can lead to significant issues.

3. Check Compatibility: Verify that your chosen Kubernetes version is compatible with your container runtime (e.g., Docker, containerd, CRI-O) and CNI plugin (e.g., Calico, Flannel, Cilium). Refer to the documentation for your runtime and CNI plugin.

4. Backup Everything! This is paramount. Back up your etcd data (the Kubernetes data store), your Kubernetes configuration files, and any persistent volumes/data. Consider a full system backup of your master nodes.

5. Drain Nodes Strategically: Before upgrading a node, you must drain it. This safely evicts all Pods from the node, rescheduling them elsewhere in the cluster. This prevents downtime for your applications.

6. Monitor Your Cluster: Use kubectl get nodes, kubectl get pods --all-namespaces, and other monitoring tools to observe the health of your cluster throughout the upgrade process.

7. Consider a Staging Environment: Ideally, test the upgrade process in a non-production environment that mirrors your production setup as closely as possible. This will help you identify potential problems before they impact your users.

8. Red Hat Specifics: Red Hat sometimes has its own Kubernetes distributions (like OpenShift). If you are using OpenShift, the upgrade process is drastically different. Refer to the OpenShift documentation for upgrading: https://docs.openshift.com/container-platform/latest/updating/understanding-updating-versions.html The following instructions are not for OpenShift.

Upgrade Steps (General, Assuming kubeadm – Most Common for DIY Kubernetes):

These instructions assume you’re using kubeadm to manage your Kubernetes cluster. If you used a different method (e.g., kops, kubespray, a cloud provider’s managed Kubernetes service), the steps will vary significantly.

1. Upgrade the Control Plane (Master Nodes):

• On the First Master Node:

  # SSH into the first master node.
  
  # 1. Drain the node:  This is *crucial*.
  kubectl drain <node-name> --ignore-daemonsets --force  # Replace <node-name>
  # Consider adding --delete-local-data if you have pods using local volumes that you're okay with losing.
  
  # 2. Update kubeadm, kubelet, and kubectl packages:
  yum update kubeadm kubelet kubectl --disableexcludes=kubernetes
  systemctl restart kubelet
  
  # 3. Verify kubeadm upgrade plan:
  kubeadm upgrade plan
  
  # 4. Upgrade the control plane:
  kubeadm upgrade apply v1.XX.YY # Replace v1.XX.YY with the target version (e.g., v1.24.0)
  
  # 5. Update kubelet configuration:
  kubeadm kubelet config migrate
  
  # 6. Uncordon the node:
  kubectl uncordon <node-name>
  

• On Remaining Master Nodes (If you have a HA Control Plane):

  # SSH into each remaining master node.
  
  # 1. Drain the node (same as above).
  kubectl drain <node-name> --ignore-daemonsets --force
  
  # 2. Update kubeadm, kubelet, and kubectl (same as above).
  yum update kubeadm kubelet kubectl --disableexcludes=kubernetes
  systemctl restart kubelet
  
  # 3. Join the control plane:
  kubeadm upgrade node
  
  # 4. Update kubelet configuration:
  kubeadm kubelet config migrate
  
  # 5. Uncordon the node.
  kubectl uncordon <node-name>
  

2. Upgrade Worker Nodes:

• On Each Worker Node:

  # SSH into each worker node.
  
  # 1. Drain the node.
  kubectl drain <node-name> --ignore-daemonsets --force
  
  # 2. Update kubeadm, kubelet, and kubectl.
  yum update kubeadm kubelet kubectl --disableexcludes=kubernetes
  systemctl restart kubelet
  
  # 3. Join the cluster:
  kubeadm upgrade node
  
  # 4. Update kubelet configuration:
  kubeadm kubelet config migrate
  
  # 5. Uncordon the node.
  kubectl uncordon <node-name>
  

Important Considerations and Troubleshooting:

• --ignore-daemonsets: Use this with kubectl drain because DaemonSets are designed to run on every node. You generally don’t want to drain them.
• --force: Use with caution! Only use --force if kubectl drain is stuck and you’ve confirmed it’s safe to proceed. Understand the implications of forcibly evicting Pods.
• --delete-local-data: Use this only if you’re okay with deleting data stored in local volumes on the node being drained. This is typically not what you want for stateful applications.
• CNI Plugin Compatibility: Ensure your CNI plugin (e.g., Calico, Flannel) is compatible with the target Kubernetes version. You may need to upgrade the CNI plugin separately. Refer to the CNI plugin’s documentation for upgrade instructions.
• Container Runtime Compatibility: Verify your container runtime (e.g., Docker, containerd, CRI-O) is compatible with the target Kubernetes version. You may need to upgrade the container runtime separately.
• kubeadm upgrade apply vs. kubeadm upgrade node:
  • kubeadm upgrade apply is used only on the first master node to upgrade the control plane components.
  • kubeadm upgrade node is used on the remaining master nodes (in a HA setup) and on all worker nodes.
• Package Holding (--disableexcludes=kubernetes): The --disableexcludes=kubernetes option is used with yum update to temporarily override any package exclusion rules that might be in place for Kubernetes packages. This is often necessary to ensure that the latest versions of kubeadm, kubelet, and kubectl are installed during the upgrade process. Review your yum configuration to understand why these exclusions might be in place in the first place.
• Etcd Backup: Know how to restore your etcd backup if something goes wrong. The etcd documentation has the best information on this. Practice a restore in your staging environment!
• API Deprecations: Pay close attention to API deprecations in the Kubernetes release notes. Deprecated APIs might be removed in future versions, so you’ll need to update your manifests and tooling to use the new APIs. This is a common source of upgrade problems.
• Monitoring: Keep a close eye on your cluster’s health during and after the upgrade. Monitor CPU, memory, disk usage, and network traffic. Check your application logs for errors.

Example Error and Solution:

• Error: kubeadm upgrade apply fails with an error like “etcd cluster is unhealthy.”
  • Solution: Investigate the etcd cluster health. Use etcdctl (if you’re using etcd directly) or the tools provided by your etcd operator (if you’re using an operator). Ensure that all etcd members are healthy and can communicate with each other. You might need to restart etcd members or adjust their configuration.

Specific to Red Hat 8:

• SELinux: Ensure that SELinux is configured correctly and is not blocking any Kubernetes components. Check the audit logs for SELinux denials. You might need to create custom SELinux policies to allow Kubernetes to function properly. (Ideally, find the correct pre-built policies).
• Firewall: Verify that the firewall is configured to allow communication between Kubernetes components. The default Kubernetes ports (e.g., 6443, 2379, 2380, 10250) must be open. Use firewall-cmd to manage the firewall rules.
• Package Manager (Yum/DNF): Red Hat 8 uses dnf as its package manager, but yum commands are generally aliased to dnf for backwards compatibility. The commands above use yum. If you have issues, try dnf directly.
• Red Hat Subscription: Ensure that your Red Hat subscription is active and that you have access to the necessary repositories to download the Kubernetes packages.
• CRI-O: CRI-O is a popular container runtime on Red Hat. Make sure it is properly configured and compatible with the Kubernetes version you are upgrading to. You may need to update the CRI-O configuration.

In summary: Upgrading Kubernetes is a significant undertaking. Thorough planning, careful execution, and diligent monitoring are essential to a successful upgrade. Always consult the official Kubernetes documentation and the documentation for your specific components (CNI, container runtime, etc.) before starting the upgrade process. Test in a staging environment first! Good luck!