Tencent Cloud TKE allows you to upgrade the Kubernetes version. You can use this feature to upgrade a running Kubernetes cluster. The upgrade process includes pre-upgrade checking, upgrading the Master, and upgrading the node.
Upgrade Notice
The upgrading action is irreversible. Perform this operation with caution.
Before upgrading the cluster, check whether the cluster is healthy. If the cluster is abnormal, you can fix it yourself or consult online.
Upgrade sequence: when upgrading a cluster, you must first upgrade the Master, and then upgrade the node as quickly as possible. During the upgrade process, we recommend that you do not perform any operations in the cluster.
Only upgrading to the next Kubernetes version offered by TKE is supported. Upgrading across multiple versions (such as upgrading from 1.8 to 1.12, skipping 1.10) is not supported. You can upgrade to the next version only if the Master and node versions are consistent.
Incompatibility of CSI-CFS add-on: as for the CSI add-ons: COS CSI and CFS CSI, the CSI add-on versions adapted to different Kubernetes versions have the following differences. Therefore, we recommend that you reinstall the CSI add-on in add-on management page when you upgrade the cluster to TKE 1.14 and later version. The rebuilding of the add-on does not affect COS and CFS storage already in use.
The version of the CSI add-on adapted for Kubernetes 1.10 and Kubernetes 1.12 is 0.3.
The CSI add-on version for Kubernetes 1.14 and later is 1.0.
HPA Invalidation Issue: In Kubernetes versions prior to 1.18, the apiversion of the deployment object referenced in HPA might be extensions/v1beta1. However, in Kubernetes 1.18 and later, the apiversion of deployment is only apps/v1, which may cause HPA to become invalid after upgrading to Kubernetes 1.18.
If you are using HPA, it is recommended to execute the following command before upgrading to switch the apiVersion in the HPA object to apps/v1.
kubectl patch hpa test -p '{"spec":{"scaleTargetRef":{"apiVersion":"apps/v1"}}}'
The failure of Helm applications: each application, including those installed through the application marketplace or third parties, supports different versions of Kubernetes. Before upgrading a cluster, you are advised to view the list of applications installed in the cluster and check the range of cluster versions supported by the applications. Some applications are adaptable to higher versions of Kubernetes, and you may need to upgrade them. Some applications may not be adaptable to higher versions of Kubernetes, and in which case, upgrade the cluster with caution.
Nginx-ingress version issue: extensions/v1beta1 and networking.k8s.io/v1beta1 ingress APIs are no longer provided in v1.22. For more information, see here. If the Nginx-ingress version in your cluster is too early, upgrade the Nginx-ingress add-on on the add-on management page after you upgrade Kubernetes to v1.22 or later.
Currently, Master version upgrades of managed clusters and self-deployed clusters are supported, and the upgrade takes 5-10 minutes, during which you are unable to operate your cluster.
Notes for Master major version and minor version upgrade
Currently, the upgrade of Master supports the major version upgrade (for example, from 1.14 to 1.16), and the minor version upgrade (for example, from 1.14.3 to 1.14.6, or from v1.18.4-tke.5 to v1.18.4-tke.6). We strongly recommend that you check the corresponding feature release records before upgrading:
When upgrading major versions (e.g., from 1.12 to 1.14), if you have set custom parameters, you need to reset the custom parameters for the new version. The original parameters will not be retained. For more information, see Custom Kubernetes Component Launch Parameters.
For minor version upgrades, the custom parameters will be retained, and you do not need to reconfigure them.
Supports and Limits
Before upgrading, please read the Upgrade Notice carefully.
For TKE clusters of the v1.7.8, the network mode is bridge. Upgrading the cluster does not automatically switch the network mode to cni.
Upgrading the cluster does not switch kube-dns to core-dns.
When you upgrade a cluster to v1.10 and 1.12, some features configured when the cluster is created, such as support for ipvs, will become unavailable.
After the upgrade of an existing cluster, if the master version is 1.10 or later, and the node version is V1.8 or earlier, the PVC feature will be unavailable.
After upgrading the master version, we recommend that you upgrade the node version as soon as possible.
Technical principles of Master upgrade
The master node upgrade involves 3 steps: pre-dependent component upgrade, Master node component upgrade, and post-dependent component upgrade.
Pre-dependent component upgrade: the pre-dependent components, such as monitoring components, will be upgraded to prevent component exceptions due to compatibility problems.
Master node component upgrade: The corresponding components of all Masters will be upgraded in sequence. After all Masters' components are upgraded, the next component will be upgraded. TKE will first upgrade kube-apiserver, followed by kube-controller-manager and kube-scheduler, and finally kubelet. The specific steps are as follows:
Regenerate the yaml file corresponding to the static Pod of the kube-apiserver component.
Check whether the current kube-apiserver Pod is healthy and whether the kubernetes version is normal.
Similarly, upgrade kube-controller-manager and kube-scheduler in sequence.
Upgrade kubelet and check whether the master node is ready.
Post-dependent component upgrade:
Upgrade the post-dependent components as needed, such as kube-proxy (and change its rolling update strategy to on delete), and cluster-autoscaler components.
Perform some compatibility operations related to post-dependent components to prevent component exceptions due to compatibility problems.
Master upgrade
1. Log in to the TKE console and click Cluster in the left sidebar.
2. On the Cluster Management page, select the ID of the desired cluster, and enter the cluster details page.
3. On the cluster details page, click Basic Information on the left.
4. In the "Cluster Information" page, click Upgrade on the right side of the Master version, as shown below:
5. In the pop-up window, click Submit and wait until the upgrade is complete.
6. You can view the upgrade progress in the cluster status on the cluster management page, as well as in the upgrade progress pop-up window. This includes the current upgrade progress, Master node upgrade progress (managed clusters do not display specific Master node lists), and upgrade duration. As shown in the image below:
7. In this example, the Kubernetes version of the Master is 1.10.5 before the upgrade, and 1.12.4 after the upgrade, as shown below:
Upgrading the node Kubernetes version
After the cluster Master Kubernetes version is upgraded, the cluster list page will display that the cluster nodes have available upgrades, as shown below:
Supports and Limits
Before upgrading, please read the Upgrade Notice carefully.
Reinstall and rolling upgrade: reinstall the node to upgrade the node version. This method only supports major version upgrades, such as upgrades from version 1.10 to version 1.12.
In-place rolling upgrade: upgrade directly without re-installation. This only replaces components such as Kubelet and kube-proxy. Currently, this method supports both major and minor upgrades, such as from version 1.10 to version 1.12, or from version 1.14.3 to version 1.14.8.
Reinstallation Rolling Upgrade Execution Principle
The node upgrade based on reinstallation uses a rolling upgrade method. Only one node will be upgraded at a time, and the next node will be upgraded only after the current node is successfully upgraded, as shown below:
Pre-upgrade check: check the Pods on a node before draining. The specific items for pre-upgrade checking are as follows:
Calculate the number of pods of all workloads in this node. If the Pod number of any workload changes to 0 after the node is drained, then the check fails, and the upgrade cannot be performed.
The following system control plane workloads will be ignored:
l7-lb-controller
cbs-provisioner
hpa-metrics-server
service-controller
cluster-autoscaler
Evicting Pods: first mark the node as unschedulable. Then, evict or delete all Pods from the node.
Removing nodes: remove the node from the cluster. This step only performs basic cleanup, and will not delete the node instance of the node in the cluster. Therefore, the node’s attributes such as label and taint are retained.
Reinstalling nodes: reinstall the node’s operating system and reinstall the new version of kubelet.
Post-upgrade check: check whether the node is ready and schedulable, and check whether the current proportion of unavailable pods exceeds the max limit.
Reinstall and rolling upgrade (node Kubernetes version)
1. Log in to the TKE console and click Cluster in the left sidebar.
2. On the Cluster Management page, select the ID of the cluster for node Kubernetes version upgrade to enter the cluster details page.
3. In the cluster information module, click Upgrade to the right of the Node Kubernetes version, as shown in the figure below:
4. In the "Notes on Upgrade" step, select the upgrade method as Reinstall and rolling upgrade and carefully read the upgrade notice. Check I have read and agreed to the above technical terms and click Next as shown below:
Note
This upgrade method will reinstall the operating system, and the original data will be cleared. Back up the data in advance.
5. In the "Select a node" step, choose the nodes to be upgraded in this batch and click Next.
6. In the "Upgrade Settings" step, fill in the node information as needed and click Next.
7. In the "Confirmation" step, verify the information and click Finish to start the upgrade.
8. Monitor the node upgrade progress until all nodes are successfully upgraded.
In-place Rolling Upgrade Execution Principle
Nodes are upgraded in place using a rolling upgrade method, where only one node is upgraded at a time, and the next node is upgraded only after the current node is successfully upgraded. In-place upgrades currently support both major version upgrades and minor version upgrades within the same major version. As shown in the following diagram:
The steps are described as follows:
Component update: replace and restart the kubelet and kube-proxy components on the node.
Post-upgrade check: check whether the node is ready and check whether the proportion of currently unavailable pods exceeds the max limit.
In-place rolling upgrade
1. Log in to the TKE console and click Cluster in the left sidebar.
2. On the Cluster Management page, select the ID of the desired cluster and enter the cluster details page.
3. In the cluster's "Basic Information" page, click Upgrade on the right side of the Node Kubernetes version.
4. In the Notes on Upgrade, select the upgrade method as "In-place rolling upgrade" and carefully read the upgrade notice. Check I have read and agree to the above technical terms and click Next. As shown below:
5. In the "Select a node Selection" step, choose the nodes to be upgraded in this batch and click Next.
6. In the "Confirmation" step, verify the information and click Finish to start the upgrade.
7. Monitor the node upgrade progress until all nodes are successfully upgraded.
