经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 数据库/运维 » Kubernetes » 查看文章
Centos7 升级 Kubernetes(k8s) 集群
来源：cnblogs　　作者：人生的哲理　　时间：2023/7/12 10:57:55　　对本文有异议

目录

一.系统环境

本文主要基于Kubernetes1.21.0和Linux操作系统CentOS7.4。

服务器版本 docker软件版本 Kubernetes(k8s)集群版本 Kubernetes(k8s)集群升级后的版本 CPU架构
CentOS Linux release 7.4.1708 (Core) Docker version 20.10.12 v1.21.0 v1.21.9 x86_64

Kubernetes集群架构：k8scloude1作为master节点，k8scloude2，k8scloude3作为worker节点。

服务器 操作系统版本 CPU架构 进程 功能描述
k8scloude1/192.168.110.130 CentOS Linux release 7.4.1708 (Core) x86_64 docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico k8s master节点
k8scloude2/192.168.110.129 CentOS Linux release 7.4.1708 (Core) x86_64 docker，kubelet，kube-proxy，calico k8s worker节点
k8scloude3/192.168.110.128 CentOS Linux release 7.4.1708 (Core) x86_64 docker，kubelet，kube-proxy，calico k8s worker节点

二.前言

在容器化应用的管理中，Kubernetes 是一个广泛使用的开源平台。为了保持集群的稳定性和安全性，定期升级 Kubernetes 是非常重要的。本篇博客将介绍如何在 CentOS 7.4 上升级 Kubernetes(k8s) 集群。

在 CentOS 7.4 上升级 Kubernetes(k8s) 集群的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Centos7 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/16686769.html。

三.Kubernetes(k8s) 集群升级简介

Kubernetes 使用 kubeadm 工具来管理集群组件的升级。在集群节点层面，升级 Kubernetes(k8s)集群的过程可以分为以下几个步骤：

  1. 检查当前环境和配置是否满足升级要求。
  2. 升级master主节点（如果是多master，则master轮着升级）。
  3. 升级worker工作节点。
  4. 升级网络插件。

在软件层面，升级 Kubernetes(k8s)集群的过程可以分为以下几个步骤：

  1. 升级kubeadm
  2. 节点执行drain操作
  3. 升级各个组件
  4. 取消drain操作
  5. 升级kubelet和kubectl

注意：Kubernetes(k8s)集群升级的时候是不能跨版本升级的，比如：Kubernetes(k8s)集群 1.19.x可以升级为1.20.y，但是

Kubernetes(k8s)集群 1.19.x不能直接升级为 1.21.y，你只能从一个次版本升级到下一个次版本，或者在次版本相同时升级补丁版本。 也

就是说，升级时不可以跳过次版本。 例如，你只能从 1.y 升级到 1.y+1，而不能从 1.y 升级到 1.y+2。

四.升级master主节点

4.1 升级kubeadm

Kubernetes(k8s)集群版本是v1.21.0。

  1. [root@k8scloude1 ~]# kubectl get node
  2. NAME         STATUS   ROLES                  AGE   VERSION
  3. k8scloude1   Ready    control-plane,master   83d   v1.21.0
  4. k8scloude2   Ready    <none>                 83d   v1.21.0
  5. k8scloude3   Ready    <none>                 83d   v1.21.0

查看可用的kubeadm版本。

  1. [root@k8scloude1 ~]# yum list --showduplicates kubeadm --disableexcludes=kubernetes
  2. 已加载插件：fastestmirror
  3. Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
  4. Determining fastest mirrors
  5. 已安装的软件包
  6. kubeadm.x86_64                                                                                        1.21.0-0                                                                                         @kubernetes
  7. 可安装的软件包
  8. kubeadm.x86_64                                                                                        1.6.0-0                                                                                          kubernetes       
  9. ......
  10. kubeadm.x86_64                                                                                        1.21.0-0                                                                                         kubernetes 
  11. kubeadm.x86_64                                                                                        1.21.1-0                                                                                         kubernetes 
  12. kubeadm.x86_64                                                                                        1.21.2-0                                                                                         kubernetes 
  13. kubeadm.x86_64                                                                                        1.21.3-0                                                                                         kubernetes 
  14. kubeadm.x86_64                                                                                        1.21.4-0                                                                                         kubernetes 
  15. kubeadm.x86_64                                                                                        1.21.5-0                                                                                         kubernetes 
  16. kubeadm.x86_64                                                                                        1.21.6-0                                                                                         kubernetes 
  17. kubeadm.x86_64                                                                                        1.21.7-0                                                                                         kubernetes 
  18. ......
  19. kubeadm.x86_64                                                                                        1.23.3-0                                                                                         kubernetes

升级kubeadm到1.21.9-0版本。

  1. [root@k8scloude1 ~]# yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes
  2. 已加载插件：fastestmirror
  3. base                                                                                                                                                                                       | 3.6 kB  00:00:00     
  4. docker-ce-stable                                                                                                                                                                           | 3.5 kB  00:00:00     
  5. epel                                                                                                                                                                                       | 4.7 kB  00:00:00     
  6. ......
  7. 更新完毕:
  8.   kubeadm.x86_64 0:1.21.9-0                                                                                                                                                                                       
  10. 完毕！

kubeadm upgrade plan验证升级计划，COMPONENT CURRENT TARGET ：告诉我们组件可以从当前版本升级到的版本。

  1. [root@k8scloude1 ~]# kubeadm upgrade plan
  2. [upgrade/config] Making sure the configuration is correct:
  3. [upgrade/config] Reading configuration from the cluster...
  4. [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
  5. [preflight] Running pre-flight checks.
  6. [upgrade] Running cluster health checks
  7. [upgrade] Fetching available versions to upgrade to
  8. [upgrade/versions] Cluster version: v1.21.0
  9. [upgrade/versions] kubeadm version: v1.21.9
  10. I0402 18:09:32.079344   16243 version.go:254] remote version is much newer: v1.23.5; falling back to: stable-1.21
  11. [upgrade/versions] Target version: v1.21.11
  12. [upgrade/versions] Latest version in the v1.21 series: v1.21.11
  14. Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
  15. COMPONENT   CURRENT       TARGET
  16. kubelet     3 x v1.21.0   v1.21.11
  18. Upgrade to the latest version in the v1.21 series:
  20. COMPONENT                 CURRENT    TARGET
  21. kube-apiserver            v1.21.0    v1.21.11
  22. kube-controller-manager   v1.21.0    v1.21.11
  23. kube-scheduler            v1.21.0    v1.21.11
  24. kube-proxy                v1.21.0    v1.21.11
  25. CoreDNS                   v1.8.0     v1.8.0
  26. etcd                      3.4.13-0   3.4.13-0
  28. You can now apply the upgrade by executing the following command:
  30.         kubeadm upgrade apply v1.21.11
  32. Note: Before you can perform this upgrade, you have to update kubeadm to v1.21.11.
  34. _____________________________________________________________________
  37. The table below shows the current state of component configs as understood by this version of kubeadm.
  38. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
  39. resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
  40. upgrade to is denoted in the "PREFERRED VERSION" column.
  42. API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
  43. kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
  44. kubelet.config.k8s.io     v1beta1           v1beta1             no
  45. _____________________________________________________________________

4.2 升级各个组件

上一步是升级kubeadm ,接下来升级各个组件（kube-apiserver，kube-controller-manager等等），

kubeadm upgrade apply v1.21.9升级各个组件到1.21.9版本，如果etcd这个组件不想升级，可以加上选项:kubeadm upgrade apply v1.21.9 --etcd-upgrade=false。

可以提前drain节点，后面drain也可以。

腾空节点：通过将节点标记为不可调度并腾空节点为节点作升级准备：kubectl drain --ignore-daemonsets。

  1. [root@k8scloude1 ~]# kubectl drain k8scloude1 --ignore-daemonsets
  2. node/k8scloude1 cordoned
  3. error: unable to drain node "k8scloude1", aborting command...
  5. There are pending nodes to be drained:
  6.  k8scloude1
  7. error: cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-bcfb98c76-j4gs8, kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9

因为有本地数据，需要加--delete-emptydir-data选项。

  1. [root@k8scloude1 ~]# kubectl drain k8scloude1 --ignore-daemonsets --delete-emptydir-data 
  2. node/k8scloude1 already cordoned
  3. WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-bdlgm, kube-system/kube-proxy-zxlk9, metallb-system/speaker-jfccb
  4. evicting pod kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9
  5. evicting pod kube-system/coredns-545d6fc579-7wm95
  6. evicting pod kube-system/coredns-545d6fc579-87q8j
  7. evicting pod kube-system/metrics-server-bcfb98c76-j4gs8
  8. pod/dashboard-metrics-scraper-7f458d9467-9knf9 evicted
  9. pod/metrics-server-bcfb98c76-j4gs8 evicted
  10. pod/coredns-545d6fc579-7wm95 evicted
  11. pod/coredns-545d6fc579-87q8j evicted
  12. node/k8scloude1 evicted

升级各个组件，--etcd-upgrade=false表示etcd数据库不升级。

  1. [root@k8scloude1 ~]# kubeadm upgrade apply v1.21.9 --etcd-upgrade=false
  2. [upgrade/config] Making sure the configuration is correct:
  3. [upgrade/config] Reading configuration from the cluster...
  4. [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
  5. [preflight] Running pre-flight checks.
  6. [upgrade] Running cluster health checks
  7. [upgrade/version] You have chosen to change the cluster version to "v1.21.9"
  8. [upgrade/versions] Cluster version: v1.21.0
  9. [upgrade/versions] kubeadm version: v1.21.9
  10. [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
  11. [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
  12. [upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
  13. [upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
  14. ......
  15. [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
  16. [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
  17. [addons] Applied essential addon: CoreDNS
  18. [addons] Applied essential addon: kube-proxy
  20. [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.21.9". Enjoy!
  22. [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

现在k8scloude1节点是不可调度状态的。

  1. [root@k8scloude1 ~]# kubectl get node
  2. NAME         STATUS                     ROLES                  AGE   VERSION
  3. k8scloude1   Ready,SchedulingDisabled   control-plane,master   83d   v1.21.0
  4. k8scloude2   Ready                      <none>                 83d   v1.21.0
  5. k8scloude3   Ready                      <none>                 83d   v1.21.0

解除k8scloude1节点的保护：通过将节点标记为可调度，让其重新上线。

  1. [root@k8scloude1 ~]# kubectl uncordon k8scloude1
  2. node/k8scloude1 uncordoned

现在k8scloude1节点是Ready状态的。

  1. [root@k8scloude1 ~]# kubectl get node
  2. NAME         STATUS     ROLES                  AGE   VERSION
  3. k8scloude1   Ready      control-plane,master   83d   v1.21.0
  4. k8scloude2   NotReady   <none>                 83d   v1.21.0
  5. k8scloude3   Ready      <none>                 83d   v1.21.0

4.3 升级 kubelet 和 kubectl

升级 kubelet 和 kubectl到1.21.9版本：yum install -y kubelet-1.23.x-0 kubectl-1.23.x-0 --disableexcludes=kubernetes。

  1. [root@k8scloude1 ~]# yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes
  2. 已加载插件：fastestmirror
  3. Loading mirror speeds from cached hostfile
  4. 正在解决依赖关系
  5. --> 正在检查事务
  6. ---> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级
  7. .....
  8. 更新完毕:
  9.   kubectl.x86_64 0:1.21.9-0                                                                               kubelet.x86_64 0:1.21.9-0                                                                              
  11. 完毕！

重新加载配置文件并重启 kubelet。

  1. [root@k8scloude1 ~]# systemctl daemon-reload ;systemctl restart kubelet

此时k8scloude1节点的版本就变为v1.21.9了，k8s集群的master节点升级成功，如果有多个master，则步骤一样，但是第二个master节点不需要执行kubeadm upgrade apply v1.21.9命令，第二台master节点把kubeadm upgrade apply v1.21.9变为kubeadm upgrade node。

  1. [root@k8scloude1 ~]# kubectl get node
  2. NAME         STATUS   ROLES                  AGE   VERSION
  3. k8scloude1   Ready    control-plane,master   83d   v1.21.9
  4. k8scloude2   Ready    <none>                 83d   v1.21.0
  5. k8scloude3   Ready    <none>                 83d   v1.21.0

五.升级worker工作节点

5.1 升级kubeadm

升级kubeadm到1.21.9版本。

  1. [root@k8scloude2 ~]# yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes
  2. 已加载插件：fastestmirror
  3. base                                                                                                                                                                                       | 3.6 kB  00:00:00     
  4. docker-ce-stable                                                                                                                                                                           | 3.5 kB  00:00:00     
  5. .....
  6. 更新完毕:
  7.   kubeadm.x86_64 0:1.21.9-0                                                                                                                                                                                       
  9. 完毕！

通过将节点标记为不可调度并腾空节点，为节点作升级准备。

  1. [root@k8scloude2 ~]# kubectl drain k8scloude2 --ignore-daemonsets
  2. node/k8scloude2 cordoned
  3. error: unable to drain node "k8scloude2", aborting command...
  5. There are pending nodes to be drained:
  6.  k8scloude2
  7. error: cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/kubernetes-dashboard-7bf6f979bf-c4ckm

因为有本地数据，需要加--delete-emptydir-data选项。

  1. [root@k8scloude2 ~]# kubectl drain k8scloude2 --ignore-daemonsets --delete-emptydir-data
  2. node/k8scloude2 already cordoned
  3. WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-nsbfs, kube-system/kube-proxy-r289t, metallb-system/speaker-nkrgk
  4. evicting pod volume/nfs-client-provisioner-76c576954d-5x7t2
  5. evicting pod kube-system/calico-kube-controllers-6b9fbfff44-dh5vn
  6. evicting pod ingress-nginx/ingress-nginx-admission-patch-hd7p4
  7. evicting pod kube-system/kuboard-78dccb7d9f-pgbdx
  8. ......
  9. I0402 18:47:15.461157   15424 request.go:668] Waited for 1.039246436s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/kube-system/pods/coredns-59d64cd4d4-7jkph
  10. pod/ingress-nginx-controller-59b8bf5fdc-n89sm evicted
  11. I0402 18:47:26.461436   15424 request.go:668] Waited for 1.036011034s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/volume/pods/nfs-client-provisioner-76c576954d-5x7t2
  12. pod/calico-kube-controllers-6b9fbfff44-dh5vn evicted
  13. pod/kubernetes-dashboard-7bf6f979bf-c4ckm evicted
  14. pod/kuboard-78dccb7d9f-pgbdx evicted
  15. pod/coredns-59d64cd4d4-7jkph evicted

k8scloude2节点现在是不可调度状态。

  1. [root@k8scloude2 ~]# kubectl get node
  2. NAME         STATUS                        ROLES                  AGE   VERSION
  3. k8scloude1   Ready                         control-plane,master   83d   v1.21.9
  4. k8scloude2   NotReady,SchedulingDisabled   <none>                 83d   v1.21.0
  5. k8scloude3   Ready                         <none>                 83d   v1.21.0

对于工作节点， kubeadm upgrade node 命令会升级本地的 kubelet 配置。

  1. [root@k8scloude2 ~]# kubeadm upgrade node
  2. [upgrade] Reading configuration from the cluster...
  3. [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
  4. [preflight] Running pre-flight checks
  5. [preflight] Skipping prepull. Not a control plane node.
  6. [upgrade] Skipping phase. Not a control plane node.
  7. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  8. [upgrade] The configuration for this node was successfully updated!
  9. [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

通过将k8scloude2节点标记为可调度，让其重新上线。

  1. [root@k8scloude2 ~]# kubectl uncordon k8scloude2
  2. node/k8scloude2 uncordoned
  4. [root@k8scloude2 ~]# kubectl get node
  5. NAME         STATUS     ROLES                  AGE   VERSION
  6. k8scloude1   Ready      control-plane,master   83d   v1.21.9
  7. k8scloude2   NotReady   <none>                 83d   v1.21.0
  8. k8scloude3   Ready      <none>                 83d   v1.21.0

5.2 升级kubelet和kubectl

升级kubelet和kubectl到1.21.9版本。

  1. [root@k8scloude2 ~]# yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes
  2. 已加载插件：fastestmirror
  3. Loading mirror speeds from cached hostfile
  4. 正在解决依赖关系
  5. --> 正在检查事务
  6. ---> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级
  7. ......
  8.   验证中      : kubectl-1.21.0-0.x86_64                                                                                                                                                                       4/4 
  10. 更新完毕:
  11.   kubectl.x86_64 0:1.21.9-0                                                                               kubelet.x86_64 0:1.21.9-0                                                                              
  13. 完毕！

重新加载配置文件并重启kubelet。

  1. [root@k8scloude2 ~]# systemctl daemon-reload ;systemctl restart kubelet
  3. [root@k8scloude2 ~]# kubectl get nodes
  4. NAME         STATUS     ROLES                  AGE   VERSION
  5. k8scloude1   Ready      control-plane,master   83d   v1.21.9
  6. k8scloude2   NotReady   <none>                 83d   v1.21.0
  7. k8scloude3   Ready      <none>                 83d   v1.21.0

k8scloude3这个worker工作节点的升级步骤和k8scloude2节点一模一样。

k8scloude3节点升级之后，整个Kubernetes(k8s) 集群就升级完毕了，版本都是v1.21.9。

  1. [root@k8scloude1 ~]# kubectl get nodes
  2. NAME         STATUS   ROLES                  AGE   VERSION
  3. k8scloude1   Ready    control-plane,master   83d   v1.21.9
  4. k8scloude2   Ready    <none>                 83d   v1.21.9
  5. k8scloude3   Ready    <none>                 83d   v1.21.9

六.总结

升级Kubernetes(k8s) 集群是确保集群稳定性和安全性的重要步骤。请记住，在执行任何升级操作之前，务必备份重要数据和配置文件。

本博客涵盖了在 CentOS 7.4 上升级 Kubernetes(k8s) 集群的主要步骤，但具体操作过程可能因环境和配置的不同而有所差异。

原文链接：https://www.cnblogs.com/renshengdezheli/p/17546429.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Kubernetes热门文章

  • 【Azure K8S | AKS】在AKS集群中创建 PVC(PersistentVolumeClaim)和 PV(PersistentVolume) 示例
  • 无法访问k8s.gcr.io下载镜像问题解决办法
  • Kubernetes(k8s)访问控制：身份认证
  • Prometheus服务发现之kubernetes_sd_config
  • Arthas排查Kubernetes中应用频繁挂掉重启异常
  • Kubernetes客户端认证——基于CA证书的双向认证方式
  • k8s中kubeconfig的配置以及使用详解
  • Centos7 安装部署 Kubernetes(k8s) 高可用集群
  • Kubernetes 证书详解
  • 案例分享-full gc导致k8s pod重启

    • Kubernetes推荐文章

  • 修改kubeadm证书过期时间及更新k8s集群证书
  • 探究kubernetes 探针参数periodSeconds和timeoutSeconds
  • 从 Helm 到 Operator：Kubernetes应用管理的进化
  • 深入理解Prometheus: Kubernetes环境中的监控实践
  • k8s集群搭建及对一些组件的简单理解（二）
  • K8s高可用集群二进制部署-V1.20
  • Kubernetes容器生命周期 —— 钩子函数详解（postStart、preStop）
  • Kubernetes 审计（Auditing）
  • Thanos解码：打造企业级云原生监控解决方案
  • k8s使用rbd作为存储

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号