经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 数据库/运维 » Kubernetes » 查看文章
使用 anasible 搭建一个多 master 多 worker 的 k8s 集群
来源：cnblogs　　作者：Baocang　　时间：2023/12/5 12:08:03　　对本文有异议

使用 ansible 搭建一个多 master 多 worker 的 k8s 集群

kubernetes + istio 是目前最强大，也是最易于使用的服务网格方案。要使用kubernetes + istio, 首先要搭建 kubernets 集群。搭建kubernetes 集群的方式有很多，其中使用 anisble 自动化搭建 kubernetes 集群的方案非常便捷、可靠。

服务器列表

VIP 192.168.2.111

HOST ROLE IP CPU MEMORY
k8s-lvs-01 LVS MASTER 192.168.2.58 2C 4G
k8s-lvs-02 LVS BACKUP 192.168.2.233 2C 4G
k8s-main-01 K8S MASTER 192.168.2.85 4C 8G
k8s-main-02 K8S MASTER 192.168.2.155 4C 8G
k8s-main-03 K8S MASTER 192.168.2.254 4C 8G
k8s-node-01 K8S WORKER 192.168.2.110 4C 8G
k8s-node-02 K8S WORKER 192.168.2.214 4C 8G
k8s-node-03 K8S WORKER 192.168.2.36 4C 8G

1、在工作机上安装 ansible

GitHub: https://github.com/ansible/ansible

1.1、在 Linux 上安装 ansible：

安装之前可以先更新下apt源

  1. sudo apt-get update

安装 ansible：

  1. sudo apt-get install ansible

在ansible中使用密码方式设置集群必须要安装sshpass，如果不使用密码模式可以不安装：

  1. sudo apt-get install sshpass

如果 apt 找不到ansible和sshpass，可以设置 http://mirrors.aliyun.com/ubuntu 源后再安装，参考链接：https://developer.aliyun.com/mirror/ubuntu/

1.2、在 macOS 上安装 ansible

从 AppStore 安装 Xcode 后执行以下命令

  1. xcode-select --install

安装 homebrew：

  1. /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

安装ansible:

  1. brew install --verbose ansible

要安装 sshpass，需要 使用 baocang/delicious 库：

  1. brew tap baocang/delicious

安装 sshpass:

  1. brew install --verbose sshpass

sshpass是开源的，baocang/delicious是以源码编译的方式安装。

1.3、为 anisble 编写 hosts 文件

使用自己习惯用的文件编辑器编辑hosts.ini, 并输入以下内容：

  1. [all:vars]
  2. kubernetes_vip=192.168.2.111
  3. keepalived_master_ip=192.168.2.58
  5. [lvs]
  6. k8s-lvs-01 ansible_host=192.168.2.58 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  7. k8s-lvs-02 ansible_host=192.168.2.233 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  9. [main]
  10. k8s-main-01 ansible_host=192.168.2.85 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  12. [masters]
  13. k8s-main-02 ansible_host=192.168.2.155 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  14. k8s-main-03 ansible_host=192.168.2.254 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  16. [workers]
  17. k8s-node-01 ansible_host=192.168.2.110 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  18. k8s-node-02 ansible_host=192.168.2.214 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  19. k8s-node-03 ansible_host=192.168.2.36 ansible_ssh_port=22 ansible_ssh_user=ansible ansible_ssh_pass="P@ssw0rd" ansible_sudo_pass="P@ssw0rd"
  21. [kubernetes:children]
  22. main
  23. masters
  24. workers

如果某个服务器的 ssh 是22，可以省略ansible_ssh_port

以上内容 使用 [lvs] 、[main] 、 [masters] 、 [workers] 对服务器进行分组，格式如下：

IP地址 ssh端口号 ssh用户名 登录ssh时使用的密码 执行sudo时使用的密码

[kubernetes:children] 是将[lvs]、 [main] 、[masters] 和 [workers] 组中的服务器组合并放到名为kubernetes的组下。

默认还会有一个all组，包含所有的服务器列表

[all:vars]是为了定义变量，比如这里用变量kubernetes_vip存储 VIP，使用keepalived_master_ip存储 keepalived的 MASTER 节点 IP.

1.4、编写一个简单的ansible配置文件

这个配置夜会在每个服务器上创建一个.test.txt文件：

之后的配置不再指定文件名称，可以在全文最后找到完整的配置来执行。

文件名：demo-anisble-playbook.yml`

  1. ---
  2. - name: Demo
  3.   hosts: lvs
  4.   become: yes
  5.   tasks:
  6.     - name: Write current user to file named .test.txt
  7.       shell: |
  8.         echo `whoami` > .test.txt

然后执行以下命令：

  1. ansible-playbook -i hosts.ini demo-anisble-playbook.yml

得到如下输出：

  1. PLAY [Demo] *********************************************************************************
  3. TASK [Gathering Facts] **********************************************************************
  4. ok: [k8s-lvs-02]
  5. ok: [k8s-lvs-01]
  7. TASK [Write current user to file named .test.txt] *******************************************
  8. changed: [k8s-lvs-01]
  9. changed: [k8s-lvs-02]
  11. PLAY RECAP **********************************************************************************
  12. k8s-lvs-01                : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
  13. k8s-lvs-02                : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

执行完成后，在 lvs 的两个服务器上都有一个名为.test.txt的文件，内容为root，因为become: yes指定了默认以sudo方式运行。

将echo whoami > .test.txt改为rm -rf .test.txt后再执行一遍即可完成清理工作

2、配置 LVS 服务器

2.1、什么是 VIP

VIP（Virtual IP Address，虚拟 IP 地址）是网络中一个不直接分配给某个特定网络接口的 IP 地址。在多种网络服务和应用中，VIP 通常用于实现高可用性、负载均衡和故障转移。

VIP 在不同场景中的应用：

  1. 负载均衡：
    • 在负载均衡器（如 LVS）中，VIP 作为对外的统一访问点。客户端通过 VIP 连接服务，而负载均衡器将请求分发给后端的多个服务器。
    • 这样，即使某个后端服务器出现问题，负载均衡器仍然可以通过将流量重定向到其他健康服务器来保持服务的连续性。
  2. 高可用性集群：
    • 在高可用性（HA）集群中，VIP 可以在多个节点间迁移，通常与故障检测机制一起使用。
    • 当主节点发生故障时，VIP 可以迅速移动到备用节点上，从而实现无缝的故障转移。
  3. 网络虚拟化：
    • 在网络虚拟化环境中，VIP 用于抽象化物理服务器的实际网络地址。
    • 这允许在不同的物理服务器之间灵活地迁移虚拟机或服务。

VIP 的优势：

  • 灵活性：提供了一种机制，使得可以在不同的服务器或网络设备之间迁移服务，而无需修改客户端的配置。
  • 可用性：在故障检测和故障转移机制的帮助下，VIP 增强了服务的可用性和可靠性。
  • 简化管理：对外提供单一访问点，简化了网络管理和客户端配置。

注意事项：

  • VIP 需要正确配置和管理，以确保其在所需的网络场景中有效工作。
  • 在某些设置中，可能需要考虑安全性和权限控制，以防止非授权访问。
  • VIP 的配置和管理通常需要网络管理的专业知识。

2.2、LVS 工作模式

Linux Virtual Server (LVS) 提供了几种不同的工作模式，每种模式都有其特定的用途和网络流量处理方式。以下是 LVS 的主要工作模式：

  1. NAT (Network Address Translation) 模式：
    • 在此模式下，LVS 服务器修改经过的数据包的源或目的 IP 地址。
    • 后端服务器将看到 LVS 服务器的 IP 地址作为客户端请求的源地址。
    • 回应包发回给 LVS 服务器，然后 LVS 服务器再将包发送回原始客户端。
  2. DR (Direct Routing) 模式：
    • 客户端数据包的目的 IP 地址保持不变，直接路由到后端服务器。
    • 后端服务器响应直接发回客户端，绕过 LVS 服务器。
    • 此模式下，源 IP 地址保持不变。
  3. TUN (IP Tunneling) 模式：
    • 数据包被封装在 IP 隧道中，然后发送到后端服务器。
    • 后端服务器解封装数据包，然后直接回应给客户端。
    • 此模式下，原始数据包的源和目的 IP 地址保持不变。
  4. Masquerade (Masq) 模式：
    • 这是基于 NAT 的变体，其中 LVS 服务器将数据包的源 IP 地址替换为自己的 IP 地址。
    • 后端服务器看到的是 LVS 服务器的 IP 地址，而不是客户端的 IP 地址。
    • 回应包首先发回给 LVS 服务器，然后再由 LVS 服务器转发给客户端。

每种模式都有其独特的应用场景和优势。选择哪种模式取决于具体的需求，比如是否需要保留源 IP 地址、网络拓扑结构、以及性能考虑等。通常，DR 和 TUN 模式在性能方面表现更好，因为它们减少了 LVS 服务器的网络流量负载。然而，这些模式可能需要在网络配置上进行更多的调整。相反，NAT 和 Masquerade 模式更易于配置，但可能会影响性能，并且不保留原始的源 IP 地址。

2.3、LVS 负载均衡算法

Linux Virtual Server (LVS) 提供了多种负载均衡算法，用于决定如何将进入的请求分配到不同的后端服务器上。这些算法各有特点，适用于不同的场景。以下是 LVS 中常见的几种负载均衡算法：

  1. 轮询 (Round Robin, RR)：
    • 按顺序依次将请求分配给每个服务器。
    • 当列表到达末尾时，会再次从头开始。
    • 适用于所有服务器处理能力相近的情况。
  2. 加权轮询 (Weighted Round Robin, WRR)：
    • 类似于轮询，但每个服务器都有一个权重。
    • 服务器的处理能力越高，分配给它的请求就越多。
    • 适用于服务器性能不均的情况。
  3. 最少连接 (Least Connections, LC)：
    • 将新请求分配给当前连接数最少的服务器。
    • 这种方式可以更公平地分配负载，特别是在会话长度不一的情况下。
  4. 加权最少连接 (Weighted Least Connections, WLC)：
    • 结合了最少连接和服务器权重的特点。
    • 选择连接数与权重比例最低的服务器。
    • 更适合处理能力不均匀的服务器群。
  5. 基于局部性的最少连接 (Locality-Based Least Connections, LBLC)：
    • 针对具有会话亲和性（session affinity）或持久性（persistence）的应用。
    • 尝试将来自同一客户端的请求发送到同一服务器。
  6. 基于局部性的加权最少连接 (Locality-Based Weighted Least Connections, LBLCR)：
    • 类似于 LBLC，但加入了服务器权重。
    • 用于会话亲和性应用的不均匀环境。
  7. 目的 IP 哈希 (Destination Hashing, DH)：
    • 基于请求的目的 IP 地址来分配请求。
    • 每个请求都固定分配给某个服务器，适合需要强会话亲和性的应用。
  8. 源 IP 哈希 (Source Hashing, SH)：
    • 基于请求的源 IP 地址来分配请求。
    • 保证来自同一源 IP 的请求总是分配到同一服务器，适合需要会话亲和性的场景。

根据应用需求和服务器性能，可以选择最适合的负载均衡算法。例如，如果服务器性能大致相同，轮询或加权轮询可能是个好选择；如果服务器性能不同，可以考虑使用加权最少连接算法。对于需要会话持久性的应用，基于哈希的算法可能更加适合。

两台lvs 服务器都需要安装ipvsadm和keepalived组件，其中ipvsadm用于管理和想看ipvs规则，keepalived用于管理VIP和生成ipvs规则，进行健康检查等。

2.4、编写 keepalived.conf.j2 模板文件

在k8s-setup.yml文件同目录下的resources目录中创建 keepalived.conf.j2 文件

如果没有机器名，可以将 ansible_hostname 改为判断 ansible_host 或 ansible_default_ipv4.address 用IP判断

文件名：resources/keepalived.conf.j2

  1. vrrp_instance VI_1 {
  2.     state {{ 'MASTER' if ansible_host == keepalived_master_ip else 'BACKUP' }}
  3.     interface ens160
  4.     virtual_router_id 51
  5.     priority {{ 255 if ansible_host == keepalived_master_ip else 254 }}
  6.     advert_int 1
  8.     authentication {
  9.         auth_type PASS
  10.         auth_pass 123456
  11.     } 
  13.     virtual_ipaddress {
  14.         {{ kubernetes_vip }}/24
  15.     }
  16. }
  19. # masters with port 6443
  20. virtual_server {{ kubernetes_vip }} 6443 {
  21.     delay_loop 6
  22.     lb_algo wlc
  23.     lb_kind DR
  24.     persistence_timeout 360
  25.     protocol TCP
  27. {% for host in groups['main'] %}
  29.     # {{ host }}
  30.     real_server {{ hostvars[host]['ansible_host'] }} 6443 {
  31.         weight 1
  32.         SSL_GET {
  33.             url {
  34.                 path /livez?verbose
  35.                 status_code 200
  36.             }
  37.             connect_timeout 3
  38.             nb_get_retry 3
  39.             delay_before_retry 3
  40.         }
  41.     }
  42. {% endfor %}
  43. {% for host in groups['masters'] %}
  45.     # {{ host }}
  46.     real_server {{ hostvars[host]['ansible_host'] }} 6443 {
  47.         weight 1
  48.         SSL_GET {
  49.             url {
  50.                 path /livez?verbose
  51.                 status_code 200
  52.             }
  53.             connect_timeout 3
  54.             nb_get_retry 3
  55.             delay_before_retry 3
  56.         }
  57.     }
  58. {% endfor %}
  59. }
  61. # workers with port 80
  62. virtual_server {{ kubernetes_vip }} 80 {
  63.     delay_loop 6
  64.     lb_algo wlc
  65.     lb_kind DR
  66.     persistence_timeout 7200
  67.     protocol TCP
  68. {% for host in groups['workers'] %}
  70.     # {{ host }}
  71.     real_server {{ hostvars[host]['ansible_host'] }} 80 {
  72.         weight 1
  73.         TCP_CHECK {
  74.             connect_timeout 10
  75.             connect_port 80
  76.         }
  77.     }
  78. {% endfor %}
  79. }
  81. # workers with port 443
  82. virtual_server {{ kubernetes_vip }} 443 {
  83.     delay_loop 6
  84.     lb_algo wlc
  85.     lb_kind DR
  86.     persistence_timeout 7200
  87.     protocol TCP
  89. {% for host in groups['workers'] %}
  91.     # {{ host }}
  92.     real_server {{ hostvars[host]['ansible_host'] }} 443 {
  93.         weight 1
  94.         TCP_CHECK {
  95.             connect_timeout 10
  96.             connect_port 443
  97.         }
  98.     }
  99. {% endfor %}
  100. }

vrrp_instance 指定了实例名为 VI_1，在 k8s-lvs-01 上配置为 MASTER，在其余机器上配置为BACKUP

interface ens160 是当前网卡上的接口名称，通过 ip addr 或 ifconfig 命令可获取，输出中带有当前节点IP地址的就是，初始情况下，有一个回路lo接口和另一个类似 ens160、en0之类的接口

advert_int 1 指定了Keepalived发送VRRP通告的时间间隔为1秒

priority 一般在 BACKUP节点上应该低于 MASTER节点

virtual_ipaddress中配置的就是VIP（Virtual IP)

virtual_server 用于定义 虚拟服务器，一个虚拟服务器下有多个真实的服务器（real_server）

lb_algo wlc 指定了负载均衡算法
lb_kind DR 指定了使用 Direct Routing 模式路由数据

{% for host in groups['masters'] %} 这些语句顶格写是为了防止出现锁进错误

2.5、 ipvsadm 和 keepalived 的 ansible 配置文件

  1. ---
  2. - name: Setup Load Balancer with IPVS and Keepalived
  3.   hosts: lvs
  4.   become: yes
  5.   tasks:
  6.     # Upgrade all installed packages to their latest versions
  7.     - name: Upgrade all installed apt packages
  8.       apt:
  9.         upgrade: 'yes'
  10.         update_cache: yes
  11.         cache_valid_time: 3600  # Cache is considered valid for 1 hour
  13.     # Install IP Virtual Server (IPVS) administration utility
  14.     - name: Install ipvsadm for IPVS management
  15.       apt:
  16.         name: ipvsadm
  17.         state: present
  19.     # Install keepalived for high availability
  20.     - name: Install Keepalived for load balancing
  21.       apt:
  22.         name: keepalived
  23.         state: present
  25.     # Deploy keepalived configuration from a Jinja2 template
  26.     - name: Deploy keepalived configuration file
  27.       template:
  28.         src: resources/keepalived.conf.j2
  29.         dest: /etc/keepalived/keepalived.conf
  31.     # Restart keepalived to apply changes
  32.     - name: Restart Keepalived service
  33.       service:
  34.         name: keepalived
  35.         state: restarted

3、安装 kubernetes 和 containerd.io

3.1、各个组件的功能

  1. kubelet：
    • kubelet 是运行在所有 Kubernetes 集群节点上的一个代理，负责管理该节点上的容器。
    • 它监控由 API 服务器下发的指令（PodSpecs，Pod 的规格描述），确保容器的状态与这些规格描述相匹配。
    • kubelet 负责维护容器的生命周期，比如启动、停止、重启容器，以及实现健康检查。
  2. kubeadm：
    • kubeadm 是一个工具，用于快速部署 Kubernetes 集群。
    • 它帮助你初始化集群（设置 master 节点），加入新的节点到集群中，以及进行必要的配置以建立一个可工作的集群。
    • kubeadm 并不管理集群中的节点或者 Pod，它只负责集群的启动和扩展。
  3. kubectl：
    • kubectl 是 Kubernetes 集群的命令行工具，允许用户与集群进行交互。
    • 它可以用来部署应用、检查和管理集群资源以及查看日志等。
    • kubectl 主要与 Kubernetes API 服务器通信，执行用户的命令。
  4. containerd.io：
    • containerd.io 是一个容器运行时，Kubernetes 用它来运行容器。
    • 它负责镜像的拉取、容器的运行以及容器的生命周期管理。
    • containerd.io 是 Docker 的核心组件之一，但在 Kubernetes 中可以独立使用，不依赖于完整的 Docker 工具集。

3.2、安装 Kubernetes 依赖项

  1. ---
  2. - name: Install kubernetes packages and containerd.io
  3.   hosts: kubernetes
  4.   become: yes
  5.   tasks:
  7.     # Upgrade all installed packages to their latest versions
  8.     - name: Upgrade all installed apt packages
  9.       apt:
  10.         upgrade: 'yes'
  11.         update_cache: yes
  12.         cache_valid_time: 3600  # Cache is considered valid for 1 hour
  14.     # Install required packages for Kubernetes and Docker setup
  15.     - name: Install prerequisites for Kubernetes and Docker
  16.       apt:
  17.         name:
  18.           - ca-certificates
  19.           - curl
  20.           - gnupg
  21.         update_cache: yes
  22.         cache_valid_time: 3600
  24.     # Ensure the keyring directory exists for storing GPG keys
  25.     - name: Create /etc/apt/keyrings directory for GPG keys
  26.       file:
  27.         path: /etc/apt/keyrings
  28.         state: directory
  29.         mode: '0755'
  31.     # Add Docker's official GPG key
  32.     - name: Add official Docker GPG key to keyring
  33.       apt_key:
  34.         url: https://download.docker.com/linux/ubuntu/gpg
  35.         keyring: /etc/apt/keyrings/docker.gpg
  36.         state: present
  38.     # Add Docker's apt repository
  39.     - name: Add Docker repository to apt sources
  40.       apt_repository:
  41.         # repo: "deb [arch={{ ansible_architecture }} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
  42.         repo: "deb [signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
  43.         filename: docker
  44.         update_cache: yes
  45.       notify: Update apt cache
  47.     # Add Kubernetes' GPG key
  48.     - name: Add Kubernetes GPG key to keyring
  49.       apt_key:
  50.         url: https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key
  51.         keyring: /etc/apt/keyrings/kubernetes-apt-keyring.gpg
  52.         state: present
  54.     # Add Kubernetes' apt repository
  55.     - name: Add Kubernetes repository to apt sources
  56.       lineinfile:
  57.         path: /etc/apt/sources.list.d/kubernetes.list
  58.         line: 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /'
  59.         create: yes
  60.       notify: Update apt cache
  62.     # Install Kubernetes packages
  63.     - name: Install Kubernetes packages (kubelet, kubeadm, kubectl) and containerd.io
  64.       apt:
  65.         name:
  66.           - kubelet
  67.           - kubeadm
  68.           - kubectl
  69.           - containerd.io
  70.         state: present
  72.     # Hold the installed packages to prevent automatic updates
  73.     - name: Hold Kubernetes packages and containerd.io
  74.       dpkg_selections:
  75.         name: "{{ item }}"
  76.         selection: hold
  77.       loop:
  78.         - kubelet
  79.         - kubeadm
  80.         - kubectl
  81.         - containerd.io
  83.   handlers:
  84.     # Handler to update apt cache when notified
  85.     - name: Update apt cache
  86.       apt:
  87.         update_cache: yes

如果要指定 repo 的 arch，可以如下方式使用（以docker为例)：

  1.         repo: "deb [arch={{ ansible_architecture }} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"

3.3、转发 IPv4 并让 iptables 看到桥接流量

  1. ---
  2. - name: Configure Kubernetes prerequisites
  3.   hosts: kubernetes
  4.   become: yes  # to run tasks that require sudo
  5.   tasks:
  6.     - name: Load Kernel Modules
  7.       copy:
  8.         content: |
  9.           overlay
  10.           br_netfilter
  11.         dest: /etc/modules-load.d/k8s.conf
  12.       notify: Load Modules
  14.     - name: Set Sysctl Parameters
  15.       copy:
  16.         content: |
  17.           net.bridge.bridge-nf-call-iptables  = 1
  18.           net.bridge.bridge-nf-call-ip6tables = 1
  19.           net.ipv4.ip_forward                 = 1
  20.         dest: /etc/sysctl.d/k8s.conf
  21.       notify: Apply Sysctl
  23.   handlers:
  24.     - name: Load Modules
  25.       modprobe:
  26.         name: "{{ item }}"
  27.         state: present
  28.       loop:
  29.         - overlay
  30.         - br_netfilter
  32.     - name: Apply Sysctl
  33.       command: sysctl --system

4、预加载 k8s 需要的镜像

containerd 其实也用的到一个 k8s中用的到的镜像，只不过可能版本更低，不过可以改成相同版本的，所以在配置 containerd 之前预加载 k8s 需要的镜像。

  1. ---
  2. - name: Prefetch kubernetes images
  3.   hosts: kubernetes
  4.   become: true
  5.   tasks:
  6.     - name: Get kubeadm version
  7.       command: kubeadm version -o short
  8.       register: kubeadm_version
  10.     - name: List Kubernetes images for the specific kubeadm version
  11.       command: "kubeadm config images list --kubernetes-version={{ kubeadm_version.stdout }}"
  12.       register: kubernetes_images
  14.     - name: Pull and retag Kubernetes images from Aliyun registry
  15.       block:
  16.         - name: List old images in k8s.io namespace
  17.           command: ctr -n k8s.io images list -q
  18.           register: old_images_list
  20.         - name: Pull Kubernetes image from Aliyun
  21.           command: "ctr -n k8s.io images pull registry.aliyuncs.com/google_containers/{{ item.split('/')[-1] }}"
  22.           loop: "{{ kubernetes_images.stdout_lines }}"
  23.           when: item not in old_images_list.stdout
  24.           loop_control:
  25.             label: "{{ item }}"
  27.         - name: Retag Kubernetes image
  28.           command: "ctr -n k8s.io images tag registry.aliyuncs.com/google_containers/{{ item.split('/')[-1] }} {{ item }}"
  29.           loop: "{{ kubernetes_images.stdout_lines }}"
  30.           when: item not in old_images_list.stdout
  31.           loop_control:
  32.             label: "{{ item }}"
  34.         - name: List new images in k8s.io namespace
  35.           command: ctr -n k8s.io images list -q
  36.           register: new_images_list
  38.         - name: Remove images from Aliyun registry
  39.           command: "ctr -n k8s.io images remove {{ item }}"
  40.           loop: "{{ new_images_list.stdout_lines }}"
  41.           when: item.startswith('registry.aliyuncs.com/google_containers')
  42.           loop_control:
  43.             label: "{{ item }}"
  45.         # # Optional: Remove old SHA256 tags if necessary
  46.         # - name: Remove old SHA256 tags
  47.         #   command: "ctr -n k8s.io images remove {{ item }}"
  48.         #   loop: "{{ new_images_list.stdout_lines }}"
  49.         #   when: item.startswith('sha256:')
  50.         #   loop_control:
  51.         #     label: "{{ item }}"
  53.     - name: Restart containerd service
  54.       service:
  55.         name: containerd
  56.         state: restarted

通过anisble 执行后就完成了所有的镜像预加载工作，也可以在kubeadm的init配置文件中改写imageRepository: registry.k8s.io来省略这步操作。

注释的内容是删除 sha256:开头的镜像，建议不要删除.

  1. ctr -n k8s.io images ls -q
  2. registry.k8s.io/coredns/coredns:v1.10.1
  3. registry.k8s.io/etcd:3.5.9-0
  4. registry.k8s.io/kube-apiserver:v1.28.4
  5. registry.k8s.io/kube-controller-manager:v1.28.4
  6. registry.k8s.io/kube-proxy:v1.28.4
  7. registry.k8s.io/kube-scheduler:v1.28.4
  8. registry.k8s.io/pause:3.9
  9. sha256:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9
  10. sha256:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257
  11. sha256:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e
  12. sha256:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591
  13. sha256:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1
  14. sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c
  15. sha256:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc

SHA256 标签在 Docker 和容器技术中扮演着重要角色，主要涉及到镜像的完整性和版本控制。以下是 SHA256 标签的作用和删除它们的潜在好处与坏处：

SHA256 标签的作用：

  1. 完整性验证：SHA256 标签是镜像内容的加密哈希。它们用于验证镜像内容的完整性，确保在传输过程中没有被篡改。
  2. 唯一标识：每个镜像都有一个唯一的 SHA256 哈希，这有助于区分不同的镜像版本，即使它们可能有相同的标签（如 latest）。
  3. 版本控制：在多次构建和更新镜像时，SHA256 哈希有助于追踪特定的镜像版本。

删除 SHA256 标签的好处：

  1. 简化镜像管理：删除长的 SHA256 标签可以简化镜像管理，使得只通过人类可读的标签（如 v1.0.0）来识别镜像。
  2. 减少混淆：对于那些不需要详细追踪每个镜像版本的场景，移除这些长的哈希标签可以减少视觉上的混淆。

删除 SHA256 标签的坏处：

  1. 丢失详细版本信息：删除 SHA256 标签意味着丢失了一个精确指向特定镜像版本的方式。如果需要回溯或审核，这可能是个问题。
  2. 潜在的安全风险：在某些环境中，保留 SHA256 标签可以作为额外的安全措施，以确保镜像在拉取和部署过程中保持不变。

总的来说，是否删除 SHA256 标签取决于您的具体需求和管理偏好。在生产环境中，通常建议保留这些标签以便于版本控制和安全审核。在开发或测试环境中，如果标签的数量导致管理变得繁琐，可以考虑移除它们。

5、配置 containerd

containerd 默认的配置文件禁用了cri插件，并且默认SystemdCgroup = false，在开始配置kubernetes 前，需要启用cri插件，并且设置SystemdCgroup = true

SystemdCgroup=true 是一个配置选项，通常在容器运行时环境的配置文件中使用，特别是在与 Docker 或 Kubernetes 相关的设置中。这个选项涉及到 Linux 中的 cgroup（控制组）管理，具体来说是 cgroup 的第二版本（cgroup v2）。

4.1、cgroup 简介

  • 控制组 (cgroup)：cgroup 是 Linux 内核的一个特性，允许您分配、优先排序、限制、记录和隔离资源的使用（如 CPU、内存、磁盘 I/O 等）。
  • cgroup v1 与 v2：cgroup v1 是最初的版本，而 cgroup v2 是一个更新的版本，提供了更一致和简化的接口。

5.2、Systemd 与 cgroup

  • Systemd：Systemd 是一个系统和服务管理器，它为现代 Linux 发行版提供了初始化系统、服务管理等功能。Systemd 也可以用来管理 cgroup。
  • Systemd 与 cgroup 集成：当 Systemd 作为进程管理器时，它可以使用 cgroup 来管理系统资源。在 cgroup v2 中，Systemd 提供了更好的集成和资源管理。

5.3、SystemdCgroup=true 的含义

当您在容器运行时的配置文件中设置 SystemdCgroup=true 时，您告诉容器运行时（如 Docker 或 containerd）使用 Systemd 来管理容器的 cgroup。这样做的优点包括：

  1. 更好的资源管理：Systemd 提供了对资源限制和优先级的细粒度控制，允许更有效地管理容器使用的资源。
  2. 统一的管理：使用 Systemd 管理 cgroup 可以和系统中的其他服务管理机制保持一致。
  3. 更好的兼容性：特别是对于使用 cgroup v2 的系统，使用 Systemd 进行管理可以确保更好的兼容性和稳定性。

5.4、使用场景

在 Kubernetes 环境中，这个配置通常出现在 Kubelet 的配置中，确保容器的资源管理与系统的其它部分保持一致。这对于保证系统资源的高效利用和稳定运行非常重要。

5.5、修改 containerd 配置文件

通过将 disabled_plugins = ["cri"] 中内容清空可以启用cri插件，设置 SystemdCgroup=true 可以使用 Systemd 来管理容器的 cgroup，并且修改 sandbox_image，使其与 k8s 保持一致。

  1. ---
  2. - name: Configure containerd
  3.   hosts: kubernetes
  4.   become: true
  5.   tasks:
  7.     - name: Get Kubernetes images list
  8.       command: kubeadm config images list
  9.       register: kubernetes_images
  11.     - name: Set pause image variable
  12.       set_fact:
  13.         pause_image: "{{ kubernetes_images.stdout_lines | select('match', '^registry.k8s.io/pause:') | first }}"
  15.     - name: Generate default containerd config
  16.       command: containerd config default
  17.       register: containerd_config
  18.       changed_when: false
  20.     - name: Write containerd config to file
  21.       copy:
  22.         dest: /etc/containerd/config.toml
  23.         content: "{{ containerd_config.stdout }}"
  24.         mode: '0644'
  25.         
  26.     - name: Replace 'sandbox_image' and 'SystemdCgroup' in containerd config
  27.       lineinfile:
  28.         path: /etc/containerd/config.toml
  29.         regexp: "{{ item.regexp }}"
  30.         line: "{{ item.line }}"
  31.       loop:
  32.         - { regexp: '^\s*sandbox_image\s*=.*$', line: '    sandbox_image = "{{ pause_image }}"' }
  33.         - { regexp: 'SystemdCgroup =.*', line: '            SystemdCgroup = true' }
  35.     - name: Restart containerd service
  36.       service:
  37.         name: containerd
  38.         state: restarted

6、初始化 main master 节点

使用 kubeadm 初始化 k8s 集群时，只需要初始化一个 master 节点，其余的 master 和 worker 节点都可以使用 kubeadm join命令加入，所以需要先初始化 main master 节点。

6.1、生成 kubeadm init用的 token

kubeadm 默认的 token 是 abcdef.0123456789abcdef，其格式要求为"[a-z0-9]{6}.[a-z0-9]{16}， 可以通过下面这条命令来生成：

  1. LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 6; echo -n '.'; LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 16

6.2、编写 kubeadm-init.yaml.j2 模板文件

文件名：resources/kubeadm-init.yaml.j2

通过kubeadm config print init-defaults命令获取到默认的配置文件，然后修改为如下内容：

  1. apiVersion: kubeadm.k8s.io/v1beta3
  2. bootstrapTokens:
  3. - token: {{ token }}
  4.   ttl: 0s
  5.   usages:
  6.     - signing
  7.     - authentication
  8.   description: "kubeadm bootstrap token"
  9.   groups:
  10.     - system:bootstrappers:kubeadm:default-node-token
  11. kind: InitConfiguration
  12. localAPIEndpoint:
  13.   advertiseAddress: {{ ansible_host }}
  14.   bindPort: 6443
  15. nodeRegistration:
  16.   criSocket: unix:///var/run/containerd/containerd.sock
  17.   imagePullPolicy: IfNotPresent
  18.   name: k8s-main-01
  19.   taints: null
  20. ---
  21. apiServer:
  22.   certSANs:
  23.     - {{ kubernetes_vip }}
  24.     - {{ ansible_host }}
  25. {% for host in groups['masters'] %}
  26.     - {{ hostvars[host]['ansible_host'] }}
  27. {% endfor %}
  28.     - k8s-main-01
  29.     - k8s-main-02
  30.     - k8s-main-03
  31.     - kubernetes.cluster
  32.   timeoutForControlPlane: 4m0s
  33. apiVersion: kubeadm.k8s.io/v1beta3
  34. certificatesDir: /etc/kubernetes/pki
  35. clusterName: kubernetes
  36. controllerManager: {}
  37. dns: {}
  38. etcd:
  39.   local:
  40.     dataDir: /var/lib/etcd
  41. imageRepository: registry.k8s.io
  42. kind: ClusterConfiguration
  43. kubernetesVersion: 1.28.4
  44. controlPlaneEndpoint: "{{ kubernetes_vip }}:6443"
  45. networking:
  46.   dnsDomain: cluster.local
  47.   podSubnet: 10.244.0.0/12
  48.   serviceSubnet: 10.96.0.0/12
  49. scheduler: {}
  50. ---
  51. kind: KubeletConfiguration
  52. apiVersion: kubelet.config.k8s.io/v1beta1
  53. cgroupDriver: systemd
  54. ---
  55. ---
  56. apiVersion: kubeproxy.config.k8s.io/v1alpha1
  57. kind: KubeProxyConfiguration
  58. mode: ipvs

ttl: 0s指定了永不过期。

推荐的做法是设置合理的过期时间，后续使用kubeadm token create --print-join-command的方式获取 token。

6.3、生成 kubeadm init 配置文件

通过 register 捕获 shell 脚本生成的内容，并且在 Generate kubeadm config file 子任务中定义名为token的变量接收来自 stdout 输出的内容。

使用 register 的基本原理：

  • 捕获输出：当您在某个任务上使用 register 关键字时，Ansible 会捕获该任务的输出并将其存储在您指定的变量中。
  • 变量使用：注册后的变量可以在 playbook 的后续任务中使用，允许您根据之前任务的输出来决定后续操作

编写 ansible 任务配置文件内容如下：

  1. ---
  2. - name: Initialize Kubernetes Cluster on Main Master
  3.   hosts: main
  4.   become: true
  5.   tasks:
  6.     - name: Generate Kubernetes init token
  7.       shell: >
  8.         LC_CTYPE=C tr -dc 'a-z' </dev/urandom | head -c 1;
  9.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 5; 
  10.         echo -n '.'; 
  11.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 16
  12.       register: k8s_init_token
  14.     - name: Generate kubeadm config file
  15.       template:
  16.         src: resources/kubeadm-init.yaml.j2
  17.         dest: kubeadm-init.yaml
  18.       vars:
  19.         token: "{{ k8s_init_token.stdout }}"

以上配置文件会在 /home/ansible 目录下生成一个名为kubeadm-init.yaml的文件，用户名是在 hosts.ini 中 main 组的服务器通过 ansible_ssh_user 指定的。可以登录到服务器检查 kubeadm-init.yaml 文件的内容。

6.4、使用 kubeadm 创建 主 master 节点

resources/cilium-linux-amd64.tar.gz文件下载方式参考

https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-cilium-cli

在 linux 上安装 cilium-cli 时这样下载

  1. CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
  2. CLI_ARCH=amd64
  3. if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
  4. curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
  5. # 在Linux上这样检查文件 sha256
  6. # sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
  7. # 在Mac上这样检查 sha256
  8. # shasum -a 256 -c cilium-linux-${CLI_ARCH}.tar.gz.sha256sum

在 Mac 上安装 cilium-cli 时这样下载

  1. CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
  2. CLI_ARCH=amd64
  3. if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
  4. curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
  5. shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum

这是在上面的配置文件基础上增加的内容

建议与后面的加入其它 masters 和 workers 的步骤放同一个文件中一起执行，这样可以通过set_fact设置的变量直接执行kubeadm join命令。

下面的配置文件也将 masters 和 workers 上需要的join命令存到了 .master_join_command.txt和.worker_join_command.txt中，单独加入节点，或后续手动执行kubeadm join可以使用其中的命令。

下面的配置直接在 k8s 的 main master 上安装cilium-cli， 将下载好的文件放到 resources/cilium-linux-amd64.tar.gz 目录:

  1. ---
  2. - name: Initialize Kubernetes Cluster on Main Master
  3.   hosts: main
  4.   become: true
  5.   tasks:
  7.     - name: Check if IP address is already present
  8.       shell: "ip addr show dev lo | grep {{ kubernetes_vip }}"
  9.       register: ip_check
  10.       ignore_errors: yes
  11.       failed_when: false
  12.       changed_when: false
  13.     
  14.     - name: Debug print ip_check result
  15.       debug:
  16.         msg: "{{ ip_check }}"
  18.     - name: Add IP address to loopback interface
  19.       command:
  20.         cmd: "ip addr add {{ kubernetes_vip }}/32 dev lo"
  21.       when: ip_check.rc != 0
  23.     - name: Generate Kubernetes init token
  24.       shell: >
  25.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 6; 
  26.         echo -n '.'; 
  27.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 16
  28.       register: k8s_init_token
  30.     - name: Generate kubeadm config file
  31.       template:
  32.         src: resources/kubeadm-init.yaml.j2
  33.         dest: kubeadm-init.yaml
  34.       vars:
  35.         token: "{{ k8s_init_token.stdout }}"
  37.     - name: Initialize the Kubernetes cluster using kubeadm
  38.       command:
  39.         cmd: kubeadm init --v=5 --skip-phases=addon/kube-proxy --upload-certs --config kubeadm-init.yaml
  40.       register: kubeadm_init
  41.     
  42.     - name: Set fact for master join command
  43.       set_fact:
  44.         master_join_command: "{{ kubeadm_init.stdout | regex_search('kubeadm join(.*\\n)+?.*--control-plane.*', multiline=True) }}"
  45.         cacheable: yes
  46.       run_once: true
  48.     - name: Set fact for worker join command
  49.       set_fact:
  50.         worker_join_command: "{{ kubeadm_init.stdout | regex_search('kubeadm join(.*\\n)+?.*sha256:[a-z0-9]{64}', multiline=True) }}"
  51.         cacheable: yes
  52.       run_once: true
  54.     # - name: Create the target directory if it doesn't exist
  55.     #   file:
  56.     #     path: ~/.kube
  57.     #     state: directory
  58.     #     owner: "{{ ansible_user_id }}"
  59.     #     group: "{{ ansible_user_id }}"
  60.     #     mode: '0755'
  61.     #   when: not ansible_check_mode  # This ensures it only runs when not in check mode
  63.     # - name: Copy kube admin config to ansible user directory
  64.     #   copy:
  65.     #     src: /etc/kubernetes/admin.conf
  66.     #     dest: ~/.kube/config
  67.     #     remote_src: yes
  68.     #     owner: "{{ ansible_user_id }}"
  69.     #     group: "{{ ansible_user_id }}"
  70.     #     mode: '0644'
  72.     - name: Write master join command to .master_join_command.txt
  73.       copy:
  74.         content: "{{ master_join_command }}"
  75.         dest: ".master_join_command.txt"
  76.         mode: '0664'
  77.       delegate_to: localhost
  79.     - name: Append worker join command to .worker_join_command.txt
  80.       lineinfile:
  81.         path: ".worker_join_command.txt"
  82.         line: "{{ worker_join_command }}"
  83.         create: yes
  84.       delegate_to: localhost
  87. - name: Install cilium on Main Master
  88.   hosts: main
  89.   become: true
  90.   tasks:
  92.     - name: Ensure tar is installed (Debian/Ubuntu)
  93.       apt:
  94.         name: tar
  95.         state: present
  96.       when: ansible_os_family == "Debian"
  98.     - name: Check for Cilium binary in /usr/local/bin
  99.       stat:
  100.         path: /usr/local/bin/cilium
  101.       register: cilium_binary
  103.     - name: Transfer and Extract Cilium
  104.       unarchive:
  105.         src: resources/cilium-linux-amd64.tar.gz
  106.         dest: /usr/local/bin
  107.         remote_src: no
  108.       when: not cilium_binary.stat.exists
  110.     - name: Install cilium to the Kubernetes cluster
  111.       environment:
  112.         KUBECONFIG: /etc/kubernetes/admin.conf
  113.       command:
  114.         cmd: cilium install --version 1.14.4 --set kubeProxyReplacement=true
  115.     
  116.     - name: Wait for Kubernetes cluster to become ready
  117.       environment:
  118.         KUBECONFIG: /etc/kubernetes/admin.conf
  119.       command: kubectl get nodes
  120.       register: kubectl_output
  121.       until: kubectl_output.stdout.find("Ready") != -1
  122.       retries: 20
  123.       delay: 30

如果想继续使用kube-proxy，删除kubeadm init命令中的--skip-phases=addon/kube-proxy参数和cilium install命令中的--set kubeProxyReplacement=true。

6.5、将其余的 masters 加入集群

  1. ---
  2. - name: Join Masters to the Cluster
  3.   hosts: masters
  4.   become: true
  5.   tasks:
  6.     - name: Joining master node to the Kubernetes cluster
  7.       shell:
  8.         cmd: "{{ hostvars['k8s-main-01']['master_join_command'] }}"
  9.       ignore_errors: yes
  11.     - name: Wait for node to become ready
  12.       environment:
  13.         KUBECONFIG: /etc/kubernetes/admin.conf
  14.       command: kubectl get nodes
  15.       register: kubectl_output
  16.       until: kubectl_output.stdout.find("NotReady") == -1
  17.       retries: 20
  18.       delay: 30
  20.     - name: Check if IP address is already present
  21.       shell: "ip addr show dev lo | grep {{ kubernetes_vip }}"
  22.       register: ip_check
  23.       ignore_errors: yes
  24.       failed_when: false
  25.       changed_when: false
  26.     
  27.     - name: Debug print ip_check result
  28.       debug:
  29.         msg: "{{ ip_check }}"
  31.     - name: Add IP address to loopback interface
  32.       command:
  33.         cmd: "ip addr add {{ kubernetes_vip }}/32 dev lo"
  34.       when: ip_check.rc != 0

6.6、将其余的 works 加入集群

  1. ---
  2. - name: Join Worker Nodes to the Cluster
  3.   hosts: workers
  4.   become: true
  5.   tasks:
  6.     - name: Joining master node to the Kubernetes cluster
  7.       shell:
  8.         cmd: "{{ hostvars['k8s-main-01']['worker_join_command'] }}"
  9.       ignore_errors: yes
  11.     - name: Check if IP address is already present
  12.       shell: "ip addr show dev lo | grep {{ kubernetes_vip }}"
  13.       register: ip_check
  14.       ignore_errors: yes
  15.       failed_when: false
  16.       changed_when: false
  17.     
  18.     - name: Debug print ip_check result
  19.       debug:
  20.         msg: "{{ ip_check }}"
  22.     - name: Add IP address to loopback interface
  23.       command:
  24.         cmd: "ip addr add {{ kubernetes_vip }}/32 dev lo"
  25.       when: ip_check.rc != 0

完整的 ansible playbook 配置

只需要执行这个配置文件，加上resources目录中的三个文件，就可以完成上面所有的操作

依赖的文件：

  • 参考 2.4、编写 keepalived.conf.j2 模板文件
  • 参考 6.2、编写 kubeadm-init.yaml.j2 模板文件
  • 参考 6.4、使用 kubeadm 创建 主 master 节点 中下载的 cilium-cli 文件，需要 linux 版本
  1. ---
  2. - name: Setup Load Balancer with IPVS and Keepalived
  3.   hosts: lvs
  4.   become: yes
  5.   tasks:
  6.     # Upgrade all installed packages to their latest versions
  7.     - name: Upgrade all installed apt packages
  8.       apt:
  9.         upgrade: 'yes'
  10.         update_cache: yes
  11.         cache_valid_time: 3600  # Cache is considered valid for 1 hour
  13.     # Install IP Virtual Server (IPVS) administration utility
  14.     - name: Install ipvsadm for IPVS management
  15.       apt:
  16.         name: ipvsadm
  17.         state: present
  19.     # Install keepalived for high availability
  20.     - name: Install Keepalived for load balancing
  21.       apt:
  22.         name: keepalived
  23.         state: present
  25.     # Deploy keepalived configuration from a Jinja2 template
  26.     - name: Deploy keepalived configuration file
  27.       template:
  28.         src: resources/keepalived.conf.j2
  29.         dest: /etc/keepalived/keepalived.conf
  31.     # Restart keepalived to apply changes
  32.     - name: Restart Keepalived service
  33.       service:
  34.         name: keepalived
  35.         state: restarted
  38. - name: Install kubernetes packages and containerd.io
  39.   hosts: kubernetes
  40.   become: yes
  41.   tasks:
  43.     # Upgrade all installed packages to their latest versions
  44.     - name: Upgrade all installed apt packages
  45.       apt:
  46.         upgrade: 'yes'
  47.         update_cache: yes
  48.         cache_valid_time: 3600  # Cache is considered valid for 1 hour
  50.     # Install required packages for Kubernetes and Docker setup
  51.     - name: Install prerequisites for Kubernetes and Docker
  52.       apt:
  53.         name:
  54.           - ca-certificates
  55.           - curl
  56.           - gnupg
  57.         update_cache: yes
  58.         cache_valid_time: 3600
  60.     # Ensure the keyring directory exists for storing GPG keys
  61.     - name: Create /etc/apt/keyrings directory for GPG keys
  62.       file:
  63.         path: /etc/apt/keyrings
  64.         state: directory
  65.         mode: '0755'
  67.     # Add Docker's official GPG key
  68.     - name: Add official Docker GPG key to keyring
  69.       apt_key:
  70.         url: https://download.docker.com/linux/ubuntu/gpg
  71.         keyring: /etc/apt/keyrings/docker.gpg
  72.         state: present
  74.     # Add Docker's apt repository
  75.     - name: Add Docker repository to apt sources
  76.       apt_repository:
  77.         # repo: "deb [arch={{ ansible_architecture }} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
  78.         repo: "deb [signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
  79.         filename: docker
  80.         update_cache: yes
  81.       notify: Update apt cache
  83.     # Add Kubernetes' GPG key
  84.     - name: Add Kubernetes GPG key to keyring
  85.       apt_key:
  86.         url: https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key
  87.         keyring: /etc/apt/keyrings/kubernetes-apt-keyring.gpg
  88.         state: present
  90.     # Add Kubernetes' apt repository
  91.     - name: Add Kubernetes repository to apt sources
  92.       lineinfile:
  93.         path: /etc/apt/sources.list.d/kubernetes.list
  94.         line: 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /'
  95.         create: yes
  96.       notify: Update apt cache
  98.     # Install Kubernetes packages
  99.     - name: Install Kubernetes packages (kubelet, kubeadm, kubectl) and containerd.io
  100.       apt:
  101.         name:
  102.           - kubelet
  103.           - kubeadm
  104.           - kubectl
  105.           - containerd.io
  106.         state: present
  108.     # Hold the installed packages to prevent automatic updates
  109.     - name: Hold Kubernetes packages and containerd.io
  110.       dpkg_selections:
  111.         name: "{{ item }}"
  112.         selection: hold
  113.       loop:
  114.         - kubelet
  115.         - kubeadm
  116.         - kubectl
  117.         - containerd.io
  119.   handlers:
  120.     # Handler to update apt cache when notified
  121.     - name: Update apt cache
  122.       apt:
  123.         update_cache: yes
  126. - name: Configure Kubernetes prerequisites
  127.   hosts: kubernetes
  128.   become: yes  # to run tasks that require sudo
  129.   tasks:
  130.     - name: Load Kernel Modules
  131.       copy:
  132.         content: |
  133.           overlay
  134.           br_netfilter
  135.         dest: /etc/modules-load.d/k8s.conf
  136.       notify: Load Modules
  138.     - name: Set Sysctl Parameters
  139.       copy:
  140.         content: |
  141.           net.bridge.bridge-nf-call-iptables  = 1
  142.           net.bridge.bridge-nf-call-ip6tables = 1
  143.           net.ipv4.ip_forward                 = 1
  144.         dest: /etc/sysctl.d/k8s.conf
  145.       notify: Apply Sysctl
  147.   handlers:
  148.     - name: Load Modules
  149.       modprobe:
  150.         name: "{{ item }}"
  151.         state: present
  152.       loop:
  153.         - overlay
  154.         - br_netfilter
  156.     - name: Apply Sysctl
  157.       command: sysctl --system
  160. - name: Prefetch kubernetes images
  161.   hosts: kubernetes
  162.   become: true
  163.   tasks:
  164.     - name: Get kubeadm version
  165.       command: kubeadm version -o short
  166.       register: kubeadm_version
  168.     - name: List Kubernetes images for the specific kubeadm version
  169.       command: "kubeadm config images list --kubernetes-version={{ kubeadm_version.stdout }}"
  170.       register: kubernetes_images
  172.     - name: Pull and retag Kubernetes images from Aliyun registry
  173.       block:
  174.         - name: List old images in k8s.io namespace
  175.           command: ctr -n k8s.io images list -q
  176.           register: old_images_list
  178.         - name: Pull Kubernetes image from Aliyun
  179.           command: "ctr -n k8s.io images pull registry.aliyuncs.com/google_containers/{{ item.split('/')[-1] }}"
  180.           loop: "{{ kubernetes_images.stdout_lines }}"
  181.           when: item not in old_images_list.stdout
  182.           loop_control:
  183.             label: "{{ item }}"
  185.         - name: Retag Kubernetes image
  186.           command: "ctr -n k8s.io images tag registry.aliyuncs.com/google_containers/{{ item.split('/')[-1] }} {{ item }}"
  187.           loop: "{{ kubernetes_images.stdout_lines }}"
  188.           when: item not in old_images_list.stdout
  189.           loop_control:
  190.             label: "{{ item }}"
  192.         - name: List new images in k8s.io namespace
  193.           command: ctr -n k8s.io images list -q
  194.           register: new_images_list
  196.         - name: Remove images from Aliyun registry
  197.           command: "ctr -n k8s.io images remove {{ item }}"
  198.           loop: "{{ new_images_list.stdout_lines }}"
  199.           when: item.startswith('registry.aliyuncs.com/google_containers')
  200.           loop_control:
  201.             label: "{{ item }}"
  203.         # # Optional: Remove old SHA256 tags if necessary
  204.         # - name: Remove old SHA256 tags
  205.         #   command: "ctr -n k8s.io images remove {{ item }}"
  206.         #   loop: "{{ new_images_list.stdout_lines }}"
  207.         #   when: item.startswith('sha256:')
  208.         #   loop_control:
  209.         #     label: "{{ item }}"
  212. - name: Configure containerd
  213.   hosts: kubernetes
  214.   become: true
  215.   tasks:
  217.     - name: Get Kubernetes images list
  218.       command: kubeadm config images list
  219.       register: kubernetes_images
  221.     - name: Set pause image variable
  222.       set_fact:
  223.         pause_image: "{{ kubernetes_images.stdout_lines | select('match', '^registry.k8s.io/pause:') | first }}"
  225.     - name: Generate default containerd config
  226.       command: containerd config default
  227.       register: containerd_config
  228.       changed_when: false
  230.     - name: Write containerd config to file
  231.       copy:
  232.         dest: /etc/containerd/config.toml
  233.         content: "{{ containerd_config.stdout }}"
  234.         mode: '0644'
  235.         
  236.     - name: Replace 'sandbox_image' and 'SystemdCgroup' in containerd config
  237.       lineinfile:
  238.         path: /etc/containerd/config.toml
  239.         regexp: "{{ item.regexp }}"
  240.         line: "{{ item.line }}"
  241.       loop:
  242.         - { regexp: '^\s*sandbox_image\s*=.*$', line: '    sandbox_image = "{{ pause_image }}"' }
  243.         - { regexp: 'SystemdCgroup =.*', line: '            SystemdCgroup = true' }
  245.     - name: Restart containerd service
  246.       service:
  247.         name: containerd
  248.         state: restarted
  250. - name: Initialize Kubernetes Cluster on Main Master
  251.   hosts: main
  252.   become: true
  253.   tasks:
  255.     - name: Check if IP address is already present
  256.       shell: "ip addr show dev lo | grep {{ kubernetes_vip }}"
  257.       register: ip_check
  258.       ignore_errors: yes
  259.       failed_when: false
  260.       changed_when: false
  261.     
  262.     - name: Debug print ip_check result
  263.       debug:
  264.         msg: "{{ ip_check }}"
  266.     - name: Add IP address to loopback interface
  267.       command:
  268.         cmd: "ip addr add {{ kubernetes_vip }}/32 dev lo"
  269.       when: ip_check.rc != 0
  271.     - name: Generate Kubernetes init token
  272.       shell: >
  273.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 6; 
  274.         echo -n '.'; 
  275.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 16
  276.       register: k8s_init_token
  278.     - name: Generate kubeadm config file
  279.       template:
  280.         src: resources/kubeadm-init.yaml.j2
  281.         dest: kubeadm-init.yaml
  282.       vars:
  283.         token: "{{ k8s_init_token.stdout }}"
  285.     - name: Initialize the Kubernetes cluster using kubeadm
  286.       command:
  287.         cmd: kubeadm init --v=5 --skip-phases=addon/kube-proxy --config kubeadm-init.yaml --upload-certs
  288.       register: kubeadm_init
  289.     
  290.     - name: Set fact for master join command
  291.       set_fact:
  292.         master_join_command: "{{ kubeadm_init.stdout | regex_search('kubeadm join(.*\\n)+?.*--control-plane', multiline=True) }}"
  293.         cacheable: yes
  294.       run_once: true
  296.     - name: Set fact for worker join command
  297.       set_fact:
  298.         worker_join_command: "{{ kubeadm_init.stdout | regex_search('kubeadm join(.*\\n)+?.*sha256:[a-z0-9]{64}', multiline=True) }}"
  299.         cacheable: yes
  300.       run_once: true
  302.     # - name: Create the target directory if it doesn't exist
  303.     #   file:
  304.     #     path: ~/.kube
  305.     #     state: directory
  306.     #     owner: "{{ ansible_user_id }}"
  307.     #     group: "{{ ansible_user_id }}"
  308.     #     mode: '0755'
  309.     #   when: not ansible_check_mode  # This ensures it only runs when not in check mode
  311.     # - name: Copy kube admin config to ansible user directory
  312.     #   copy:
  313.     #     src: /etc/kubernetes/admin.conf
  314.     #     dest: ~/.kube/config
  315.     #     remote_src: yes
  316.     #     owner: "{{ ansible_user_id }}"
  317.     #     group: "{{ ansible_user_id }}"
  318.     #     mode: '0644'
  320.     - name: Write master join command to .master_join_command.txt
  321.       copy:
  322.         content: "{{ master_join_command }}"
  323.         dest: ".master_join_command.txt"
  324.         mode: '0664'
  325.       delegate_to: localhost
  327.     - name: Append worker join command to .worker_join_command.txt
  328.       lineinfile:
  329.         path: ".worker_join_command.txt"
  330.         line: "{{ worker_join_command }}"
  331.         create: yes
  332.       delegate_to: localhost
  335. - name: Install cilium on Main Master
  336.   hosts: main
  337.   become: true
  338.   tasks:
  340.     - name: Ensure tar is installed (Debian/Ubuntu)
  341.       apt:
  342.         name: tar
  343.         state: present
  344.       when: ansible_os_family == "Debian"
  346.     - name: Check for Cilium binary in /usr/local/bin
  347.       stat:
  348.         path: /usr/local/bin/cilium
  349.       register: cilium_binary
  351.     - name: Transfer and Extract Cilium
  352.       unarchive:
  353.         src: resources/cilium-linux-amd64.tar.gz
  354.         dest: /usr/local/bin
  355.         remote_src: no
  356.       when: not cilium_binary.stat.exists
  358.     - name: Install cilium to the Kubernetes cluster
  359.       environment:
  360.         KUBECONFIG: /etc/kubernetes/admin.conf
  361.       command:
  362.         cmd: cilium install --version 1.14.4 --set kubeProxyReplacement=true
  363.     
  364.     - name: Wait for Kubernetes cluster to become ready
  365.       environment:
  366.         KUBECONFIG: /etc/kubernetes/admin.conf
  367.       command: kubectl get nodes
  368.       register: kubectl_output
  369.       until: kubectl_output.stdout.find("Ready") != -1
  370.       retries: 20
  371.       delay: 30
  374. - name: Join Masters to the Cluster
  375.   hosts: masters
  376.   become: true
  377.   tasks:
  378.     - name: Joining master node to the Kubernetes cluster
  379.       shell:
  380.         cmd: "{{ hostvars['k8s-main-01']['master_join_command'] }}"
  381.       ignore_errors: yes
  384. - name: Join Worker Nodes to the Cluster
  385.   hosts: workers
  386.   become: true
  387.   tasks:
  388.     - name: Joining master node to the Kubernetes cluster
  389.       shell:
  390.         cmd: "{{ hostvars['k8s-main-01']['worker_join_command'] }}"
  391.       ignore_errors: yes

用于重置 Kubernetes 节点的配置

  1. ---
  2. - name: Setup Load Balancer with IPVS and Keepalived
  3.   hosts: lvs
  4.   become: yes
  5.   tasks:
  6.     # Upgrade all installed packages to their latest versions
  7.     - name: Upgrade all installed apt packages
  8.       apt:
  9.         upgrade: 'yes'
  10.         update_cache: yes
  11.         cache_valid_time: 3600  # Cache is considered valid for 1 hour
  13.     # Install IP Virtual Server (IPVS) administration utility
  14.     - name: Install ipvsadm for IPVS management
  15.       apt:
  16.         name: ipvsadm
  17.         state: present
  19.     # Install keepalived for high availability
  20.     - name: Install Keepalived for load balancing
  21.       apt:
  22.         name: keepalived
  23.         state: present
  25.     # Deploy keepalived configuration from a Jinja2 template
  26.     - name: Deploy keepalived configuration file
  27.       template:
  28.         src: resources/keepalived.conf.j2
  29.         dest: /etc/keepalived/keepalived.conf
  31.     # Restart keepalived to apply changes
  32.     - name: Restart Keepalived service
  33.       service:
  34.         name: keepalived
  35.         state: restarted
  38. - name: Install kubernetes packages and containerd.io
  39.   hosts: kubernetes
  40.   become: yes
  41.   tasks:
  43.     # Upgrade all installed packages to their latest versions
  44.     - name: Upgrade all installed apt packages
  45.       apt:
  46.         upgrade: 'yes'
  47.         update_cache: yes
  48.         cache_valid_time: 3600  # Cache is considered valid for 1 hour
  50.     # Install required packages for Kubernetes and Docker setup
  51.     - name: Install prerequisites for Kubernetes and Docker
  52.       apt:
  53.         name:
  54.           - ca-certificates
  55.           - curl
  56.           - gnupg
  57.         update_cache: yes
  58.         cache_valid_time: 3600
  60.     # Ensure the keyring directory exists for storing GPG keys
  61.     - name: Create /etc/apt/keyrings directory for GPG keys
  62.       file:
  63.         path: /etc/apt/keyrings
  64.         state: directory
  65.         mode: '0755'
  67.     # Add Docker's official GPG key
  68.     - name: Add official Docker GPG key to keyring
  69.       apt_key:
  70.         url: https://download.docker.com/linux/ubuntu/gpg
  71.         keyring: /etc/apt/keyrings/docker.gpg
  72.         state: present
  74.     # Add Docker's apt repository
  75.     - name: Add Docker repository to apt sources
  76.       apt_repository:
  77.         # repo: "deb [arch={{ ansible_architecture }} signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
  78.         repo: "deb [signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
  79.         filename: docker
  80.         update_cache: yes
  81.       notify: Update apt cache
  83.     # Add Kubernetes' GPG key
  84.     - name: Add Kubernetes GPG key to keyring
  85.       apt_key:
  86.         url: https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key
  87.         keyring: /etc/apt/keyrings/kubernetes-apt-keyring.gpg
  88.         state: present
  90.     # Add Kubernetes' apt repository
  91.     - name: Add Kubernetes repository to apt sources
  92.       lineinfile:
  93.         path: /etc/apt/sources.list.d/kubernetes.list
  94.         line: 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /'
  95.         create: yes
  96.       notify: Update apt cache
  98.     # Install Kubernetes packages
  99.     - name: Install Kubernetes packages (kubelet, kubeadm, kubectl) and containerd.io
  100.       apt:
  101.         name:
  102.           - kubelet
  103.           - kubeadm
  104.           - kubectl
  105.           - containerd.io
  106.         state: present
  108.     # Hold the installed packages to prevent automatic updates
  109.     - name: Hold Kubernetes packages and containerd.io
  110.       dpkg_selections:
  111.         name: "{{ item }}"
  112.         selection: hold
  113.       loop:
  114.         - kubelet
  115.         - kubeadm
  116.         - kubectl
  117.         - containerd.io
  119.   handlers:
  120.     # Handler to update apt cache when notified
  121.     - name: Update apt cache
  122.       apt:
  123.         update_cache: yes
  126. - name: Configure Kubernetes prerequisites
  127.   hosts: kubernetes
  128.   become: yes  # to run tasks that require sudo
  129.   tasks:
  130.     - name: Load Kernel Modules
  131.       copy:
  132.         content: |
  133.           overlay
  134.           br_netfilter
  135.         dest: /etc/modules-load.d/k8s.conf
  136.       notify: Load Modules
  138.     - name: Set Sysctl Parameters
  139.       copy:
  140.         content: |
  141.           net.bridge.bridge-nf-call-iptables  = 1
  142.           net.bridge.bridge-nf-call-ip6tables = 1
  143.           net.ipv4.ip_forward                 = 1
  144.         dest: /etc/sysctl.d/k8s.conf
  145.       notify: Apply Sysctl
  147.   handlers:
  148.     - name: Load Modules
  149.       modprobe:
  150.         name: "{{ item }}"
  151.         state: present
  152.       loop:
  153.         - overlay
  154.         - br_netfilter
  156.     - name: Apply Sysctl
  157.       command: sysctl --system
  160. - name: Prefetch kubernetes images
  161.   hosts: kubernetes
  162.   become: true
  163.   tasks:
  164.     - name: Get kubeadm version
  165.       command: kubeadm version -o short
  166.       register: kubeadm_version
  168.     - name: List Kubernetes images for the specific kubeadm version
  169.       command: "kubeadm config images list --kubernetes-version={{ kubeadm_version.stdout }}"
  170.       register: kubernetes_images
  172.     - name: Pull and retag Kubernetes images from Aliyun registry
  173.       block:
  174.         - name: List old images in k8s.io namespace
  175.           command: ctr -n k8s.io images list -q
  176.           register: old_images_list
  178.         - name: Pull Kubernetes image from Aliyun
  179.           command: "ctr -n k8s.io images pull registry.aliyuncs.com/google_containers/{{ item.split('/')[-1] }}"
  180.           loop: "{{ kubernetes_images.stdout_lines }}"
  181.           when: item not in old_images_list.stdout
  182.           loop_control:
  183.             label: "{{ item }}"
  185.         - name: Retag Kubernetes image
  186.           command: "ctr -n k8s.io images tag registry.aliyuncs.com/google_containers/{{ item.split('/')[-1] }} {{ item }}"
  187.           loop: "{{ kubernetes_images.stdout_lines }}"
  188.           when: item not in old_images_list.stdout
  189.           loop_control:
  190.             label: "{{ item }}"
  192.         - name: List new images in k8s.io namespace
  193.           command: ctr -n k8s.io images list -q
  194.           register: new_images_list
  196.         - name: Remove images from Aliyun registry
  197.           command: "ctr -n k8s.io images remove {{ item }}"
  198.           loop: "{{ new_images_list.stdout_lines }}"
  199.           when: item.startswith('registry.aliyuncs.com/google_containers')
  200.           loop_control:
  201.             label: "{{ item }}"
  203.         # # Optional: Remove old SHA256 tags if necessary
  204.         # - name: Remove old SHA256 tags
  205.         #   command: "ctr -n k8s.io images remove {{ item }}"
  206.         #   loop: "{{ new_images_list.stdout_lines }}"
  207.         #   when: item.startswith('sha256:')
  208.         #   loop_control:
  209.         #     label: "{{ item }}"
  211.     - name: Restart containerd service
  212.       service:
  213.         name: containerd
  214.         state: restarted
  217. - name: Configure containerd
  218.   hosts: kubernetes
  219.   become: true
  220.   tasks:
  222.     - name: Get Kubernetes images list
  223.       command: kubeadm config images list
  224.       register: kubernetes_images
  226.     - name: Set pause image variable
  227.       set_fact:
  228.         pause_image: "{{ kubernetes_images.stdout_lines | select('match', '^registry.k8s.io/pause:') | first }}"
  230.     - name: Generate default containerd config
  231.       command: containerd config default
  232.       register: containerd_config
  233.       changed_when: false
  235.     - name: Write containerd config to file
  236.       copy:
  237.         dest: /etc/containerd/config.toml
  238.         content: "{{ containerd_config.stdout }}"
  239.         mode: '0644'
  240.         
  241.     - name: Replace 'sandbox_image' and 'SystemdCgroup' in containerd config
  242.       lineinfile:
  243.         path: /etc/containerd/config.toml
  244.         regexp: "{{ item.regexp }}"
  245.         line: "{{ item.line }}"
  246.       loop:
  247.         - { regexp: '^\s*sandbox_image\s*=.*$', line: '    sandbox_image = "{{ pause_image }}"' }
  248.         - { regexp: 'SystemdCgroup =.*', line: '            SystemdCgroup = true' }
  250.     - name: Restart containerd service
  251.       service:
  252.         name: containerd
  253.         state: restarted
  255. - name: Initialize Kubernetes Cluster on Main Master
  256.   hosts: main
  257.   become: true
  258.   tasks:
  260.     - name: Check if IP address is already present
  261.       shell: "ip addr show dev lo | grep {{ kubernetes_vip }}"
  262.       register: ip_check
  263.       ignore_errors: yes
  264.       failed_when: false
  265.       changed_when: false
  266.     
  267.     - name: Debug print ip_check result
  268.       debug:
  269.         msg: "{{ ip_check }}"
  271.     - name: Add IP address to loopback interface
  272.       command:
  273.         cmd: "ip addr add {{ kubernetes_vip }}/32 dev lo"
  274.       when: ip_check.rc != 0
  276.     - name: Generate Kubernetes init token
  277.       shell: >
  278.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 6; 
  279.         echo -n '.'; 
  280.         LC_CTYPE=C tr -dc 'a-z0-9' </dev/urandom | head -c 16
  281.       register: k8s_init_token
  283.     - name: Generate kubeadm config file
  284.       template:
  285.         src: resources/kubeadm-init.yaml.j2
  286.         dest: kubeadm-init.yaml
  287.       vars:
  288.         token: "{{ k8s_init_token.stdout }}"
  290.     - name: Initialize the Kubernetes cluster using kubeadm
  291.       command:
  292.         cmd: kubeadm init --v=5 --skip-phases=addon/kube-proxy --upload-certs --config kubeadm-init.yaml
  293.       register: kubeadm_init
  294.     
  295.     - name: Set fact for master join command
  296.       set_fact:
  297.         master_join_command: "{{ kubeadm_init.stdout | regex_search('kubeadm join(.*\\n)+?.*--control-plane.*', multiline=True) }}"
  298.         cacheable: yes
  299.       run_once: true
  301.     - name: Set fact for worker join command
  302.       set_fact:
  303.         worker_join_command: "{{ kubeadm_init.stdout | regex_search('kubeadm join(.*\\n)+?.*sha256:[a-z0-9]{64}', multiline=True) }}"
  304.         cacheable: yes
  305.       run_once: true
  307.     # - name: Create the target directory if it doesn't exist
  308.     #   file:
  309.     #     path: ~/.kube
  310.     #     state: directory
  311.     #     owner: "{{ ansible_user_id }}"
  312.     #     group: "{{ ansible_user_id }}"
  313.     #     mode: '0755'
  314.     #   when: not ansible_check_mode  # This ensures it only runs when not in check mode
  316.     # - name: Copy kube admin config to ansible user directory
  317.     #   copy:
  318.     #     src: /etc/kubernetes/admin.conf
  319.     #     dest: ~/.kube/config
  320.     #     remote_src: yes
  321.     #     owner: "{{ ansible_user_id }}"
  322.     #     group: "{{ ansible_user_id }}"
  323.     #     mode: '0644'
  325.     - name: Write master join command to .master_join_command.txt
  326.       copy:
  327.         content: "{{ master_join_command }}"
  328.         dest: ".master_join_command.txt"
  329.         mode: '0664'
  330.       delegate_to: localhost
  332.     - name: Append worker join command to .worker_join_command.txt
  333.       lineinfile:
  334.         path: ".worker_join_command.txt"
  335.         line: "{{ worker_join_command }}"
  336.         create: yes
  337.       delegate_to: localhost
  340. - name: Install cilium on Main Master
  341.   hosts: main
  342.   become: true
  343.   tasks:
  345.     - name: Ensure tar is installed (Debian/Ubuntu)
  346.       apt:
  347.         name: tar
  348.         state: present
  349.       when: ansible_os_family == "Debian"
  351.     - name: Check for Cilium binary in /usr/local/bin
  352.       stat:
  353.         path: /usr/local/bin/cilium
  354.       register: cilium_binary
  356.     - name: Transfer and Extract Cilium
  357.       unarchive:
  358.         src: resources/cilium-linux-amd64.tar.gz
  359.         dest: /usr/local/bin
  360.         remote_src: no
  361.       when: not cilium_binary.stat.exists
  363.     - name: Install cilium to the Kubernetes cluster
  364.       environment:
  365.         KUBECONFIG: /etc/kubernetes/admin.conf
  366.       command:
  367.         cmd: cilium install --version 1.14.4 --set kubeProxyReplacement=true
  368.     
  369.     - name: Wait for Kubernetes cluster to become ready
  370.       environment:
  371.         KUBECONFIG: /etc/kubernetes/admin.conf
  372.       command: kubectl get nodes
  373.       register: kubectl_output
  374.       until: kubectl_output.stdout.find("Ready") != -1
  375.       retries: 20
  376.       delay: 30
  379. - name: Join Masters to the Cluster
  380.   hosts: masters
  381.   become: true
  382.   tasks:
  383.     - name: Joining master node to the Kubernetes cluster
  384.       shell:
  385.         cmd: "{{ hostvars['k8s-main-01']['master_join_command'] }}"
  386.       ignore_errors: yes
  388.     - name: Wait for node to become ready
  389.       environment:
  390.         KUBECONFIG: /etc/kubernetes/admin.conf
  391.       command: kubectl get nodes
  392.       register: kubectl_output
  393.       until: kubectl_output.stdout.find("NotReady") == -1
  394.       retries: 20
  395.       delay: 30
  397.     - name: Check if IP address is already present
  398.       shell: "ip addr show dev lo | grep {{ kubernetes_vip }}"
  399.       register: ip_check
  400.       ignore_errors: yes
  401.       failed_when: false
  402.       changed_when: false
  403.     
  404.     - name: Debug print ip_check result
  405.       debug:
  406.         msg: "{{ ip_check }}"
  408.     - name: Add IP address to loopback interface
  409.       command:
  410.         cmd: "ip addr add {{ kubernetes_vip }}/32 dev lo"
  411.       when: ip_check.rc != 0
  414. - name: Join Worker Nodes to the Cluster
  415.   hosts: workers
  416.   become: true
  417.   tasks:
  418.     - name: Joining master node to the Kubernetes cluster
  419.       shell:
  420.         cmd: "{{ hostvars['k8s-main-01']['worker_join_command'] }}"
  421.       ignore_errors: yes
  423.     - name: Check if IP address is already present
  424.       shell: "ip addr show dev lo | grep {{ kubernetes_vip }}"
  425.       register: ip_check
  426.       ignore_errors: yes
  427.       failed_when: false
  428.       changed_when: false
  429.     
  430.     - name: Debug print ip_check result
  431.       debug:
  432.         msg: "{{ ip_check }}"
  434.     - name: Add IP address to loopback interface
  435.       command:
  436.         cmd: "ip addr add {{ kubernetes_vip }}/32 dev lo"
  437.       when: ip_check.rc != 0

原文链接：https://www.cnblogs.com/javennie/p/setup-k8s-cluster-via-anisble.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Kubernetes热门文章

  • 【Azure K8S | AKS】在AKS集群中创建 PVC(PersistentVolumeClaim)和 PV(PersistentVolume) 示例
  • 无法访问k8s.gcr.io下载镜像问题解决办法
  • Kubernetes(k8s)访问控制：身份认证
  • Prometheus服务发现之kubernetes_sd_config
  • Arthas排查Kubernetes中应用频繁挂掉重启异常
  • Kubernetes客户端认证——基于CA证书的双向认证方式
  • k8s中kubeconfig的配置以及使用详解
  • Centos7 安装部署 Kubernetes(k8s) 高可用集群
  • Kubernetes 证书详解
  • 案例分享-full gc导致k8s pod重启

    • Kubernetes推荐文章

  • 修改kubeadm证书过期时间及更新k8s集群证书
  • 探究kubernetes 探针参数periodSeconds和timeoutSeconds
  • 从 Helm 到 Operator：Kubernetes应用管理的进化
  • 深入理解Prometheus: Kubernetes环境中的监控实践
  • k8s集群搭建及对一些组件的简单理解（二）
  • K8s高可用集群二进制部署-V1.20
  • Kubernetes容器生命周期 —— 钩子函数详解（postStart、preStop）
  • Kubernetes 审计（Auditing）
  • Thanos解码：打造企业级云原生监控解决方案
  • k8s使用rbd作为存储

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号