1.基础环境:Centos7.9,kubernetes:v1.21.5
node-1@112(master):docker,containerd,harbornginx(80),git,etcdnode-2@109(master/worker):docker,containerd,ingress_nginx(80),etcd,glusterfsnode-3@107(worker):docker,containerd,harbor(80),etcd,glusterfsnode-4@106(worker):docker,containerd,harbor(80),glusterfsnode-6@121(单体/应用):docker,containerd,nginx(80/前端),consul,nacos,yapi
node-1@112(master):docker,containerd,harbornginx(80),git,etcd
node-2@109(master/worker):docker,containerd,ingress_nginx(80),etcd,glusterfs
node-3@107(worker):docker,containerd,harbor(80),etcd,glusterfs
node-4@106(worker):docker,containerd,harbor(80),glusterfs
node-6@121(单体/应用):docker,containerd,nginx(80/前端),consul,nacos,yapi
2.公有代码仓库,gitee,配置网络钩子
3.私有镜像仓库,harbor双主复制,node-3/node-4,地址:myhub.com
参考:https://github.com/goharbor/harbor
4.持久化存储 ,gluster-kubernetes ,node-2/node-3/node-4,heketi使用Ingress四层代理
参考:https://github.com/gluster/gluster-kubernetes
? http://docs.kubernetes.org.cn/803.html#Glusterfs
5.jenkins,k8s Deployment方式部署,推荐本地部署
参考:https://www.jenkins.io/doc/book/installing/kubernetes/
(1)Local Persistent Volume 替换为 GlusterFs(基于Local Persistent Volume搭建的分布式文件系统),存储配置信息以及workspace
volume.yaml:
apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: jenkins-glusterfs-storage-classprovisioner: kubernetes.io/glusterfsparameters: # 这里heketi使用ingress四层代理 # 192.168.0.109为ingress-nginx-controller监听地址 resturl: "http://192.168.0.109:30001" restauthenabled: "true" restuser: "admin" restuserkey: "admin123"---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: jenkins-pv-claim namespace: devops-toolsspec: storageClassName: jenkins-glusterfs-storage-class accessModes: - ReadWriteOnce resources: requests: storage: 3Gi---# agent workspace-volumeapiVersion: v1kind: PersistentVolumeClaimmetadata: name: jenkins-agent-pv-claim namespace: devops-toolsspec: storageClassName: jenkins-glusterfs-storage-class accessModes: - ReadWriteOnce resources: requests: storage: 30Gi
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: jenkins-glusterfs-storage-class
provisioner: kubernetes.io/glusterfs
parameters:
# 这里heketi使用ingress四层代理
# 192.168.0.109为ingress-nginx-controller监听地址
resturl: "http://192.168.0.109:30001"
restauthenabled: "true"
restuser: "admin"
restuserkey: "admin123"
---
apiVersion: v1
kind: PersistentVolumeClaim
name: jenkins-pv-claim
namespace: devops-tools
spec:
storageClassName: jenkins-glusterfs-storage-class
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
# agent workspace-volume
name: jenkins-agent-pv-claim
storage: 30Gi
(2)jenkins-server服务暴露方式NodePort替换为Ingress七层代理+四层代理
$ kubectl patch cm tcp-services -n ingress-nginx --patch='{"data": {"32000": "devops-tools/jenkins-service:80"}}'
service.yaml
apiVersion: v1kind: Servicemetadata: name: jenkins-service namespace: devops-tools annotations: prometheus.io/scrape: 'true' prometheus.io/path: / prometheus.io/port: '8080'spec: selector: app: jenkins-server ports: - name: web port: 80 targetPort: 8080 - name: agent port: 50000 targetPort: 50000---apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: jenkins-service namespace: devops-toolsspec: rules: - host: myjenkins.com http: paths: - path: / pathType: Prefix backend: service: name: jenkins-service port: name: web
kind: Service
name: jenkins-service
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8080'
selector:
app: jenkins-server
ports:
- name: web
port: 80
targetPort: 8080
- name: agent
port: 50000
targetPort: 50000
apiVersion: networking.k8s.io/v1
kind: Ingress
rules:
- host: myjenkins.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
port:
name: web
(3)sa的集群角色中增加对集群中deployment,ingress资源的所有操作权限
serviceAccount.yaml
apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: jenkins-adminrules: #"":core - apiGroups: [""] resources: ["*"] verbs: ["*"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["*"] - apiGroups: ["networking.k8s.io"] resources: ["ingresses"] verbs: ["*"] ---apiVersion: v1kind: ServiceAccountmetadata: name: jenkins-admin namespace: devops-tools---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: jenkins-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins-adminsubjects:- kind: ServiceAccount name: jenkins-admin namespace: devops-tools
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
name: jenkins-admin
#"":core
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["apps"]
resources: ["deployments"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
kind: ServiceAccount
kind: ClusterRoleBinding
roleRef:
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
(4)安装插件:
? 参考: https://plugins.jenkins.io/kubernetes/
? https://plugins.jenkins.io/kubernetes-cli/
? https://gitee.com/help/articles/4193#article-header0
? (a)kubernetes:用于和k8s交互,配置创建agent pod
? (b)Kubernetes Cli:提供agent使用kubectl与k8s交互环境
? (c)Gitee:用于自动触发构建
? (d)其他:系统开始建议安装的插件都装一下
(5)工作原理:
? (a)pod中的容器挂载infra容器的Namespace,共享Mount Namespace,工作目录/home/jenkins/agent都挂载workspace-volume数据卷,git拉下来的项目文件都是可见的
? (b)agent主容器:jnlp默认自动生成,jenkins+kubernetes的cicd整个流程是定义sidecar容器,通过jenkins及其插件,手动一步步的实现sidecar容器对项目文件的具体操作
(6)编译环境准备-build项目时:
? golang与alpine镜像对应版本:https://hub.docker.com/_/golang
? (a)用的是go项目,需要go进程install项目,使用容器化方式,让k8s去管理
? (b)kubernetes 插件pod模板添加容器,编译环境镜像: golang:1.19.1-alpine,上传到harbor
? (c)容器启动进程:/bin/sh -c cat,防止容器进程关闭
? (d)其他语言项目,使用对应语言环境镜像编译即可
(7)agent pod系统用户权限处理-docker创建镜像时:
? (a)由于每个节点都装有docker,root用户登录,所以使用本地数据卷挂载方式,agent pod 的User Namespace相同才能执行即系统用户需要root执行权限才能使用当前节点docker
? (b)uid(用户id)和gid(用户组id),securityContext设置针对controller的uid和gid,agent无效,controller和agent默认是jenkins和1000
? (c)可以在kubernetes 插件pod模板中修改,使用root用户,uid=0,gid=0,或者Raw YAML for the Pod覆盖主容器
? (d)也可以在构建shell脚本中将jenkins用户添加到root用户组中
? (e)制作镜像的基础镜像:alpine:3.16,go语言在云原生优势体现,不需要虚拟机,直接编译成对应系统的二进制文件运行,镜像可以瘦身很小
? (f)写shell脚本定时查询节点创建的项目镜像并清理一定时间段的镜像,以DaemonSet部署方式亲和到每个worker节点,master节点存在NoSchedule污点,或者镜像上传后立即删除
? (g)镜像上传harbor,第一次登陆需要docker login用户注册认证,本地保存在~/.docker/config.json,每个节点不一定都存在,使用cm方式挂载到该路径
$ kubectl create cm docker-hub-credential --from-file=config=config.json -n devops-tools
? (h)也可以使用docker in docker 好处是生命周期和agent pod一致,缺点是问题排查难度变大
注:使用docker的多阶段编译制作镜像,可以将(4)的go build过程放在(5)build image中构建,一个build阶段完成
(8)agent pod与k8s api server交互-项目部署到k8s集群时:
? (a)使用kubectl,当前只存在master节点中,需要在worker节点使用,为了不污染worker节点,使用容器化方式,让k8s去管理
? (b)k8s集群外kubectl与k8s api server交互,客户端证书TLS双向认证,需要useraccount配置信息,客户端保存在~/.kube/config,可以将凭证以cm/pvc方式挂载进去,api server地址一般设置为集群地址或本地代理地址进行解耦
? (c)k8s集群内pod可以通过ServiceAccount与k8s api server进行交互,ServiceAccount会创建对应secret自动挂载到pod文件系统中:/var/run/secrets/kubernetes.io/serviceaccount ,包含有token信息,为通信凭证
? (d)配置的ServiceAccount具有对集群中core所有资源,deployment,ingress的所有操作权限,获取jenkins-admin的token:
$ kubectl -ndevops-tools describe secret $(kubectl -ndevops-tools get secret | grep jenkins-admin | awk '{print $1}')
? (e)可以使用Kubernetes Cli插件进行配置token与k8s api server集群内部地址,自动配置并使用上下文,实现集群内kubectl 与k8s api server交互
? (f)kubernetes 插件pod模板添加容器,镜像: registry.cn-shanghai.aliyuncs.com/mydlq/kubectl:1.15.3,上传到harbor
? (g)容器启动进程:/bin/sh -c cat,防止容器进程关闭
(9)pipline测试脚本:
def label = "golang1.19.1"def credential = "global-kubernetes-credential"timeout(time: 900, unit: 'SECONDS') { podTemplate(label: label,cloud: 'kubernetes' ){ node (label) { stage('Git阶段'){ sh '''echo "开始拉取代码" echo "拉取代码完成"''' } stage('Build阶段'){ container('golang') { sh '''echo "构建开始" id go version echo "构建完成"''' } } stage('Docker阶段'){ sh '''echo "创建镜像开始" id docker images echo "创建镜像完成"''' } stage('Kubernetes 阶段'){ container('kubectl') { withKubeConfig([credentialsId: credential,serverUrl: "https://kubernetes.default.svc.cluster.local"]) { sh '''echo "k8s部署开始" id kubectl get nodes kubectl get pods echo "k8s部署完成"''' } } } } }}
def label = "golang1.19.1"
def credential = "global-kubernetes-credential"
timeout(time: 900, unit: 'SECONDS') {
podTemplate(label: label,cloud: 'kubernetes' ){
node (label) {
stage('Git阶段'){
sh '''echo "开始拉取代码"
echo "拉取代码完成"'''
}
stage('Build阶段'){
container('golang') {
sh '''echo "构建开始"
id
go version
echo "构建完成"'''
stage('Docker阶段'){
sh '''echo "创建镜像开始"
docker images
echo "创建镜像完成"'''
stage('Kubernetes 阶段'){
container('kubectl') {
withKubeConfig([credentialsId: credential,serverUrl: "https://kubernetes.default.svc.cluster.local"]) {
sh '''echo "k8s部署开始"
kubectl get nodes
kubectl get pods
echo "k8s部署完成"'''
注:jenkins全局变量会自动注入到容器环境变量中,每个容器都可以获取使用
原文链接:https://www.cnblogs.com/jn-shao/p/16949424.html
本站QQ群:前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728