经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 数据库/运维 » Redis » 查看文章
k8s实战案例之部署redis单机和redis cluster - Linux-1874
来源：cnblogs　　作者：Linux-1874　　时间：2023/6/7 10:13:03　　对本文有异议

1、在k8s上部署redis单机

1.1、redis简介

redis是一款基于BSD协议，开源的非关系型数据库（nosql数据库），作者是意大利开发者Salvatore Sanfilippo在2009年发布，使用C语言编写；redis是基于内存存储，而且是目前比较流行的键值数据库（key-value database），它提供将内存通过网络远程共享的一种服务，提供类似功能的还有memcache，但相比 memcache，redis 还提供了易扩展、高性能、具备数据持久性等功能。主要的应用场景有session共享，常用于web集群中的tomcat或PHP中多web服务器的session共享；消息队列，ELK的日志缓存，部分业务的订阅发布系统；计数器，常用于访问排行榜，商品浏览数等和次数相关的数值统计场景；缓存，常用于数据查询、电商网站商品信息、新闻内容等；相对memcache，redis支持数据的持久化，可以将内存的数据保存在磁盘中，重启redis服务或者服务器之后可以从备份文件中恢复数据到内存继续使用；

1.2、PV/PVC 及 Redis 单机

由于redis的数据（主要是redis快照）都存放在存储系统中，即便redis pod挂掉，对应数据都不会丢；因为在k8s上部署redis单机，redis pod挂了，k8s会将对应pod重建，重建时会把对应pvc挂载至pod中，加载快照，从而使得redis的数据不被pod的挂掉而丢数据；

1.3、构建redis镜像

  1. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# ll
  2. total 1784
  3. drwxr-xr-x  2 root root    4096 Jun  5 15:22 ./
  4. drwxr-xr-x 11 root root    4096 Aug  9  2022 ../
  5. -rw-r--r--  1 root root     717 Jun  5 15:20 Dockerfile
  6. -rwxr-xr-x  1 root root     235 Jun  5 15:21 build-command.sh*
  7. -rw-r--r--  1 root root 1740967 Jun 22  2021 redis-4.0.14.tar.gz
  8. -rw-r--r--  1 root root   58783 Jun 22  2021 redis.conf
  9. -rwxr-xr-x  1 root root      84 Jun  5 15:21 run_redis.sh*
  10. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# cat Dockerfile 
  11. #Redis Image
  12. # 导入自定义centos基础镜像
  13. FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 
  14. # 添加redis源码包至/usr/local/src
  15. ADD redis-4.0.14.tar.gz /usr/local/src
  16. # 编译安装redis
  17. RUN ln -sv /usr/local/src/redis-4.0.14 /usr/local/redis && cd /usr/local/redis && make && cp src/redis-cli /usr/sbin/ && cp src/redis-server  /usr/sbin/ && mkdir -pv /data/redis-data 
  18. # 添加redis配置文件
  19. ADD redis.conf /usr/local/redis/redis.conf 
  20. # 暴露redis服务端口
  21. EXPOSE 6379
  23. #ADD run_redis.sh /usr/local/redis/run_redis.sh
  24. #CMD ["/usr/local/redis/run_redis.sh"]
  25. # 添加启动脚本
  26. ADD run_redis.sh /usr/local/redis/entrypoint.sh
  27. # 启动redis
  28. ENTRYPOINT ["/usr/local/redis/entrypoint.sh"]
  29. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# cat build-command.sh 
  30. #!/bin/bash
  31. TAG=$1
  32. #docker build -t harbor.ik8s.cc/magedu/redis:${TAG} .
  33. #sleep 3
  34. #docker push  harbor.ik8s.cc/magedu/redis:${TAG}
  36. nerdctl build -t  harbor.ik8s.cc/magedu/redis:${TAG} .
  37. nerdctl push harbor.ik8s.cc/magedu/redis:${TAG}
  38. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# cat run_redis.sh 
  39. #!/bin/bash
  40. # Redis启动命令
  41. /usr/sbin/redis-server /usr/local/redis/redis.conf
  42. # 使用tail -f 在pod内部构建守护进程
  43. tail -f  /etc/hosts
  44. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis# grep -v '^#\|^$' redis.conf 
  45. bind 0.0.0.0
  46. protected-mode yes
  47. port 6379
  48. tcp-backlog 511
  49. timeout 0
  50. tcp-keepalive 300
  51. daemonize yes
  52. supervised no
  53. pidfile /var/run/redis_6379.pid
  54. loglevel notice
  55. logfile ""
  56. databases 16
  57. always-show-logo yes
  58. save 900 1
  59. save 5 1
  60. save 300 10
  61. save 60 10000
  62. stop-writes-on-bgsave-error no
  63. rdbcompression yes
  64. rdbchecksum yes
  65. dbfilename dump.rdb
  66. dir /data/redis-data
  67. slave-serve-stale-data yes
  68. slave-read-only yes
  69. repl-diskless-sync no
  70. repl-diskless-sync-delay 5
  71. repl-disable-tcp-nodelay no
  72. slave-priority 100
  73. requirepass 123456
  74. lazyfree-lazy-eviction no
  75. lazyfree-lazy-expire no
  76. lazyfree-lazy-server-del no
  77. slave-lazy-flush no
  78. appendonly no
  79. appendfilename "appendonly.aof"
  80. appendfsync everysec
  81. no-appendfsync-on-rewrite no
  82. auto-aof-rewrite-percentage 100
  83. auto-aof-rewrite-min-size 64mb
  84. aof-load-truncated yes
  85. aof-use-rdb-preamble no
  86. lua-time-limit 5000
  87. slowlog-log-slower-than 10000
  88. slowlog-max-len 128
  89. latency-monitor-threshold 0
  90. notify-keyspace-events ""
  91. hash-max-ziplist-entries 512
  92. hash-max-ziplist-value 64
  93. list-max-ziplist-size -2
  94. list-compress-depth 0
  95. set-max-intset-entries 512
  96. zset-max-ziplist-entries 128
  97. zset-max-ziplist-value 64
  98. hll-sparse-max-bytes 3000
  99. activerehashing yes
  100. client-output-buffer-limit normal 0 0 0
  101. client-output-buffer-limit slave 256mb 64mb 60
  102. client-output-buffer-limit pubsub 32mb 8mb 60
  103. hz 10
  104. aof-rewrite-incremental-fsync yes
  105. root@k8s-master01:~/k8s-data/dockerfile/web/magedu/redis#

1.3.1、验证rdis镜像是否上传至harbor?

1.4、测试redis 镜像

1.4.1、验证将redis镜像运行为容器，看看是否正常运行?

1.4.2、远程连接redis，看看是否可正常连接？

能够将redis镜像运行为容器，并且能够通过远程主机连接至redis进行数据读写，说明我们构建的reids镜像没有问题；

1.5、创建PV和PVC

1.5.1、在nfs服务器上准备redis数据存储目录

  1. root@harbor:~# mkdir -pv /data/k8sdata/magedu/redis-datadir-1
  2. mkdir: created directory '/data/k8sdata/magedu/redis-datadir-1'
  3. root@harbor:~# cat /etc/exports
  4. # /etc/exports: the access control list for filesystems which may be exported
  5. #               to NFS clients.  See exports(5).
  6. #
  7. # Example for NFSv2 and NFSv3:
  8. # /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
  9. #
  10. # Example for NFSv4:
  11. # /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
  12. # /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
  13. #
  14. /data/k8sdata/kuboard *(rw,no_root_squash)
  15. /data/volumes *(rw,no_root_squash)
  16. /pod-vol *(rw,no_root_squash)
  17. /data/k8sdata/myserver *(rw,no_root_squash)
  18. /data/k8sdata/mysite *(rw,no_root_squash)
  20. /data/k8sdata/magedu/images *(rw,no_root_squash)
  21. /data/k8sdata/magedu/static *(rw,no_root_squash)
  24. /data/k8sdata/magedu/zookeeper-datadir-1 *(rw,no_root_squash)
  25. /data/k8sdata/magedu/zookeeper-datadir-2 *(rw,no_root_squash)
  26. /data/k8sdata/magedu/zookeeper-datadir-3 *(rw,no_root_squash)
  29. /data/k8sdata/magedu/redis-datadir-1 *(rw,no_root_squash) 
  31. root@harbor:~# exportfs -av
  32. exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/kuboard".
  33.   Assuming default behaviour ('no_subtree_check').
  34.   NOTE: this default has changed since nfs-utils version 1.0.x
  36. exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/volumes".
  37.   Assuming default behaviour ('no_subtree_check').
  38.   NOTE: this default has changed since nfs-utils version 1.0.x
  40. exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/pod-vol".
  41.   Assuming default behaviour ('no_subtree_check').
  42.   NOTE: this default has changed since nfs-utils version 1.0.x
  44. exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/myserver".
  45.   Assuming default behaviour ('no_subtree_check').
  46.   NOTE: this default has changed since nfs-utils version 1.0.x
  48. exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/mysite".
  49.   Assuming default behaviour ('no_subtree_check').
  50.   NOTE: this default has changed since nfs-utils version 1.0.x
  52. exportfs: /etc/exports [7]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/images".
  53.   Assuming default behaviour ('no_subtree_check').
  54.   NOTE: this default has changed since nfs-utils version 1.0.x
  56. exportfs: /etc/exports [8]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/static".
  57.   Assuming default behaviour ('no_subtree_check').
  58.   NOTE: this default has changed since nfs-utils version 1.0.x
  60. exportfs: /etc/exports [11]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-1".
  61.   Assuming default behaviour ('no_subtree_check').
  62.   NOTE: this default has changed since nfs-utils version 1.0.x
  64. exportfs: /etc/exports [12]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-2".
  65.   Assuming default behaviour ('no_subtree_check').
  66.   NOTE: this default has changed since nfs-utils version 1.0.x
  68. exportfs: /etc/exports [13]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-3".
  69.   Assuming default behaviour ('no_subtree_check').
  70.   NOTE: this default has changed since nfs-utils version 1.0.x
  72. exportfs: /etc/exports [16]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis-datadir-1".
  73.   Assuming default behaviour ('no_subtree_check').
  74.   NOTE: this default has changed since nfs-utils version 1.0.x
  76. exporting *:/data/k8sdata/magedu/redis-datadir-1
  77. exporting *:/data/k8sdata/magedu/zookeeper-datadir-3
  78. exporting *:/data/k8sdata/magedu/zookeeper-datadir-2
  79. exporting *:/data/k8sdata/magedu/zookeeper-datadir-1
  80. exporting *:/data/k8sdata/magedu/static
  81. exporting *:/data/k8sdata/magedu/images
  82. exporting *:/data/k8sdata/mysite
  83. exporting *:/data/k8sdata/myserver
  84. exporting *:/pod-vol
  85. exporting *:/data/volumes
  86. exporting *:/data/k8sdata/kuboard
  87. root@harbor:~#

1.5.2、创建pv

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv# cat redis-persistentvolume.yaml     
  2. ---
  3. apiVersion: v1
  4. kind: PersistentVolume
  5. metadata:
  6.   name: redis-datadir-pv-1
  7. spec:
  8.   capacity:
  9.     storage: 10Gi
  10.   accessModes:
  11.     - ReadWriteOnce
  12.   nfs:
  13.     path: /data/k8sdata/magedu/redis-datadir-1 
  14.     server: 192.168.0.42
  15. root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv#

1.5.3、创建pvc

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv# cat redis-persistentvolumeclaim.yaml 
  2. ---
  3. apiVersion: v1
  4. kind: PersistentVolumeClaim
  5. metadata:
  6.   name: redis-datadir-pvc-1 
  7.   namespace: magedu
  8. spec:
  9.   volumeName: redis-datadir-pv-1 
  10.   accessModes:
  11.     - ReadWriteOnce
  12.   resources:
  13.     requests:
  14.       storage: 10Gi
  15. root@k8s-master01:~/k8s-data/yaml/magedu/redis/pv#

1.6、部署redis服务

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis# cat redis.yaml
  2. kind: Deployment
  3. #apiVersion: extensions/v1beta1
  4. apiVersion: apps/v1
  5. metadata:
  6.   labels:
  7.     app: devops-redis 
  8.   name: deploy-devops-redis
  9.   namespace: magedu
  10. spec:
  11.   replicas: 1 
  12.   selector:
  13.     matchLabels:
  14.       app: devops-redis
  15.   template:
  16.     metadata:
  17.       labels:
  18.         app: devops-redis
  19.     spec:
  20.       containers:
  21.         - name: redis-container
  22.           image: harbor.ik8s.cc/magedu/redis:v4.0.14 
  23.           imagePullPolicy: Always
  24.           volumeMounts:
  25.           - mountPath: "/data/redis-data/"
  26.             name: redis-datadir
  27.       volumes:
  28.         - name: redis-datadir
  29.           persistentVolumeClaim:
  30.             claimName: redis-datadir-pvc-1 
  32. ---
  33. kind: Service
  34. apiVersion: v1
  35. metadata:
  36.   labels:
  37.     app: devops-redis
  38.   name: srv-devops-redis
  39.   namespace: magedu
  40. spec:
  41.   type: NodePort
  42.   ports:
  43.   - name: http
  44.     port: 6379 
  45.     targetPort: 6379
  46.     nodePort: 36379 
  47.   selector:
  48.     app: devops-redis
  49.   sessionAffinity: ClientIP
  50.   sessionAffinityConfig:
  51.     clientIP:
  52.       timeoutSeconds: 10800
  53. root@k8s-master01:~/k8s-data/yaml/magedu/redis#

上述报错说我们的服务端口超出范围，这是因为我们在初始化k8s集群时指定的服务端口范围；

1.6.1、修改nodeport端口范围

编辑/etc/systemd/system/kube-apiserver.service,将其--service-node-port-range选项指定的值修改即可；其他两个master节点也需要修改哦

1.6.2、重载kube-apiserver.service,重启kube-apiserver

  1. root@k8s-master01:~# systemctl daemon-reload                 
  2. root@k8s-master01:~# systemctl restart kube-apiserver.service
  3. root@k8s-master01:~#

再次部署redis

1.7、验证redis数据读写

1.7.1、连接k8s任意节点的36376端口，测试redis读写数据

1.8、验证redis pod 重建对应数据是否丢失？

1.8.1、查看redis快照文件是否存储到存储上呢？

  1. root@harbor:~# ll /data/k8sdata/magedu/redis-datadir-1
  2. total 12
  3. drwxr-xr-x 2 root root 4096 Jun  5 16:29 ./
  4. drwxr-xr-x 8 root root 4096 Jun  5 15:53 ../
  5. -rw-r--r-- 1 root root  116 Jun  5 16:29 dump.rdb
  6. root@harbor:~#

可以看到刚才我们向redis写入数据，对应redis在规定时间内发现key的变化就做了快照，因为redis数据目录时通过pv/pvc挂载的nfs，所以我们在nfs对应目录里时可以正常看到这个快照文件的；

1.8.2、删除redis pod 等待k8s重建redis pod

1.8.3、验证重建后的redis pod数据

可以看到k8s重建后的redis pod 还保留着原有pod的数据；这说明k8s重建时挂载了前一个pod的pvc;

2、在k8s上部署redis集群

2.1、PV/PVC及Redis Cluster-StatefulSet

redis cluster相比redis单机要稍微复杂一点，我们也是通过pv/pvc将redis cluster数据存放在存储系统中，不同于redis单机，redis cluster对存入的数据会做crc16计算，然后和16384做取模计算，得出一个数字，这个数字就是存入redis cluster的一个槽位；即redis cluster将16384个槽位，平均分配给集群所有master节点，每个master节点存放整个集群数据的一部分；这样一来就存在一个问题，如果master宕机，那么对应槽位的数据也就不可用，为了防止master单点故障，我们还需要对master做高可用，即专门用一个slave节点对master做备份，master宕机的情况下，对应slave会接管master继续向集群提供服务，从而实现redis cluster master的高可用；如上图所示，我们使用3主3从的redis cluster，redis0,1,2为master，那么3，4，5就对应为0，1，2的slave，负责备份各自对应的master的数据；这六个pod都是通过k8s集群的pv/pvc将数据存放在存储系统中；

2.2、创建PV

2.2.1、在nfs上准备redis cluster 数据目录

  1. root@harbor:~# mkdir -pv /data/k8sdata/magedu/redis{0,1,2,3,4,5}
  2. mkdir: created directory '/data/k8sdata/magedu/redis0'
  3. mkdir: created directory '/data/k8sdata/magedu/redis1'
  4. mkdir: created directory '/data/k8sdata/magedu/redis2'
  5. mkdir: created directory '/data/k8sdata/magedu/redis3'
  6. mkdir: created directory '/data/k8sdata/magedu/redis4'
  7. mkdir: created directory '/data/k8sdata/magedu/redis5'
  8. root@harbor:~# tail -6 /etc/exports 
  9. /data/k8sdata/magedu/redis0 *(rw,no_root_squash)
  10. /data/k8sdata/magedu/redis1 *(rw,no_root_squash)
  11. /data/k8sdata/magedu/redis2 *(rw,no_root_squash)
  12. /data/k8sdata/magedu/redis3 *(rw,no_root_squash)
  13. /data/k8sdata/magedu/redis4 *(rw,no_root_squash)
  14. /data/k8sdata/magedu/redis5 *(rw,no_root_squash)
  15. root@harbor:~# exportfs  -av
  16. exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/kuboard".
  17.   Assuming default behaviour ('no_subtree_check').
  18.   NOTE: this default has changed since nfs-utils version 1.0.x
  20. exportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/volumes".
  21.   Assuming default behaviour ('no_subtree_check').
  22.   NOTE: this default has changed since nfs-utils version 1.0.x
  24. exportfs: /etc/exports [3]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/pod-vol".
  25.   Assuming default behaviour ('no_subtree_check').
  26.   NOTE: this default has changed since nfs-utils version 1.0.x
  28. exportfs: /etc/exports [4]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/myserver".
  29.   Assuming default behaviour ('no_subtree_check').
  30.   NOTE: this default has changed since nfs-utils version 1.0.x
  32. exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/mysite".
  33.   Assuming default behaviour ('no_subtree_check').
  34.   NOTE: this default has changed since nfs-utils version 1.0.x
  36. exportfs: /etc/exports [7]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/images".
  37.   Assuming default behaviour ('no_subtree_check').
  38.   NOTE: this default has changed since nfs-utils version 1.0.x
  40. exportfs: /etc/exports [8]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/static".
  41.   Assuming default behaviour ('no_subtree_check').
  42.   NOTE: this default has changed since nfs-utils version 1.0.x
  44. exportfs: /etc/exports [11]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-1".
  45.   Assuming default behaviour ('no_subtree_check').
  46.   NOTE: this default has changed since nfs-utils version 1.0.x
  48. exportfs: /etc/exports [12]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-2".
  49.   Assuming default behaviour ('no_subtree_check').
  50.   NOTE: this default has changed since nfs-utils version 1.0.x
  52. exportfs: /etc/exports [13]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/zookeeper-datadir-3".
  53.   Assuming default behaviour ('no_subtree_check').
  54.   NOTE: this default has changed since nfs-utils version 1.0.x
  56. exportfs: /etc/exports [16]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis-datadir-1".
  57.   Assuming default behaviour ('no_subtree_check').
  58.   NOTE: this default has changed since nfs-utils version 1.0.x
  60. exportfs: /etc/exports [18]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis0".
  61.   Assuming default behaviour ('no_subtree_check').
  62.   NOTE: this default has changed since nfs-utils version 1.0.x
  64. exportfs: /etc/exports [19]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis1".
  65.   Assuming default behaviour ('no_subtree_check').
  66.   NOTE: this default has changed since nfs-utils version 1.0.x
  68. exportfs: /etc/exports [20]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis2".
  69.   Assuming default behaviour ('no_subtree_check').
  70.   NOTE: this default has changed since nfs-utils version 1.0.x
  72. exportfs: /etc/exports [21]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis3".
  73.   Assuming default behaviour ('no_subtree_check').
  74.   NOTE: this default has changed since nfs-utils version 1.0.x
  76. exportfs: /etc/exports [22]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis4".
  77.   Assuming default behaviour ('no_subtree_check').
  78.   NOTE: this default has changed since nfs-utils version 1.0.x
  80. exportfs: /etc/exports [23]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/k8sdata/magedu/redis5".
  81.   Assuming default behaviour ('no_subtree_check').
  82.   NOTE: this default has changed since nfs-utils version 1.0.x
  84. exporting *:/data/k8sdata/magedu/redis5
  85. exporting *:/data/k8sdata/magedu/redis4
  86. exporting *:/data/k8sdata/magedu/redis3
  87. exporting *:/data/k8sdata/magedu/redis2
  88. exporting *:/data/k8sdata/magedu/redis1
  89. exporting *:/data/k8sdata/magedu/redis0
  90. exporting *:/data/k8sdata/magedu/redis-datadir-1
  91. exporting *:/data/k8sdata/magedu/zookeeper-datadir-3
  92. exporting *:/data/k8sdata/magedu/zookeeper-datadir-2
  93. exporting *:/data/k8sdata/magedu/zookeeper-datadir-1
  94. exporting *:/data/k8sdata/magedu/static
  95. exporting *:/data/k8sdata/magedu/images
  96. exporting *:/data/k8sdata/mysite
  97. exporting *:/data/k8sdata/myserver
  98. exporting *:/pod-vol
  99. exporting *:/data/volumes
  100. exporting *:/data/k8sdata/kuboard
  101. root@harbor:~#

2.2.2、创建pv

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat pv/redis-cluster-pv.yaml 
  2. apiVersion: v1
  3. kind: PersistentVolume
  4. metadata:
  5.   name: redis-cluster-pv0
  6. spec:
  7.   capacity:
  8.     storage: 5Gi
  9.   accessModes:
  10.     - ReadWriteOnce
  11.   nfs:
  12.     server: 192.168.0.42
  13.     path: /data/k8sdata/magedu/redis0 
  15. ---
  16. apiVersion: v1
  17. kind: PersistentVolume
  18. metadata:
  19.   name: redis-cluster-pv1
  20. spec:
  21.   capacity:
  22.     storage: 5Gi
  23.   accessModes:
  24.     - ReadWriteOnce
  25.   nfs:
  26.     server: 192.168.0.42
  27.     path: /data/k8sdata/magedu/redis1 
  29. ---
  30. apiVersion: v1
  31. kind: PersistentVolume
  32. metadata:
  33.   name: redis-cluster-pv2
  34. spec:
  35.   capacity:
  36.     storage: 5Gi
  37.   accessModes:
  38.     - ReadWriteOnce
  39.   nfs:
  40.     server: 192.168.0.42
  41.     path: /data/k8sdata/magedu/redis2 
  43. ---
  44. apiVersion: v1
  45. kind: PersistentVolume
  46. metadata:
  47.   name: redis-cluster-pv3
  48. spec:
  49.   capacity:
  50.     storage: 5Gi
  51.   accessModes:
  52.     - ReadWriteOnce
  53.   nfs:
  54.     server: 192.168.0.42
  55.     path: /data/k8sdata/magedu/redis3 
  57. ---
  58. apiVersion: v1
  59. kind: PersistentVolume
  60. metadata:
  61.   name: redis-cluster-pv4
  62. spec:
  63.   capacity:
  64.     storage: 5Gi
  65.   accessModes:
  66.     - ReadWriteOnce
  67.   nfs:
  68.     server: 192.168.0.42
  69.     path: /data/k8sdata/magedu/redis4 
  71. ---
  72. apiVersion: v1
  73. kind: PersistentVolume
  74. metadata:
  75.   name: redis-cluster-pv5
  76. spec:
  77.   capacity:
  78.     storage: 5Gi
  79.   accessModes:
  80.     - ReadWriteOnce
  81.   nfs:
  82.     server: 192.168.0.42
  83.     path: /data/k8sdata/magedu/redis5 
  84. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

2.3、部署redis cluster

2.3.1、基于redis.conf文件创建configmap

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat redis.conf 
  2. appendonly yes
  3. cluster-enabled yes
  4. cluster-config-file /var/lib/redis/nodes.conf
  5. cluster-node-timeout 5000
  6. dir /var/lib/redis
  7. port 6379
  8. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

2.3.2、创建configmap

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl create cm redis-conf --from-file=./redis.conf -n magedu 
  2. configmap/redis-conf created
  3. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl get cm -n magedu 
  4. NAME               DATA   AGE
  5. kube-root-ca.crt   1      35h
  6. redis-conf         1      6s
  7. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

2.3.3、验证configmap

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl describe cm redis-conf -n magedu 
  2. Name:         redis-conf
  3. Namespace:    magedu
  4. Labels:       <none>
  5. Annotations:  <none>
  7. Data
  8. ====
  9. redis.conf:
  10. ----
  11. appendonly yes
  12. cluster-enabled yes
  13. cluster-config-file /var/lib/redis/nodes.conf
  14. cluster-node-timeout 5000
  15. dir /var/lib/redis
  16. port 6379
  19. BinaryData
  20. ====
  22. Events:  <none>
  23. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

2.3.4、部署redis cluster

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat redis.yaml 
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: redis
  6.   namespace: magedu
  7.   labels:
  8.     app: redis
  9. spec:
  10.   selector:
  11.     app: redis
  12.     appCluster: redis-cluster
  13.   ports:
  14.   - name: redis
  15.     port: 6379
  16.   clusterIP: None
  17.   
  18. ---
  19. apiVersion: v1
  20. kind: Service
  21. metadata:
  22.   name: redis-access
  23.   namespace: magedu
  24.   labels:
  25.     app: redis
  26. spec:
  27.   type: NodePort
  28.   selector:
  29.     app: redis
  30.     appCluster: redis-cluster
  31.   ports:
  32.   - name: redis-access
  33.     protocol: TCP
  34.     port: 6379
  35.     targetPort: 6379
  36.     nodePort: 36379
  38. ---
  39. apiVersion: apps/v1
  40. kind: StatefulSet
  41. metadata:
  42.   name: redis
  43.   namespace: magedu
  44. spec:
  45.   serviceName: redis
  46.   replicas: 6
  47.   selector:
  48.     matchLabels:
  49.       app: redis
  50.       appCluster: redis-cluster
  51.   template:
  52.     metadata:
  53.       labels:
  54.         app: redis
  55.         appCluster: redis-cluster
  56.     spec:
  57.       terminationGracePeriodSeconds: 20
  58.       affinity:
  59.         podAntiAffinity:
  60.           preferredDuringSchedulingIgnoredDuringExecution:
  61.           - weight: 100
  62.             podAffinityTerm:
  63.               labelSelector:
  64.                 matchExpressions:
  65.                 - key: app
  66.                   operator: In
  67.                   values:
  68.                   - redis
  69.               topologyKey: kubernetes.io/hostname
  70.       containers:
  71.       - name: redis
  72.         image: redis:4.0.14
  73.         command:
  74.           - "redis-server"
  75.         args:
  76.           - "/etc/redis/redis.conf"
  77.           - "--protected-mode"
  78.           - "no"
  79.         resources:
  80.           requests:
  81.             cpu: "500m"
  82.             memory: "500Mi"
  83.         ports:
  84.         - containerPort: 6379
  85.           name: redis
  86.           protocol: TCP
  87.         - containerPort: 16379
  88.           name: cluster
  89.           protocol: TCP
  90.         volumeMounts:
  91.         - name: conf
  92.           mountPath: /etc/redis
  93.         - name: data
  94.           mountPath: /var/lib/redis
  95.       volumes:
  96.       - name: conf
  97.         configMap:
  98.           name: redis-conf
  99.           items:
  100.           - key: redis.conf
  101.             path: redis.conf
  102.   volumeClaimTemplates:
  103.   - metadata:
  104.       name: data
  105.       namespace: magedu
  106.     spec:
  107.       accessModes: [ "ReadWriteOnce" ]
  108.       resources:
  109.         requests:
  110.           storage: 5Gi
  111. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

上述配置清单，主要用sts控制器创建了6个pod副本，每个副本都使用configmap中的配置文件作为redis配置文件，使用pvc模板指定pod在k8s上自动关联pv，并在magedu名称空间创建pvc,即只要k8s上有空余的pv，对应pod就会在magedu这个名称空间按pvc模板信息创建pvc；当然我们可以使用存储类自动创建pvc，也可以提前创建好pvc，一般情况下使用sts控制器，我们可以使用pvc模板的方式来指定pod自动创建pvc(前提是k8s有足够的pv可用)；

应用配置清单部署redis cluster

使用sts控制器创建pod，pod名称是sts控制器的名称-id，使用pvc模板创建pvc的名称为pvc模板名称-pod名称,即pvc模板名-sts控制器名-id;

2.4、初始化redis cluster

2.4.1、在k8s上创建临时容器,安装redis cluster 初始化工具

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl run -it ubuntu1804 --image=ubuntu:18.04 --restart=Never -n magedu bash
  2. If you don't see a command prompt, try pressing enter.
  3. root@ubuntu1804:/#
  4. root@ubuntu1804:/# apt update
  5. # 安装必要工具
  6. root@ubuntu1804:/# apt install python2.7 python-pip redis-tools dnsutils iputils-ping net-tools
  7. # 更新pip
  8. root@ubuntu1804:/# pip install --upgrade pip
  9. # 使用pip安装redis cluster初始化工具redis-trib
  10. root@ubuntu1804:/# pip install redis-trib==0.5.1
  11. root@ubuntu1804:/#

2.4.2、初始化redis cluster

  1. root@ubuntu1804:/# redis-trib.py create  `dig +short redis-0.redis.magedu.svc.cluster.local`:6379  `dig +short redis-1.redis.magedu.svc.cluster.local`:6379  `dig +short redis-2.redis.magedu.svc.cluster.local`:6379

在k8s上我们使用sts创建pod，对应pod的名称是固定不变的，所以我们初始化redis 集群就直接使用redis pod名称就可以直接解析到对应pod的IP地址；在传统虚拟机或物理机上初始化redis集群，我们可用直接使用IP地址，原因是物理机或虚拟机IP地址是固定的，在k8s上pod的IP地址是不固定的；

2.4.3、给master指定slave

  • 给redis-0指定slave为 redis-3
  1. root@ubuntu1804:/# redis-trib.py replicate  --master-addr `dig +short redis-0.redis.magedu.svc.cluster.local`:6379  --slave-addr `dig +short redis-3.redis.magedu.svc.cluster.local`:6379

  • 给redis-1指定slave为 redis-4
  1. root@ubuntu1804:/# redis-trib.py replicate  --master-addr `dig +short redis-1.redis.magedu.svc.cluster.local`:6379  --slave-addr `dig +short redis-4.redis.magedu.svc.cluster.local`:6379

  • 给redis-2指定slave为 redis-5
  1. root@ubuntu1804:/# redis-trib.py replicate --master-addr `dig +short redis-2.redis.magedu.svc.cluster.local`:6379 --slave-addr `dig +short redis-5.redis.magedu.svc.cluster.local`:6379

2.5、验证redis cluster状态

2.5.1、进入redis cluster 任意pod 查看集群信息

2.5.2、查看集群节点

集群节点信息中记录了master节点id和slave id，其中slave后面会对应master的id，表示该slave备份对应master数据；

2.5.3、查看当前节点信息

  1. 127.0.0.1:6379> info
  2. # Server
  3. redis_version:4.0.14
  4. redis_git_sha1:00000000
  5. redis_git_dirty:0
  6. redis_build_id:165c932261a105d7
  7. redis_mode:cluster
  8. os:Linux 5.15.0-73-generic x86_64
  9. arch_bits:64
  10. multiplexing_api:epoll
  11. atomicvar_api:atomic-builtin
  12. gcc_version:8.3.0
  13. process_id:1
  14. run_id:aa8ef00d843b4f622374dbb643cf27cdbd4d5ba3
  15. tcp_port:6379
  16. uptime_in_seconds:4303
  17. uptime_in_days:0
  18. hz:10
  19. lru_clock:8272053
  20. executable:/data/redis-server
  21. config_file:/etc/redis/redis.conf
  23. # Clients
  24. connected_clients:1
  25. client_longest_output_list:0
  26. client_biggest_input_buf:0
  27. blocked_clients:0
  29. # Memory
  30. used_memory:2642336
  31. used_memory_human:2.52M
  32. used_memory_rss:5353472
  33. used_memory_rss_human:5.11M
  34. used_memory_peak:2682248
  35. used_memory_peak_human:2.56M
  36. used_memory_peak_perc:98.51%
  37. used_memory_overhead:2559936
  38. used_memory_startup:1444856
  39. used_memory_dataset:82400
  40. used_memory_dataset_perc:6.88%
  41. total_system_memory:16740012032
  42. total_system_memory_human:15.59G
  43. used_memory_lua:37888
  44. used_memory_lua_human:37.00K
  45. maxmemory:0
  46. maxmemory_human:0B
  47. maxmemory_policy:noeviction
  48. mem_fragmentation_ratio:2.03
  49. mem_allocator:jemalloc-4.0.3
  50. active_defrag_running:0
  51. lazyfree_pending_objects:0
  53. # Persistence
  54. loading:0
  55. rdb_changes_since_last_save:0
  56. rdb_bgsave_in_progress:0
  57. rdb_last_save_time:1685992849
  58. rdb_last_bgsave_status:ok
  59. rdb_last_bgsave_time_sec:0
  60. rdb_current_bgsave_time_sec:-1
  61. rdb_last_cow_size:245760
  62. aof_enabled:1
  63. aof_rewrite_in_progress:0
  64. aof_rewrite_scheduled:0
  65. aof_last_rewrite_time_sec:-1
  66. aof_current_rewrite_time_sec:-1
  67. aof_last_bgrewrite_status:ok
  68. aof_last_write_status:ok
  69. aof_last_cow_size:0
  70. aof_current_size:0
  71. aof_base_size:0
  72. aof_pending_rewrite:0
  73. aof_buffer_length:0
  74. aof_rewrite_buffer_length:0
  75. aof_pending_bio_fsync:0
  76. aof_delayed_fsync:0
  78. # Stats
  79. total_connections_received:7
  80. total_commands_processed:17223
  81. instantaneous_ops_per_sec:1
  82. total_net_input_bytes:1530962
  83. total_net_output_bytes:108793
  84. instantaneous_input_kbps:0.04
  85. instantaneous_output_kbps:0.00
  86. rejected_connections:0
  87. sync_full:1
  88. sync_partial_ok:0
  89. sync_partial_err:1
  90. expired_keys:0
  91. expired_stale_perc:0.00
  92. expired_time_cap_reached_count:0
  93. evicted_keys:0
  94. keyspace_hits:0
  95. keyspace_misses:0
  96. pubsub_channels:0
  97. pubsub_patterns:0
  98. latest_fork_usec:853
  99. migrate_cached_sockets:0
  100. slave_expires_tracked_keys:0
  101. active_defrag_hits:0
  102. active_defrag_misses:0
  103. active_defrag_key_hits:0
  104. active_defrag_key_misses:0
  106. # Replication
  107. role:master
  108. connected_slaves:1
  109. slave0:ip=10.200.155.175,port=6379,state=online,offset=1120,lag=1
  110. master_replid:60381a28fee40b44c409e53eeef49215a9d3b0ff
  111. master_replid2:0000000000000000000000000000000000000000
  112. master_repl_offset:1120
  113. second_repl_offset:-1
  114. repl_backlog_active:1
  115. repl_backlog_size:1048576
  116. repl_backlog_first_byte_offset:1
  117. repl_backlog_histlen:1120
  119. # CPU
  120. used_cpu_sys:12.50
  121. used_cpu_user:7.51
  122. used_cpu_sys_children:0.01
  123. used_cpu_user_children:0.00
  125. # Cluster
  126. cluster_enabled:1
  128. # Keyspace
  129. 127.0.0.1:6379>

2.5.4、验证redis cluster读写数据是否正常？

2.5.4.1、手动连接redis cluster 进行数据读写

手动连接redis 集群master节点进行数据读写，存在一个问题就是当我们写入的key经过crc16计算对16384取模后，对应槽位可能不在当前节点，redis它会告诉我们该key该在哪里去写；从上面的截图可用看到，现在redis cluster 是可用正常读写数据的

2.5.4.2、使用python脚本连接redis cluster 进行数据读写

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# cat redis-client-test.py
  2. #!/usr/bin/env python
  3. #coding:utf-8
  4. #Author:Zhang ShiJie
  5. #python 2.7/3.8
  6. #pip install redis-py-cluster
  8. import sys,time
  9. from rediscluster import RedisCluster
  10. def init_redis():
  11.     startup_nodes = [
  12.         {'host': '192.168.0.34', 'port': 36379},
  13.         {'host': '192.168.0.35', 'port': 36379},
  14.         {'host': '192.168.0.36', 'port': 36379},
  15.         {'host': '192.168.0.34', 'port': 36379},
  16.         {'host': '192.168.0.35', 'port': 36379},
  17.         {'host': '192.168.0.36', 'port': 36379},
  18.     ]
  19.     try:
  20.         conn = RedisCluster(startup_nodes=startup_nodes,
  21.                             # 有密码要加上密码哦
  22.                             decode_responses=True, password='')
  23.         print('连接成功！！！！！1', conn)
  24.         #conn.set("key-cluster","value-cluster")
  25.         for i in range(100):
  26.             conn.set("key%s" % i, "value%s" % i)
  27.             time.sleep(0.1)
  28.             data = conn.get("key%s" % i)
  29.             print(data)
  31.         #return conn
  33.     except Exception as e:
  34.         print("connect error ", str(e))
  35.         sys.exit(1)
  37. init_redis()
  38. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

运行脚本，向redis cluster 写入数据

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# python redis-client-test.py
  2. Traceback (most recent call last):
  3.   File "/root/k8s-data/yaml/magedu/redis-cluster/redis-client-test.py", line 8, in <module>
  4.     from rediscluster import RedisCluster
  5. ModuleNotFoundError: No module named 'rediscluster'
  6. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

这里提示没有找到rediscluster模块，解决办法就是通过pip安装redis-py-cluster模块即可；

安装redis-py-cluster模块

运行脚本连接redis cluster进行数据读写

连接redis pod，验证数据是否正常写入？


从上面的截图可用看到三个reids cluster master pod各自都存放了一部分key,并非全部；说明刚才我们用python脚本把数据正常写入了redis cluster;

验证在slave 节点是否可用正常读取数据？

从上面的截图可以了解到在slave节点是不可以读取数据；

到slave对应的master节点读取数据

上述验证说明了redis cluster 只有master可以读写数据，slave只是对master数据做备份，不可以在slave上读写数据；

2.6、验证验证redis cluster高可用

2.6.1、在k8s node节点将redis:4.0.14镜像上传至本地harbor

  • 修改镜像tag
  1. root@k8s-node01:~# nerdctl tag redis:4.0.14 harbor.ik8s.cc/redis-cluster/redis:4.0.14
  • 上传redis镜像至本地harbor
  1. root@k8s-node01:~# nerdctl push harbor.ik8s.cc/redis-cluster/redis:4.0.14
  2. INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:1ae9e0f790001af4b9f83a2b3d79c593c6f3e9a881b754a99527536259fb6625) 
  3. WARN[0000] skipping verifying HTTPS certs for "harbor.ik8s.cc" 
  4. index-sha256:1ae9e0f790001af4b9f83a2b3d79c593c6f3e9a881b754a99527536259fb6625:    done           |++++++++++++++++++++++++++++++++++++++| 
  5. manifest-sha256:5bd4fe08813b057df2ae55003a75c39d80a4aea9f1a0fbc0fbd7024edf555786: done           |++++++++++++++++++++++++++++++++++++++| 
  6. config-sha256:191c4017dcdd3370f871a4c6e7e1d55c7d9abed2bebf3005fb3e7d12161262b8:   done           |++++++++++++++++++++++++++++++++++++++| 
  7. elapsed: 1.4 s                                                                    total:  8.5 Ki (6.1 KiB/s)                                       
  8. root@k8s-node01:~#

2.6.2、修改redis cluster部署清单镜像和镜像拉取策略

修改镜像为本地harbor镜像和拉取策略是方便我们测试redis cluster的高可用；

2.6.3、重新apply redis cluster部署清单

  1. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster# kubectl apply -f redis.yaml
  2. service/redis unchanged
  3. service/redis-access unchanged
  4. statefulset.apps/redis configured
  5. root@k8s-master01:~/k8s-data/yaml/magedu/redis-cluster#

这里相当于给redis cluster更新，他们之间的集群关系还存在，因为集群关系配置都保存在远端存储之上；

  • 验证pod是否都正常running?
  • 验证集群状态和集群关系

不同于之前，这里rdis-0变成了slave ,redis-3变成了master；从上面的截图我们也发现，在k8s上部署redis cluster pod重建以后（IP地址发生变化），对应集群关系不会发生变化；对应master和salve一对关系始终只是再对应的master和salve两个pod中切换，这其实就是高可用；

2.6.4、停掉本地harbor，删除redis master pod，看看对应slave是否会提升为master？

  • 停止harbor服务
  1. root@harbor:~# systemctl stop harbor
  • 删除redis-3,看看redis-0是否会提升为master？

可用看到我们把redis-3删除（相当于master宕机）以后，对应slave提升为master了；

2.6.5、恢复harbor服务，看看对应redis-3恢复会议后是否还是redis-0的slave呢？

  • 恢复harbor服务
  • 验证redis-3pod是否恢复？

再次删除redis-3以后，对应pod正常被重建，并处于running状态；

  • 验证redis-3的主从关系、

可以看到redis-3恢复以后，对应自动加入集群成为redis-0的slave;

原文链接：https://www.cnblogs.com/qiuhom-1874/p/17459116.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Redis热门文章

  • Redis-复制
  • Redis
  • 关于SpringBoot整合redis使用Lettuce客户端超时问题
  • RedisTemplate常用方法封装
  • RedisTemplate中opsForValue和opsForList方法的使用详解
  • Redis7.0部署集群的实现步骤
  • Java&nbsp;Redis&nbsp;Redisson配置教程详解
  • Go语言开发redis封装及简单使用详解
  • SpringBoot Redis批量存取数据的操作
  • .NetCore下基于FreeRedis实现的Redis6.0客户端缓存之缓存键条件优雅过滤

    • Redis推荐文章

  • Redis系列：使用Stream实现消息队列 （图文总结+Go案例） - Hello-Brand
  • 记一次 redis 事件注册不当导致的内存泄露
  • Java Redis多限流
  • 【Azure Developer】一个复制Redis Key到另一个Redis服务的工具(redis_copy_net8)
  • Redis 高阶应用
  • 【长文】带你搞明白Redis
  • StackExchange.Redis跑起来，为什么这么溜？
  • 【进阶篇】基于 Redis 实现分布式锁的全过程
  • Higress 基于自定义插件访问 Redis
  • 美团二面：如何保证Redis与Mysql双写一致性？连续两个面试问到了！

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号