经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 大数据/云/AI » Apache Kafka » 查看文章
如何使用kafka增加topic的备份数量，让业务更上一层楼
来源：cnblogs　　作者：腾讯云+社区　　时间：2019/1/9 9:30:21　　对本文有异议

本文由云+社区发表

一、困难点

建立topic的时候，可以通过指定参数 --replication-factor 设置备份数量。但是，一旦完成建立topic，则无法通过kafka-topic.sh 或者 命令修改replica数量。

二、解决办法

? 实际上，我们可以考虑一种 “另类” 的办法：可以利用 kafka-reassign-partitions.sh 命令对所有分区进行重新分布，在做分区重新分布的时候，通过增加每个分区的replica备份数量来达到目的。

? 本文将介绍如何利用 kafka-reassign-partitions.sh 命令增加topic的备份数量。

注意：以下命令使用到的topic名称、zookeeper的ip和port，需要读者替换成为实际集群的参数。

（假设kafka集群有4个broker，id分别为：1001,1002,1003,1004）

2.1、获取当前topic的所有分区分布在broker的情况

  1. [root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
  2. Topic:ranger_audits     PartitionCount:10       ReplicationFactor:1     Configs:
  3.         Topic: ranger_audits    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001
  4.         Topic: ranger_audits    Partition: 1    Leader: 1002    Replicas: 1002  Isr: 1002
  5.         Topic: ranger_audits    Partition: 2    Leader: 1001    Replicas: 1001  Isr: 1001
  6.         Topic: ranger_audits    Partition: 3    Leader: 1002    Replicas: 1002  Isr: 1002
  7.         Topic: ranger_audits    Partition: 4    Leader: 1001    Replicas: 1001  Isr: 1001
  8.         Topic: ranger_audits    Partition: 5    Leader: 1002    Replicas: 1002  Isr: 1002
  9.         Topic: ranger_audits    Partition: 6    Leader: 1001    Replicas: 1001  Isr: 1001
  10.         Topic: ranger_audits    Partition: 7    Leader: 1002    Replicas: 1002  Isr: 1002
  11.         Topic: ranger_audits    Partition: 8    Leader: 1001    Replicas: 1001  Isr: 1001
  12.         Topic: ranger_audits    Partition: 9    Leader: 1002    Replicas: 1002  Isr: 1002

可以看出，ranger_audits 这个topic有10个分区，每个分区只有一个feplica备份，分布在1001和1002两台broker上面。

下面我们需要将ranger_audits 的每个分区数据都增加到2个replica备份，且分布到4个broker上面。

2.2、创建增加replica备份数量的配置文件

（注意：尽量保持topic的原有每个分区的主备份不变化。因此，配置文件的每个分区的第一个broker保持不变。）

  1. [root@tbds bin]# vim ../config/increase-replication-factor.json
  2. {"version":1,
  3. "partitions":[
  4. {"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},
  5. {"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},
  6. {"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},
  7. {"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},
  8. {"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},
  9. {"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},
  10. {"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},
  11. {"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},
  12. {"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},
  13. {"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}
  14. ]}

上面的配置文件说明，我们将topic的每个分区都增加了一个replica，且保持每个分区原有的主备份所在broker不变化，将每个分区新增的replica备份数据放到到1003和1004两个broker上面。

2.3、开始执行增加分区

  1. [root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --execute
  2. Current partition replica assignment
  3. {"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":0,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}
  5. Save this to use as the --reassignment-json-file option during rollback
  6. Successfully started reassignment of partitions 
  7. {"version":1,"partitions":[{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}

2.4、查看执行进度

  1. [root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --verify
  2. Status of partition reassignment:
  3. Reassignment of partition [ranger_audits,0] completed successfully
  4. Reassignment of partition [ranger_audits,8] completed successfully
  5. Reassignment of partition [ranger_audits,5] completed successfully
  6. Reassignment of partition [ranger_audits,2] completed successfully
  7. Reassignment of partition [ranger_audits,9] completed successfully
  8. Reassignment of partition [ranger_audits,1] completed successfully
  9. Reassignment of partition [ranger_audits,3] completed successfully
  10. Reassignment of partition [ranger_audits,4] completed successfully
  11. Reassignment of partition [ranger_audits,7] completed successfully
  12. Reassignment of partition [ranger_audits,6] completed successfully

上面显示增加分区操作成功

2.5、再次查看topic的情况

  1. [root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
  2. Topic:ranger_audits     PartitionCount:10       ReplicationFactor:2     Configs:
  3.         Topic: ranger_audits    Partition: 0    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
  4.         Topic: ranger_audits    Partition: 1    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
  5.         Topic: ranger_audits    Partition: 2    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
  6.         Topic: ranger_audits    Partition: 3    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
  7.         Topic: ranger_audits    Partition: 4    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
  8.         Topic: ranger_audits    Partition: 5    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
  9.         Topic: ranger_audits    Partition: 6    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
  10.         Topic: ranger_audits    Partition: 7    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
  11.         Topic: ranger_audits    Partition: 8    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
  12.         Topic: ranger_audits    Partition: 9    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004

从上面可以看出，备份数量增加成功

三、进一步思考

? 利用上述介绍的办法，除了可以用来增加topic的备份数量之外，还能够处理以下几个场景：

1、对topic的所有分区数据进行整体迁移。怎么理解呢？假如集群有N个broker，后来新扩容M个broker。由于新扩容的broker磁盘都是空的，原有的broker磁盘占用都很满。那么我们可以利用上述方法，将存储在原有N个broker的某些topic整体搬迁到新扩容的M个broker，进而实现kafka集群的整体数据均衡。

? 具体使用方法就是：通过编写2.2章节的配置文件，将topic的所有分区都配置到新的M个broker上面去，再执行excute，即可完成topic的所有分区数据整体迁移到新扩容的M个broker节点。

*2、broker坏掉的情况。*导致某些topic的某些分区的replica数量减少，可以利用kafka-reassign-partitions.sh增加replica；

*3、kafka 某些broker磁盘占用很满，某些磁盘占用又很少。*可以利用kafka-reassign-partitions.sh迁移某些topic的分区数据到磁盘占用少的broker，实现数据均衡；

*4、kafka集群扩容。*需要把原来broker的topic数据整体迁移到新的broker，合理利用新扩容的broker，实现负载均衡。

此文已由作者授权腾讯云+社区在各渠道发布

获取更多新鲜技术干货，可以关注我们腾讯云技术社区-云加社区官方号及知乎机构号

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Apache Kafka热门文章

  • Kafka监控工具汇总
  • Kraft模式下Kafka脚本的使用
  • Kafka web console安装
  • kafka_2.11-2.0.0_安装部署
  • ELK日志方案--使用Filebeat收集日志并输出到Kafka
  • Kafka相关内容总结（存储和性能）
  • 一次flume exec source采集日志到kafka因为单条日志数据非常大同步失败的踩坑带来的思考
  • Kafka使用jmxtrans+influxdb+grafana监控JMX指标
  • Micronaut 微服务中使用 Kafka
  • 什么是Kafka？

    • Apache Kafka推荐文章

  • 异源数据同步 → DataX 为什么要支持 kafka？
  • 解锁强强组合: 使用 Kafka + ClickHouse 快速搭建流数据实时处理平台(DoubleCloud 博客)
  • Kafka 线上性能调优
  • Kafka原理剖析之「位点提交」
  • Kafka 社区KIP-382中文译文（MirrorMaker2/集群复制/高可用/灾难恢复）
  • kafka源码阅读之MacBook Pro M1搭建Kafka2.7版本源码运行环境
  • Kafka干货之「零拷贝」
  • 扫盲Kafka？看这一篇就够了！
  • 从一次Kafka宕机说起（JVM hang）
  • 从Kafka中学习高性能系统如何设计

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号