经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置:技术经验 » 大数据/云/AI » Apache Kafka » 查看文章
kafka修改topic副本数
来源:cnblogs  作者:锋锋2019  时间:2019/7/19 8:35:53  对本文有异议

 工作案例:

    大数据开发用系统脚本自动在kafka建topic,检查后才发现副本数只有1个,存在数据丢失的风险。需要立刻把副本数改为3个。

 

  开始干活,首先想到的是下面的命令:    

${BIN_PATH}/kafka-topics.sh --zookeeper ${ZOOKEEPER_ADDR} --topic $1 --partitions $2 --alter
${BIN_PATH}/kafka-topics.sh --zookeeper ${ZOOKEEPER_ADDR} --topic $1 --replication-factor $2 --alter

   运行时修改partition成功,修改副本数失败,错误如下:

:2181 --topic test --replication-factor 2 --alter
Option "[replication-factor]" can't be used with option "[alter]"

  马上度娘,有很多方法都是用json文件的方式修改,试验了下都不成功,直接去官网查权威

https://kafka.apache.org/21/documentation.html#basic_ops_increase_replication_factor

  思路没错,原来是网上的json文件写不对。解决方法:

1/ 建一个测试topic

data1/kafka/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER_ADDR} --create --partitions 5 --replication-factor 3 --topic test

2/ 察看test的详细信息

Topic:test PartitionCount:5 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 21 Replicas: 21,22,31 Isr: 21,22,31
Topic: test Partition: 1 Leader: 22 Replicas: 21,22,31 Isr: 22,31,21
Topic: test Partition: 2 Leader: 31 Replicas: 21,22,31 Isr: 31,21,22
Topic: test Partition: 3 Leader: 21 Replicas: 21,22,31 Isr: 21,31,22
Topic: test Partition: 4 Leader: 21 Replicas: 21,22,31 Isr: 31,22,21

3/ 准备修改副本的json文件,格式可以找个json生成器查看。[]中是集群的broker-list

[root@zk-kafka-1 ~]# cat increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"test","partition":0,"replicas":[21,22,31,11]},
{"topic":"test","partition":1,"replicas":[21,22,31,11]},
{"topic":"test","partition":2,"replicas":[21,22,31,11]},
{"topic":"test","partition":3,"replicas":[21,22,31,11]},
{"topic":"test","partition":4,"replicas":[21,22,31,11]}
]

}

4/ 执行修改命令,把副本数修改为4

[root@zk-kafka-1 ~]# /data1/kafka/bin/kafka-reassign-partitions.sh --zookeeper ${ZOOKEEPER_ADDR}  --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":1,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":4,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":3,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":2,"replicas":[21,22,31],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

5/ 看到successfully,查看现在test的副本数已经修改为4

Topic:test PartitionCount:5 ReplicationFactor:4 Configs:
Topic: test Partition: 0 Leader: 21 Replicas: 21,22,31,11 Isr: 21,22,31,11
Topic: test Partition: 1 Leader: 22 Replicas: 21,22,31,11 Isr: 22,31,21,11
Topic: test Partition: 2 Leader: 31 Replicas: 21,22,31,11 Isr: 31,21,22,11
Topic: test Partition: 3 Leader: 21 Replicas: 21,22,31,11 Isr: 21,31,22,11
Topic: test Partition: 4 Leader: 21 Replicas: 21,22,31,11 Isr: 31,22,21,11

 

原文链接:http://www.cnblogs.com/hongfeng2019/p/11210229.html

 友情链接:直通硅谷  点职佳  北美留学生论坛

本站QQ群:前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

W3xue 的所有内容仅供测试,对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接(目前9元/月)请联系QQ:27243702 沸活量
皖ICP备17017327号-2 皖公网安备34020702000426号