编程

手动平衡 Kafka Learder 倾斜

1 配置需要reblance的topic列表

vim topics.conf

1
2
3
{"topics": [{"topic": "test"}],
 "version":1
}

2 进行reblance操作

注意休怪zkServer,和brokerIdList的值
vim reblance.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/bin/bash
zkServer="hadoop002:2181"
brokerIdList=144,145,146,1031,1032,1033
echo "zkConf:"  $zkServer
echo "brokerList:" $brokerIdList
echo "---------start generate reblance conf---------"
content=`kafka-reassign-partitions --zookeeper $zkServer --topics-to-move-json-file topics.conf  --broker-list $brokerIdList --generate`
content=`echo $content | awk -F 'Proposed partition reassignment configuration' '{print $2}'`
echo $content
echo $content > ressgin_topic.conf

if [ ! -d "log" ]; then
  mkdir log
fi

echo $content >> ./log/reblance.log
echo "" >> ./log/reblance.log
echo "---------end generate reblance conf---------"
#start reblance
echo "---------start reblance---------"
kafka-reassign-partitions --zookeeper $zkServer --reassignment-json-file ressgin_topic.conf --execute

3 查看reblance的进度

1
kafka-reassign-partitions --zookeeper $zkServer --reassignment-json-file ressgin_topic.conf --verify

from https://blog.csdn.net/qq_18838991/article/details/53394013

Share