(I) environment introduction
1. Server related settings:
1,kafka1:172.20.67.52
2,kafka2:172.20.67.56
3,kafka3:172.20.67.57
2. Three ports required by zookeeper
1,2181
2,2888
3,3888
Functions of three ports:
2181: providing services to clients
2888: use of machine communication in the cluster (the Leader listens to this port)
3888: election leader use
3. Port required by kafka
9092: Port required by kafka
(II) installation and configuration of kafka cluster
(1) installation and related configuration of kafka1172.20.67.52
1. Go to the official website to download the corresponding version( https://kafka.apache.org/downloads ) I have downloaded and directly used kafka_version 2.11-0.11.0.3
2. Unzip and put it in the corresponding directory
[root@localhost opt]# tar xf kafka_2.11-0.11.0.3.tgz
[root@localhost opt]# mv kafka_2.11-0.11.0.3 /usr/local/kafka
3. Configuration of zookeeper
[root@localhost kafka]# vim /usr/local/kafka/config/zookeeper.properties
dataDir=/data/zookeeper
clientPort=2181
maxClientCnxns=0
tickTime=2000
initLimit=20
syncLimit=10
server.1=172.20.67.52:2888:3888
server.2=172.20.67.56:2888:3888
server.3=172.20.67.57:2888:3888
4. kafka configuration
[root@localhost kafka]# vim /usr/local/kafka/config/server.properties broker.id=1 host.name=172.20.67.52 listeners=PLAINTEXT://172.20.67.52:9092 log.dirs=/data/kafka-logs log.retention.check.interval.ms=300000 log.retention.hours=1 log.segment.bytes=1073741824 num.io.threads=4 default.replication.factor=2 num.network.threads=3 num.partitions=30 num.recovery.threads.per.data.dir=1 port=9092 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 zookeeper.connect=172.20.67.52:2181,172.20.67.56:2181,172.20.67.57:2181 zookeeper.connection.timeout.ms=10000 fetch.message.max.bytes=52428700 replica.fetch.max.bytes=52428700
5. Add to / etc/rc.local at startup
[root@localhost config]# vim /etc/rc.local cd /var/tmp;nohup /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties & cd /tmp;nohup /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties & touch /var/lock/subsys/local
6. Create the data directory / data/zookeeper and file / data/zookeeper/myid of zookeeper in turn. At this time, we need to create a new myid file under the directory of dataDir on the corresponding zookeeper server, and write in the corresponding 1, 2 and 3.
[root@localhost config]# mkdir -p /data/zookeeper
[root@localhost config]# vim /data/zookeeper/myid
1
7. Start the zookeeper of kafka1, kafka2 and kafka3 in turn
[root@localhost config]# /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &
8. Then start Kafka services of kafka1, kafka2 and kafka3 in the cluster
[root@localhost config]# cd /tmp;nohup /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
9. Finally, follow the steps of 2, 3, 4, 5, 6, 7 and 8 to install and start the configuration files of zookeeper and Kafka in kafka2 (172.20.67.56) and kafka3 (172.20.67.57).
10. View the ports started by the cluster
kafka1 [root@Kafka1 ~]# netstat -lntp|grep java tcp6 0 0 172.20.67.52:3888 :::* LISTEN 20184/java tcp6 0 0 127.0.0.1:9600 :::* LISTEN 20143/java tcp6 0 0 172.20.67.52:9092 :::* LISTEN 20649/java tcp6 0 0 :::35300 :::* LISTEN 20649/java tcp6 0 0 :::2181 :::* LISTEN 20184/java tcp6 0 0 :::46471 :::* LISTEN 20184/java kafka2 [root@Kafka2 ~]# netstat -lntp|grep java tcp6 0 0 172.20.67.56:3888 :::* LISTEN 30298/java tcp6 0 0 :::33395 :::* LISTEN 30298/java tcp6 0 0 :::43031 :::* LISTEN 30573/java tcp6 0 0 172.20.67.56:9092 :::* LISTEN 30573/java tcp6 0 0 :::2181 :::* LISTEN 30298/java kafka3 [root@Kafka3 ~]# netstat -lntp|grep java tcp6 0 0 172.20.67.57:3888 :::* LISTEN 26313/java tcp6 0 0 :::44194 :::* LISTEN 26313/java tcp6 0 0 172.20.67.57:9092 :::* LISTEN 26883/java tcp6 0 0 :::2181 :::* LISTEN 26313/java tcp6 0 0 172.20.67.57:2888 :::* LISTEN 26313/java tcp6 0 0 :::36041 :::* LISTEN 26883/java