Startup of kafka
Method:

1. The zookeeper service provided by kafka

//The default startup mode is foreground startup. When the current command line window is closed, the process will be closed
# ./bin/zookeeper-server-start.sh ./config/zookeeper.properties
//Background start command
# nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &
or
# ./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties
//The "- daemon" parameter represents starting kafka server as a daemon

nohup ./bin/kafka-server-start.sh ./config/server.properties & //Start kafka

//Check whether the port number is listening
lsof -i:2181  //zookeeper port
lsof -i:9092  //kafka port

• Common commands:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test //Create topic

./bin/kafka-topics.sh --describe  --zookeeper localhost:2181 --topic test 

./bin/kafka-topics.sh --list --zookeeper localhost:2181

1. Separate zookeeper service
   Note: be sure to stop the zookeeper process and kafka process of kafka

./zkServer.sh start /root/zookeeper/conf/zoo.cfg //Start zookeeper
./zkServer.sh status /root/zookeeper/conf/zoo.cfg
./zkServer.sh stop /root/zookeeper/conf/zoo.cfg

nohup ./bin/kafka-server-start.sh ./config/server.properties & //Start kafka

Check:

# lsof -i:2181
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    18676 root   53u  IPv6 561661      0t0  TCP *:eforward (LISTEN)

# lsof -i:9092
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    18968 root  154u  IPv6 561700      0t0  TCP server-1:XmlIpcRegSvc (LISTEN)
java    18968 root  173u  IPv6 558773      0t0  TCP server-1:36496->server-1:XmlIpcRegSvc (ESTABLISHED)
java    18968 root  174u  IPv6 561707      0t0  TCP server-1:XmlIpcRegSvc->server-1:36496 (ESTABLISHED)

kafka stand-alone deployment

1. Download kafka
   Download address: kafka
   
   Scala 2.13 is the compiler and its version, 3.1.0 is the kafka version, and the download compiler is version 2.13
   Upload it to the server, rename it kafka, and give it permission

2. Start zookeeper
   Kafka depends on ZooKeeper, so you need to start a ZooKeeper server first

   The zookeeper configuration is zookeeper.config under the config directory Properties, default port 2181
   Start command: nohup/ bin/zookeeper-server-start. sh ./ config/zookeeper. Properties &, you can run zookeeper in the background after startup
   Command: ps -ef | grep zookeeper to check whether zookeeper is started successfully

Profile:
dataDir=/kafka/zookeeper  
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false

1. Start kafka
   Kafka's configuration file server is provided in the config directory properties. In order to ensure remote access to Kafka, two configurations need to be modified.

listeners=PLAINTEXT://: 9092 / / remove comments
advertised.listeners=PLAINTEXT://your.host.name:9092 / / remove the comment and modify it

broker.id=0
log.dirs=/kafka/kafka-logs  
zookeeper.connect=localhost:2181
# zookeeper.connect=localhost:2181,*:2181,*:2181

Start Kafka: nohup/ bin/kafka-server-start. sh ./ config/server. properties &
View: ps -ef | grep kafka

4 . Common commands

//zookeeper startup
nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &

//kafka start
nohup ./bin/kafka-server-start.sh ./config/server.properties &

//Create topic
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

//View topic list
./bin/kafka-topics.sh --list --zookeeper localhost:2181

//View description topic information 
./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic

//Create producer 
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

//Create consumer
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

zookeeper and bootstrap server
In Kafka after 0.9.0.0, there are several new changes. One is to add the role of GroupCoordinator on the Server side. Another big change is to store the offset information of topic in a special topic (_consumer_offsets) instead of previously stored on zookeeper

[use ZK] - zookeeper localhost: 2181
[use built-in] - bootstrap server localhost: 9092

It is officially recommended that if kafka version is greater than or equal to 2.2, use – bootstrap server localhost: 9092 instead of – zookeeper localhost:2181 (it is also compatible with – zookeeper above 2.2). The command is as follows

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test1

If kafka version is less than 2.2, the command is as follows

kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test1

Note: 2181 is the listening port of ZooKeeper and 9092 is the listening port of Kafka.

The old version uses the – zookeeper parameter, and the host name (or IP) and port use zookeeper, that is, server In the properties file, zookeeper Configuration value of the connect property

Old and new – bootstrap server parameter. The host name (or IP) and port can be the of a node, that is, host name (or IP): 9092

For consumers, kafka has two settings: for old consumers, it is set by the – zookeeper parameter; For new consumers, it is set by the – bootstrap server parameter
If the – zookeeper parameter is used, the information of the consumer will be stored in zk

The way to view is to use/ Zoomeeper client, and then LS / consumers / [group_id] / offsets / [Topic] / [broker_id part_id]. This is to view a group_ offset of a topic of ID

If the – bootstrap server parameter is used, the information of the consumer will be stored in kafka

For console producers, the - broker list parameter specifies the broker to use

The bin directory contains many operational shell scripts

Script name	Purpose description
connect-distributed.sh	Connection kafka cluster mode
connect-standalone.sh	Connecting kafka stand-alone mode
kafka-acls.sh	todo
kafka-broker-api-versions.sh	todo
kafka-configs.sh	Configuration management script
kafka-console-consumer.sh	kafka consumer console
kafka-console-producer.sh	kafka producer console
kafka-consumer-groups.sh	kafka consumer group related information
kafka-consumer-perf-test.sh	kafka consumer performance test script
kafka-delegation-tokens.sh	todo
kafka-delete-records.sh	Delete log files of low water level
kafka-log-dirs.sh	kafka message log directory information
kafka-mirror-maker.sh	kafka cluster replication tool for different data centers
kafka-preferred-replica-election.sh	Trigger the preferred replica election
kafka-producer-perf-test.sh	kafka producer performance test script
kafka-reassign-partitions.sh	Partition reassignment script
kafka-replay-log-producer.sh	todo
kafka-replica-verification.sh	Copy progress verification script
kafka-run-class.sh	todo
kafka-server-start.sh	Start kafka service
kafka-server-stop.sh	Stop kafka service
kafka-simple-consumer-shell.sh	It is recommended to use Kafka console consumer sh
kafka-streams-application-reset.sh	todo
kafka-topics.sh	topic management script
kafka-verifiable-consumer.sh	Verifiable kafka consumers
kafka-verifiable-producer.sh	Verifiable kafka producer
trogdor.sh	todo
zookeeper-security-migration.sh	todo
zookeeper-server-start.sh	Start zk service
zookeeper-server-stop.sh	Stop zk service
zookeeper-shell.sh	zk client