kafka-01 deployment start

Startup of kafka

  1. The zookeeper service provided by kafka
//The default startup mode is foreground startup. When the current command line window is closed, the process will be closed
# ./bin/zookeeper-server-start.sh ./config/zookeeper.properties
//Background start command
# nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &
# ./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties
//The "- daemon" parameter represents starting kafka server as a daemon

nohup ./bin/kafka-server-start.sh ./config/server.properties & //Start kafka

//Check whether the port number is listening
lsof -i:2181  //zookeeper port
lsof -i:9092  //kafka port
  • Common commands:
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test //Create topic

./bin/kafka-topics.sh --describe  --zookeeper localhost:2181 --topic test 

./bin/kafka-topics.sh --list --zookeeper localhost:2181
  1. Separate zookeeper service
    Note: be sure to stop the zookeeper process and kafka process of kafka
./zkServer.sh start /root/zookeeper/conf/zoo.cfg //Start zookeeper
./zkServer.sh status /root/zookeeper/conf/zoo.cfg
./zkServer.sh stop /root/zookeeper/conf/zoo.cfg

nohup ./bin/kafka-server-start.sh ./config/server.properties & //Start kafka


# lsof -i:2181
java    18676 root   53u  IPv6 561661      0t0  TCP *:eforward (LISTEN)

# lsof -i:9092
java    18968 root  154u  IPv6 561700      0t0  TCP server-1:XmlIpcRegSvc (LISTEN)
java    18968 root  173u  IPv6 558773      0t0  TCP server-1:36496->server-1:XmlIpcRegSvc (ESTABLISHED)
java    18968 root  174u  IPv6 561707      0t0  TCP server-1:XmlIpcRegSvc->server-1:36496 (ESTABLISHED)

kafka stand-alone deployment

  1. Download kafka
    Download address: kafka

    Scala 2.13 is the compiler and its version, 3.1.0 is the kafka version, and the download compiler is version 2.13
    Upload it to the server, rename it kafka, and give it permission

  2. Start zookeeper
    Kafka depends on ZooKeeper, so you need to start a ZooKeeper server first

    The zookeeper configuration is zookeeper.config under the config directory Properties, default port 2181
    Start command: nohup/ bin/zookeeper-server-start. sh ./ config/zookeeper. Properties &, you can run zookeeper in the background after startup
    Command: ps -ef | grep zookeeper to check whether zookeeper is started successfully

# the port at which the clients will connect
# disable the per-ip limit on the number of connections since this is a non-production config
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
  1. Start kafka
    Kafka's configuration file server is provided in the config directory properties. In order to ensure remote access to Kafka, two configurations need to be modified.
listeners=PLAINTEXT://: 9092 / / remove comments
advertised.listeners=PLAINTEXT://your.host.name:9092 / / remove the comment and modify it

# zookeeper.connect=localhost:2181,*:2181,*:2181

Start Kafka: nohup/ bin/kafka-server-start. sh ./ config/server. properties &
View: ps -ef | grep kafka

4 . Common commands

//zookeeper startup
nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &

//kafka start
nohup ./bin/kafka-server-start.sh ./config/server.properties &

//Create topic
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

//View topic list
./bin/kafka-topics.sh --list --zookeeper localhost:2181

//View description topic information 
./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic

//Create producer 
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

//Create consumer
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

zookeeper and bootstrap server
In Kafka after, there are several new changes. One is to add the role of GroupCoordinator on the Server side. Another big change is to store the offset information of topic in a special topic (_consumer_offsets) instead of previously stored on zookeeper

[use ZK] - zookeeper localhost: 2181
[use built-in] - bootstrap server localhost: 9092

It is officially recommended that if kafka version is greater than or equal to 2.2, use – bootstrap server localhost: 9092 instead of – zookeeper localhost:2181 (it is also compatible with – zookeeper above 2.2). The command is as follows

kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test1

If kafka version is less than 2.2, the command is as follows

kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test1

Note: 2181 is the listening port of ZooKeeper and 9092 is the listening port of Kafka.

The old version uses the – zookeeper parameter, and the host name (or IP) and port use zookeeper, that is, server In the properties file, zookeeper Configuration value of the connect property

Old and new – bootstrap server parameter. The host name (or IP) and port can be the of a node, that is, host name (or IP): 9092

For consumers, kafka has two settings: for old consumers, it is set by the – zookeeper parameter; For new consumers, it is set by the – bootstrap server parameter
If the – zookeeper parameter is used, the information of the consumer will be stored in zk

The way to view is to use/ Zoomeeper client, and then LS / consumers / [group_id] / offsets / [Topic] / [broker_id part_id]. This is to view a group_ offset of a topic of ID

If the – bootstrap server parameter is used, the information of the consumer will be stored in kafka

For console producers, the - broker list parameter specifies the broker to use

The bin directory contains many operational shell scripts

Script namePurpose description
connect-distributed.shConnection kafka cluster mode
connect-standalone.shConnecting kafka stand-alone mode
kafka-configs.shConfiguration management script
kafka-console-consumer.shkafka consumer console
kafka-console-producer.shkafka producer console
kafka-consumer-groups.shkafka consumer group related information
kafka-consumer-perf-test.shkafka consumer performance test script
kafka-delete-records.shDelete log files of low water level
kafka-log-dirs.shkafka message log directory information
kafka-mirror-maker.shkafka cluster replication tool for different data centers
kafka-preferred-replica-election.shTrigger the preferred replica election
kafka-producer-perf-test.shkafka producer performance test script
kafka-reassign-partitions.shPartition reassignment script
kafka-replica-verification.shCopy progress verification script
kafka-server-start.shStart kafka service
kafka-server-stop.shStop kafka service
kafka-simple-consumer-shell.shIt is recommended to use Kafka console consumer sh
kafka-topics.shtopic management script
kafka-verifiable-consumer.shVerifiable kafka consumers
kafka-verifiable-producer.shVerifiable kafka producer
zookeeper-server-start.shStart zk service
zookeeper-server-stop.shStop zk service
zookeeper-shell.shzk client

Keywords: kafka

Added by Aaron111 on Tue, 22 Feb 2022 09:27:46 +0200