Common scripts fkaka

Welcome to: My blog

introduction

This article is reproduced, but the original text

In the Kafka installation directory ($KAFKA_HOME/bin), many built-in scripts are provided for us to use. Using scripts can test most functions of Kafka. We will explain the use of scripts below

Start broker

bin/kafka-server-start.sh script provides the function of starting broker

Foreground start

bin/kafka-server-start.sh config/server.properties

Background start:

bin/kafka-server-start.sh -daemon config/server.properties

Stop broker

bin/kafka-server-stop.sh script provides the function of stopping broker

bin/kafka-server-stop.sh

Create topic

bin/kafka-topic.sh script provides the function of creating topic When creating topic, you need to specify two parameters:

Number of partitions
Number of copies

Note that the maximum number of replicas cannot exceed the number of broker nodes in the current cluster

Next, create a topic named test, with 3 partitions and 2 copies

bin/kafka-topics.sh --zookeeper localhost:2181 --create --partitions 3 --replication-factor 2 --topic test

View topic list

bin/kafka-topic.sh script provides the function of viewing the list of topics Through the -- List parameter, you can view the list of all topics in the current cluster

bin/kafka-topics.sh --zookeeper localhost:2181 --list
__consumer_offsets
test

View the details of a topic

bin/kafka-topic.sh script provides the function of viewing the detailed information of a topic You can specify the topic name through the -- describe parameter and the -- Topic parameter

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic test
Topic:test PartitionCount:3 ReplicationFactor:2 Configs:
Topic: test Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0
Topic: test Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: test Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2

Delete topic

bin/kafka-topic.sh script provides the function of deleting topic You can specify the topic name through the -- delete parameter and the -- Topic parameter

Delete the topic named test:

bin/kafka-topics.sh --zookeeper localhost:2181/kafka --delete --topic test
Topic test is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

Note that delete must be set in the broker configuration topic. Enable = true, otherwise the deletion will not take effect

Command line producer

bin/kafka-console-producer.sh script can send data to the specified topic through command line input

Send three pieces of data to the topic named test:

bin/kafka-console-producer.sh --broker-list hostname:9092 --topic test
This is a message
This is another message
hello kafka

Command line consumer

bin/kafka-console-consumer. The SH script can consume the data in topic and print it to the console

Data in topic with the name of test:

bin/kafka-console-consumer.sh --bootstrap-server hostname:9092 --topic test --from-beginning
This is a message
This is another message
hello kafka
^CProcessed a total of 3 messages

Note that the – from beginning parameter means to consume from the beginning of topic. If this parameter is not specified, it means to consume from the end. It can be displayed on the console only when new data is written after starting the consumer

View consumer groups

bin/kafka-consumer-groups. The SH script can view the information of the consumption group in the cluster The current consumption group list can be listed through the -- List parameter

bin/kafka-consumer-groups.sh --bootstrap-server hostname:9092 --new-consumer --list
console-consumer-54336

You can view the consumption details of the consumption group through the -- describe parameter:

bin/kafka-consumer-groups.sh --bootstrap-server company01:9092 --new-consumer --describe --group console-consumer-54336

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test 0 1 1 0 consumer-1-860f59b3-ebe9-4cda-a074-d108b1fec7bf /192.168.100.109 consumer-1
test 1 1 1 0 consumer-1-860f59b3-ebe9-4cda-a074-d108b1fec7bf /192.168.100.109 consumer-1
test 2 1 1 0 consumer-1-860f59b3-ebe9-4cda-a074-d108b1fec7bf /192.168.100.109 consumer-1

CURRENT-OFFSET indicates the offset of current consumption to the partition

LOG-END-OFFSET indicates the last offset of the current partition

LAG indicates that consumers are "behind" in consumption progress

Original address: Common scripts fkaka