Programming VIP

background The business purpose is to analyze the daily logs generated by nginx and apache, monitor the url, ip, rest interface and other information, and send the data to the elastic search service. Contrast flume No repeated consumption, no data loss At present, flume supports hdfs better (personal understanding) Off line installation Java ...

(I) environment introduction1. Server related settings:1,kafka1:172.20.67.522,kafka2:172.20.67.563,kafka3:172.20.67.57 2. Three ports required by zookeeper1,21812,28883,3888Functions of three ports:2181: providing services to clients2888: use of machine communication in the cluster (the Leader listens to this port)3888: election leader use 3. ...

1. Kafka installation and configuration 1.1 download Kafka Download Kafka, kafka_2.11-2.1.1.tgz on the official website. 1.2 decompression Extract to the specified directory. tar -zxvf -C your directory 1.3 start kafka Switch to the installation path of kafka and start zookeeper bin/zookeeper-server-start.sh config/z ...

kafka is liked by many enterprises because of its high-performance sending and consumption ability. So let's look at some source code implementations of kafka as follows: 1 public void run() { 2 int messageNo = 1; 3 while (true) { 4 String messageStr = "Message_" + messageNo; 5 long startTime ...

Spring Cloud Stream is a framework for building highly scalable event-driven microservices connected with shared messaging systems. The framework provides a flexible programming model built on already established and familiar Spring idioms and best practices, including support for persistent pub/sub semantics, consumer groups, and stateful par ...

Author: Yuanyi, senior development engineer of Alibaba cloud intelligent business group Introduction: Currently, Kafka event sources are supported in knight. How to implement message push based on Kafka? The author of this article will take Alibaba cloud Kafka products as an example to unlock this new posture. background Message queuing for ...

Download elastic search Create elk user and authorize useradd elk chown -R elk:elk /home/elk/elasticsearch chown -R elk:elk /home/elk/elasticsearch1 chown -R elk:elk /home/elk/elasticsearch2 mkdir -p /home/eladata mkdir -p /var/log/elk chown -R elk:elk /home/eladata chown -R elk:elk /var/log/elk Master node master Extract from elastic search an ...

1. Implementing kafka transaction by spring annotation 1. Add Configuration to the Configuration Class in the previous article /** * The following three methods are used for management * Configure Kafka Transaction Manager, which is the transaction management class provided by Kafka to us. We need t ...

Reference resources: dapeng Log Collection, Processing and Query Application Recently, according to dapeng's log management system, I think I'll try to build an EFK myself. Here I record my experience of trampling pits. Thank you very much. Ever_00,Yangyang_3720 Support and help from big boys Technical Selection Returning to the topic, we still ...

Preface Kafka is a message queue product. Based on Topic partitions design, it can achieve very high performance of message sending and processing. Spring created a project called Spring-kafka, which encapsulates Apache's Kafka-client for rapid integration of Kafka in Spring projects. In addition to simple messaging, Spring-kafka also provides ...