logstash installation and simple test

background The business purpose is to analyze the daily logs generated by nginx and apache, monitor the url, ip, rest interface and other information, and send the data to the elastic search service. Contrast flume No repeated consumption, no data loss At present, flume supports hdfs better (personal understanding) Off line installation Java ...

Added by TheMD on Sun, 05 Jan 2020 06:29:24 +0200

Kafka cluster installation and configuration

(I) environment introduction1. Server related settings:1,kafka1:172.20.67.522,kafka2:172.20.67.563,kafka3:172.20.67.57 2. Three ports required by zookeeper1,21812,28883,3888Functions of three ports:2181: providing services to clients2888: use of machine communication in the cluster (the Leader listens to this port)3888: election leader use 3. ...

Added by f1nutter on Mon, 09 Dec 2019 02:40:08 +0200

Flume and Kafka connectivity test

1. Kafka installation and configuration 1.1 download Kafka Download Kafka, kafka_2.11-2.1.1.tgz on the official website. 1.2 decompression Extract to the specified directory. tar -zxvf -C your directory 1.3 start kafka Switch to the installation path of kafka and start zookeeper bin/zookeeper-server-start.sh config/z ...

Added by godwheel on Sat, 30 Nov 2019 07:31:38 +0200

Look at kafka of source code programming series

kafka is liked by many enterprises because of its high-performance sending and consumption ability. So let's look at some source code implementations of kafka as follows: 1 public void run() { 2 int messageNo = 1; 3 while (true) { 4 String messageStr = "Message_" + messageNo; 5 long startTime ...

Added by fatherlyons on Fri, 22 Nov 2019 21:53:14 +0200

Let's talk about spring cloud stream and kafka.

Spring Cloud Stream is a framework for building highly scalable event-driven microservices connected with shared messaging systems. The framework provides a flexible programming model built on already established and familiar Spring idioms and best practices, including support for persistent pub/sub semantics, consumer groups, and stateful par ...

Added by someberry on Wed, 30 Oct 2019 04:57:01 +0200

Knotive practice: message push based on Kafka

Author: Yuanyi, senior development engineer of Alibaba cloud intelligent business group Introduction: Currently, Kafka event sources are supported in knight. How to implement message push based on Kafka? The author of this article will take Alibaba cloud Kafka products as an example to unlock this new posture. background Message queuing for ...

Added by faraway on Fri, 25 Oct 2019 09:54:09 +0300

The latest Centos7.6 deployment of ELK log analysis system

Download elastic search Create elk user and authorize useradd elk chown -R elk:elk /home/elk/elasticsearch chown -R elk:elk /home/elk/elasticsearch1 chown -R elk:elk /home/elk/elasticsearch2 mkdir -p /home/eladata mkdir -p /var/log/elk chown -R elk:elk /home/eladata chown -R elk:elk /var/log/elk Master node master Extract from elastic search an ...

Added by crochk on Tue, 22 Oct 2019 19:50:52 +0300

Kafka Initial Learning--Kafka Transaction Support

1. Implementing kafka transaction by spring annotation 1. Add Configuration to the Configuration Class in the previous article /** * The following three methods are used for management * Configure Kafka Transaction Manager, which is the transaction management class provided by Kafka to us. We need t ...

Added by jaimitoc30 on Fri, 04 Oct 2019 03:06:45 +0300

EFK Architecture of Logging System

Reference resources: dapeng Log Collection, Processing and Query Application Recently, according to dapeng's log management system, I think I'll try to build an EFK myself. Here I record my experience of trampling pits. Thank you very much. Ever_00,Yangyang_3720 Support and help from big boys Technical Selection Returning to the topic, we still ...

Added by soianyc on Thu, 29 Aug 2019 13:02:03 +0300

spring boot integration Kafka spring-kafka in-depth exploration

Preface Kafka is a message queue product. Based on Topic partitions design, it can achieve very high performance of message sending and processing. Spring created a project called Spring-kafka, which encapsulates Apache's Kafka-client for rapid integration of Kafka in Spring projects. In addition to simple messaging, Spring-kafka also provides ...

Added by atholon on Wed, 28 Aug 2019 13:18:55 +0300