Programming VIP

preface As a load balancer, haproxy can effectively share the access load of applications and help applications avoid single point risk to a certain extent. Sometimes, there is a need to analyze data from haproxy logs. In a current environment, haproxy is used as the load balancer of the back-end query interface to capture the key informat ...

An Apache Druid architecture 1. Coordinator Monitor historical processing, and be responsible for allocating segments to the specified service to ensure that historical data is self balanced 2. Overlord Monitor the MiddleManager to process and control data loading into the druid cluster; Responsible for the ingestion tasks assigned to the M ...

Distributed full text search solution 1. Solution introduction ElasticSearch is a distributed full-text search engine based on RESTful web interface. The solution is to implement a distributed full-text search system based on three data systems: Mysql database, Hadoop Ecology (optional) and ElasticSearch search engine. It mainly includes th ...

Apache Flume 1. General Flume is a highly available, reliable and distributed software for massive log collection, aggregation and transmission provided by Cloudera. The core of flume is to collect data from the data source, and then send the collected data to the specified destination (sink). In order to ensure the success of the transp ...

catalogue I. Homebrew installation II. SSH password free login configuration III. Hadoop installation Four pseudo distributed configurations a.hadoop-env.sh configuration b.core-site.xml configuration c.hdfs-site.xml configuration d.mapred-site.xml configuration e.yarn-site.xml configuration V. startup and operation Six test WordCo ...

1, Overview In spark, many places involve network communication, such as message exchange between various components of spark, upload of user files and Jar packets, Shuffle process data transmission between nodes, copy and backup of Block data, etc. Spark1. Before 6, Spark Rpc was implemented based on Akka, which is an asynchronous message ...

The core of Spark is based on the same abstract Resilient Distributed Datasets (RDD), which enables the components of Spark to integrate seamlessly and complete big data processing in the same application 1. Basic concepts of RDD RDD is the most important abstract concept provided by spark. It is a special data set with fault-tolerant mechani ...

1. Create maven project 2. Add POM Dependency under XML file 3. Copy log4j The properties file to the project root directory log4j.rootLogger=INFO, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - % ...

The 3rd Xiamen International Bank digital innovation finance cup financial marketing modeling competition BaseLine 1. Competition background With the development of science and technology, the bank has successively created rich and diverse customer contacts online and offline to meet the needs of customers for daily business handling and ch ...

Flume log collection 1, Flume introduction Logo refers to the transfer of wood (data) from one place (data source) to another place (data destination) through the river channel (channel) Working with documents 2, Installation configuration of Flume 1. Download and upload to the server 2. Decompression 3. Configuration Copy fl ...