The first experience of flink streaming sql

Article catalog background Example explanation Construct the StreamTableEnvironment object Register table Using Tuple Using Row Pojo class using java Use external storage background sql,Structured Query Language: as a general and popular query language, Structured Query Language is becomin ...

Added by TodManPlaa on Sun, 21 Jun 2020 07:16:31 +0300

Kafka Java client use

Article catalog 1. Common configuration 2. kafka-clients 2.1 producers 2.2 consumers 2.3 multithreading 1. Common configuration producer to configure describe type Default bootstrap.servers It is used to establish the connection with kafka cluster and only affects the hosts used for init ...

Added by Gmans on Sat, 13 Jun 2020 07:34:03 +0300

Consumption mode of consumer of pulsar

client processing logic ConnectionPool netty related configurations are available in com.yahoo.pulsar.client.impl.ConnectionPool public ConnectionPool(final PulsarClientImpl client, EventLoopGroup eventLoopGroup) { this.eventLoopGroup = eventLoopGroup; this.maxConnectionsPerHosts = client.getConfiguration().getConnection ...

Added by Nothsa on Wed, 27 May 2020 14:04:46 +0300

Illustrate IO models and related technologies for Linux

Blocking IO Model (Blocking I/O) The Linux kernel initially provides read and write blocking operations. When a client connects, a corresponding file descriptor (0 standard input, 1 standard output, 2 standard error output) is generated in the file descriptor directory of the corresponding process (/proc/process number/fd), such as FD 8, FD 9 ...

Added by eulalyn09 on Sun, 10 May 2020 11:54:46 +0300

Actual | Write to Hudi using Spark Streaming

1. Project Background Traditional data warehouse organization structure is designed for OLAP (Online Transaction Analysis) requirements of offline data. The common way to import data is to use sqoop or spark timer jobs to import business database data into warehouses one by one.With the increasing requirement of real-time in data analysis, hou ...

Added by robh76 on Sun, 19 Apr 2020 03:03:03 +0300

Using Kafka message queue in springboot project

1. Kafka message producer 1) Create the springboot Kafka producer demo project and import the jar package. <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> <version>2.4.5.RELEASE</version> </dependency> 2) Add Kafka producer configuration it ...

Added by Patrick3002 on Tue, 14 Apr 2020 18:03:10 +0300

Flink processes business data in the database - associating primary and secondary tables

Article Directory 1. Requirements 1. Requirement Description 2. Data preparation 2. Analysis 3. Technical Points 4. Join Architecture 5. Tool Class and Standard Class 1. FlinkUtilsV2 Get Tool Class for KafkaSource 2. OrderMainBean Order Master Table Class 3. OrderDetailBean Order Detail Table Clas ...

Added by ThE_eNd on Tue, 17 Mar 2020 03:35:31 +0200

Flink: take away all State and go far away: State initial

State: State refers to the intermediate calculation result or metadata attribute of the calculation node in the process of flow calculation. For example, in the process of aggregation, the intermediate aggregation result should be recorded in the state. For example, when Apache Kafka is the data sourc ...

Added by CanMan2004 on Wed, 04 Mar 2020 11:19:04 +0200

Deploy Zookeeper and Kafka clusters through docker

Recently, a kafka service needs to be set up locally. It's more convenient for docker to pull an image. Record the process of setting up 1. Install virtual machine I prefer virtualbox, which gives people a very lightweight feeling. The simplest version of CentOS 7.3 installed in Linux. I used to be fond of Linux desktop, but virtual machine res ...

Added by benphp on Fri, 28 Feb 2020 14:01:53 +0200

Spring Boot2.x Integrated kafka (windows integrated environment + example explanation)

Learn kafka, this article takes you to the beginning. Article directory 1. Build windows integrated environment 1.1 java8 installation 1.2 zookeeper installation 1.2.1 download zookeeper 1.2.2 install and modify the zoo.cfg configuration file 1.2.3 setting the zookeeper environment variable 1.2.4 ve ...

Added by jbrave on Sun, 23 Feb 2020 11:54:17 +0200