Programming VIP

In the previous article, we introduced the installation, deployment and basic concepts of Flink. Today, let's learn about DataStream API, one of the core of Flink. 01 distributed stream processing foundation In the figure above, we divide the whole code into three parts, namely, the basic model of distributed stream processing: SourceTrans ...

Big data related knowledge points 1. Introduction to Kafka Kafka is a high-throughput distributed publish and subscribe message system. It can process all action flow data in consumer scale websites. It has the ability of high performance, persistence, multi copy backup and horizontal expansion Distributed system, easy to expand outward;At t ...

preface In the first section, we mentioned that on the kafka server side, we can create producers and send messages through commands. However, in the actual development, we all create producers and send messages in the project in the form of java. In this section, we will explain kafka producer based on JAVA API. 1, Introduction to JAVA API c ...

RabbitMQ message middleware 1. Message Oriented Middleware 1. Introduction **Message middleware, also known as message queue, refers to the platform independent data exchange with efficient and reliable message transmission mechanism, and the integration of distributed systems based on data communication** In the distributed environment, mes ...

Deep dive kafka producer - Core Architecture 3. Deep dive KafkaProducer infrastructure kafka has customized a set of network protocols, which can be implemented in any language to achieve the effect of pushing messages to and from kafka clusters. The clients module in the source code of kafka version 2.8.0 is the official default imple ...

Last example https://blog.csdn.net/xxkalychen/article/details/117149540?spm=1001.2014.3001.5502 Setting Flink's data source to Socket is just to provide streaming data for testing. This is not generally used in production. The standard model is to obtain streaming data from message queues. Flink provides the encapsulation of connecting with Kaf ...

Flow control Consumer In order to avoid excessive traffic hitting the Consumer end and crushing the Consumer due to the sharp increase of traffic in Kafka, we need to limit the current of the Consumer. For example, when the amount of data processed reaches a certain threshold, the consumption is suspended, and when it is lower than the thresho ...

Kafka 2.5.0 cluster installation (single machine pseudo cluster) Installation of Zookeeper 3.6.3 single node Download and unzip Zookeeper Download address https://zookeeper.apache.org/releases.html Download the binary version here. You don't need to compile apache-zookeeper-3.6.3-bin tar. gz Create the folder zookeeper under the / path and ...

Background: kafka client's producer API sends messages and simple source code analysis Starting from Kafka 0.11, Kafka producer supports two modes: idempotent producer and transaction producer. Idempotent producers strengthen Kafka's delivery semantics, from at least one delivery to precise one delivery. In particular, the retry of the produce ...

Design ideas In the past, we split the data into independent Kafka topics through diversion and other processing methods. Next, when processing the data, we should consider processing the index items used in real-time calculation. Timeliness is the pursuit of real-time data warehouse. Therefore, in some scenarios, it is not necessary to have a ...