Programming VIP

1, Foreword This paper mainly aims at the problem that Flink SQL cannot realize multi-source consolidation of multi database and multi table by using Flink CDC, and how to update the downstream Kafka synchronously after multi-source consolidation, because at present, Flink SQL can only carry out the job operation of single table Flink CDC, whic ...

Inside Kafka, there is a__ consumer_ The topic of offsets. This topic mainly saves the offset of each consumption group to the consumption of the partition under the topic, that is, to which offset it is consumed. In order to realize the function of offset management of consumption group, when a consumer obtains a message, it is necessary for t ...

Environment centos7 kafka cluster and zookeeper cluster do not have user password by default. 1. Configure zookeeper cluster SASL All nodes of zookeeper are peer-to-peer, but the roles of each node may be different. The configuration of all nodes in the following steps is the same. 1. Add SASL support for zookeeper in the configuration file ...

Internet of things Kafka configuration Kafka theory + Zookeeper deep error set It is used in the company's Internet of things project. kafka is used as message middleware and kafka is used as the company's new technology. Currently, it uses single point without cluster. Kafka configuration of Internet of things Zookeeper: used to register an ...

1. What is MQ mq is A message middleware. The simplest understanding is that service A sends A message to service B, and service B performs A series of actions after receiving the message What problem can this solve Peak shaving: if services are called through the interface, whether http or rpc, but it is a synchronous action, it means tha ...

This article is reproduced from the official account StreamCloudNative, author Xue Song, who works in the new world software as a senior software engineer.  Editor: chicken chop, StreamNative.About Apache PulsarApache Pulsar is a top-level project of the Apache Software Foundation. It is a native distributed message flow platform for the n ...

There are a lot of delay operations in Kafka, such as delayed production, delayed pull and delayed deletion. Kafka does not use the Timer or DelayQueue provided by JDK to realize the delay function, but implements a Timer (SystemTimer) for the delay function based on the time wheel algorithm. The average time complexity of inserting and deleti ...

Analysis of kafka consumption message source code Example code of consumption message package com.example.demo.kafka; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.ConsumerRecord; import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaCon ...

Author: Wen naisongWhen using Flink to consume Kafka data in real time, it involves the maintenance of offset status. In order to ensure the restart of Flink job or the failure retry of Operator level during running, if you want to achieve "breakpoint continuation", you need the support of Flink Checkpoint. The question is, if you sim ...

Here, let's sort out the relevant knowledge of consumeGroup 1. First, we will set groupId for each consumer. Consumers with the same groupId and subscribed to the same topic will form a consumeGroup, as shown in Figure 1 2. For the topic on the Server side, there is the concept of partition, as shown in Figure 2 3. Now we have multiple con ...