Introduction to RabbitMQ message middleware

RabbitMQ message middleware

1. Message Oriented Middleware

1. Introduction

**Message middleware, also known as message queue, refers to the platform independent data exchange with efficient and reliable message transmission mechanism, and the integration of distributed systems based on data communication** In the distributed environment, message delivery and process communication can be provided through the extended message queue model.

At present, the mainstream message middleware includes RabbitMQ, Kafka, ActiveMQ, RocketMQ, etc.

2. Function

1. Main functions of Message Oriented Middleware
  • Redundancy (storage)
  • Expansibility
  • Recoverability
  • buffer
  • asynchronous communication
  • Peak clipping: a common scenario in message queue, which is generally widely used in second kill or rush buying activities. Generally, because the traffic is too large, the application system configuration can not bear this instantaneous traffic, resulting in the system directly hanging up, that is, the legendary "downtime" phenomenon. In order to solve this problem, we will reject the huge traffic at the upper level of the system, that is, transfer it to MQ instead of directly pouring into our interface.
  • Decoupling (coupling)
2. Two modes of Message Oriented Middleware
1. P2P mode

P2P mode includes three roles: message Queue, sender and receiver. Each message is sent to a specific Queue, and the receiver gets the message from the Queue. The Queue holds messages until they are consumed or timed out.

Characteristics of P2P: (peer-to-peer: Queue, non repeatable consumption)

  • Each message has only one Consumer, that is, once consumed, the message no longer exists in the message queue
  • There is no time dependency between the sender and the receiver, that is, after the sender sends the message, whether the receiver is running or not, it will not affect the message to be sent to the queue
  • After receiving the message successfully, the receiver needs to reply to the queue
  • If you want every message sent to be processed successfully, you need P2P mode
2. Pub/Sub mode (publish / subscribe: Topic, which can be consumed repeatedly)

The Pub/Sub mode includes three roles: Topic, Publisher and Subscriber. Multiple publishers send messages to Topic, and the system passes these messages to multiple subscribers.

Features of Pub/Sub:

  • Each message can have multiple consumers
  • There is a time dependency between publishers and subscribers. For a subscriber of a Topic, it must create a subscriber before consuming the publisher's message
  • In order to consume messages, subscribers must remain running
  • If the message you want to send can be processed without any processing, by only one consumer, or by multiple consumers, you can use the Pub/Sub model
3. Introduction and comparison of common middleware
1,Kafka

Kafka is an open-source distributed publish subscribe messaging system of LinkedIn, which is currently a top-level project of Apache. Kafka is mainly characterized by the pursuit of high throughput, and its initial purpose is to collect and transmit logs. Version 0.8 supports replication, does not support transactions, and has no strict requirements on message duplication, loss and error. It is suitable for the data collection business of Internet services that generate a large amount of log data.

2,RabbitMQ

RabbitMQ is an open source message queue system developed in Erlang language and implemented based on AMQP protocol. The main features of AMQP are message oriented, queue oriented, routing (including point-to-point and publish / subscribe), reliability and security. AMQP protocol is more used in scenarios that require high data consistency, stability and reliability in enterprise systems, and the requirements for performance and throughput are second.

Erlang is a general concurrency oriented programming language

3,RocketMQ

RocketMQ is Alibaba's open source message middleware, * * it is developed in pure Java. It has the characteristics of high throughput, high availability and is suitable for large-scale distributed system applications** It optimizes the reliable transmission and transaction of messages. At present, it is widely used in transaction, recharge, message push, log streaming, binglog distribution and other scenarios in Alibaba group.

RabbitMQ is more reliable than Kafka. Kafka is more suitable for IO high throughput processing. It is generally used in big data log processing or scenarios with slightly lower requirements for real-time (a small amount of delay) and reliability (a small amount of data loss), such as ELK log collection.

2. RabbitMQ cluster

1. About RabbitMQ

RabbitMQ is developed by Erlang. Clustering is very convenient because Erlang is naturally a distributed language, but it does not support load balancing, high concurrency and scalability. It supports AJAX and persistence. It is used to store and forward messages in distributed systems. It performs well in terms of ease of use, scalability and high availability.

Ajax, namely "Asynchronous Javascript And XML", refers to a web development technology for creating interactive web applications. Ajax = Asynchronous Javascript And XML

2. RabbitMQ features

  • reliability

  • Expansibility

  • High availability

  • Multiple protocols

  • Multilingual client

  • Web management interface

  • Plug-in mechanism

    3. What is a message queue

    The full name of MQ is Message Queue. Is an application to application communication method. Applications communicate by reading and writing messages in and out of the queue (data for the application) without requiring a dedicated connection to link them.

    Message passing refers to the communication between programs by sending data in messages.

RabbitMQ mode

Note: RabbitMQ mode can be divided into the following three types:

(1) Stand alone mode.

(2) Normal mode (default cluster mode).

(3) Mirror mode (the required queue is made into a mirror queue, which exists in multiple nodes. It belongs to the HA scheme of RabbiMQ, which is more suitable for occasions requiring high service reliability). To realize the image mode, you need to build a common cluster mode, and then configure the image mode to achieve high availability.

Understand the basic concepts in Cluster:

RabbitMQ cluster nodes include memory nodes and disk nodes. As the name suggests, the memory node puts all data in memory, and the disk node puts data on disk.

A rabbitmq cluster can share user, vhost, queue, exchange, etc. all data and states must be replicated on all nodes.

Broker:Message queuing server entity
ConnectionFactory(Connection manager): application and Rabbit Manager for establishing connection between, which is used in program code;
Exchange(Switch): used to receive and distribute messages;
Routing Key: Routing keywords, exchange Message delivery according to this keyword;
Queue(Queue): used to store messages of producers;
Bindding: Bind, put exchange and queue Bind according to the routing rules.
vhost: Virtual host, one broker You can open more than one vhost，Used as a separation of permissions for different users.
producer: Message producer is the program that delivers messages.
consumer: A message consumer is a program that accepts messages.
channel: (Channel) message channel. Multiple channels can be established in each connection of the client channel，each channel Represents a session task. The channel used for message push;

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-m5w1ralf-16215642946022) (assets / 15713200020040. PNG)]

Interview note: there are two kinds of nodes in the cluster

1 memory node: only save the state to memory (the persistent contents of the persistent queue will be saved to disk)

2 disk node: save state to memory and disk recommend

It is better to write to the node than to the disk. In a cluster, only one disk node is needed to save the state

If there are only memory nodes in the cluster, they cannot be stopped, otherwise all status and messages will be lost.

Common cluster preparation environment

Note that for the three servers, the RabbitMQ cluster node must be in the same network segment. If it is cross domain, the effect will be worse. Turn off firewall and selinux

Modify host name and add resolution

All three machines operate:

  1. Configure the hosts file, change the computer names of the three MQ nodes to rabbitmq-1, rabbitmq-2 and rabbitmq-3 respectively, and then modify the hosts configuration

    [root@rabbitmq-1 ~]# hostnamectl set-hostname rabbitmq-1
[root@rabbitmq-1 ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.50.138 rabbitmq-1
192.168.50.139 rabbitmq-2
192.168.50.140 rabbitmq-3

    2. Configure and install rabbitmq software on three nodes

Installation dependency
[root@rabbitmq-1 ~]# yum install -y *epel* gcc-c++ unixODBC unixODBC-devel openssl-devel ncurses-devel
yum install erlang
[root@rabbitmq-1 ~]# wget --content-disposition https://packagecloud.io/rabbitmq/erlang/packages/el/7/erlang-20.3-1.el7.centos.x86_64.rpm/download.rpm
[root@rabbitmq-1 ~]# yum install erlang-20.3-1.el7.centos.x86_64.rpm -y
 Testing;
[root@rabbitmq-1 ~]# erl
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.3  (abort with ^G)
1>

install rabbitmq
[root@rabbitmq-1 ~]# wget https://dl.bintray.com/rabbitmq/all/rabbitmq-server/3.7.5/rabbitmq-server-3.7.5-1.el7.noarch.rpm
[root@rabbitmq-1 ~]# yum install rabbitmq-server-3.7.5-1.el7.noarch.rpm -y

3.start-up
[root@rabbitmq-1 ~]# systemctl daemon-reload
[root@rabbitmq-1 ~]# systemctl start rabbitmq-server
[root@rabbitmq-1 ~]# systemctl enable rabbitmq-server
[root@rabbitmq-1 ~]# systemctl status rabbitmq-server
 Start mode II:
[root@rabbitmq-1 ~]# /SBIN / service rabbitmq server status --- view status
[root@rabbitmq-1 ~]# /SBIN / service rabbitmq server start --- start
 Each machine is operated and turned on rabbitmq of web Access interface: 
[root@rabbitmq-1 ~]# rabbitmq-plugins enable rabbitmq_management

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-FiAKs8S8-1621564294609)(assets/1585804728482.png)]

Create user

be careful:Operate in one machine
 Add user and password
[root@rabbitmq-1 ~]# rabbitmqctl add_user soho soso
Adding user "soho" ...
Set as Administrator
[root@rabbitmq-1 ~]# rabbitmqctl set_user_tags soho administrator
Setting tags for user "soho" to [administrator] ...
View user
[root@rabbitmq-1 ~]# rabbitmqctl list_users
Listing users ...
guest	[administrator]
soho	[administrator]
...done.

Note when setting permissions here'.*'Three spaces are required between'.*'Each represents conf jurisdiction, read Authority and write Permission, for example: when not given
soho Before setting these three permissions, you do not have permission to query the queue ui The interface is also invisible
[root@rabbitmq-1 ~]# rabbitmqctl set_permissions -p "/" soho ".*" ".*" ".*"
Setting permissions for user "soho" in vhost "/" ...

All three machines operate: enable user remote login:

[root@rabbitmq-1 ~]# cd /etc/rabbitmq/   
[root@rabbitmq-1 rabbitmq]# cp /usr/share/doc/rabbitmq-server-3.7.5/rabbitmq.config.example /etc/rabbitmq/rabbitmq.config
[root@rabbitmq-1 rabbitmq]# ls
enabled_plugins  rabbitmq.config
[root@rabbitmq-1 rabbitmq]# vim rabbitmq.config
 Amend as follows:

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-I6DP3HnW-1621564294615)(assets/1564157816348.png)]

All three machines operate and restart the service:
[root@rabbitmq-1 ~]# systemctl restart rabbitmq-server

View port

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-7v5v6Dkx-1621564294619)(assets/1564157344617.png)]

4369 -- erlang port
5672 --Program connection port
15672 -- Management interface ui port
25672 -- server Inter room internal communication port

! Note: if it is an ECS, remember to add a security group port.

Visit: 192.168.50.138:15672

[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-hfYNreLN-1621564294625)(assets/1564157664109.png)]

[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-3vvgeiuv-16215642946227) (assets / 1564158056319. PNG)]

Note here:

rabbitmq default administrator user: guest password: Guest

The newly added user is: soho password: soso

Start to deploy the cluster. All three machines operate:

1. First create the data storage directory and log storage directory:

3. All machines operate:

[root@rabbitmq-1 ~]# mkdir -p /data/rabbitmq/data
[root@rabbitmq-1 ~]# mkdir -p /data/rabbitmq/logs
[root@rabbitmq-1 ~]# chmod 777 -R /data/rabbitmq
[root@rabbitmq-1 ~]# chown rabbitmq.rabbitmq /data/ -R
 create profile:
[root@rabbitmq-1 ~]# vim /etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_MNESIA_BASE=/data/rabbitmq/data
RABBITMQ_LOG_BASE=/data/rabbitmq/logs
 Restart service
[root@rabbitmq-1 ~]# systemctl restart rabbitmq-server

2. Copy Erlang cookie

Rabbitmq's cluster is dependent on Erlang's cluster, so Erlang's cluster must be constructed first. In Erlang's cluster

Each node is implemented through each cookie, which is stored in / var / lib / rabbitmq / erlang. In the cookie, the file is 400 permissions. Therefore, the cookie of each node must be consistent, otherwise the nodes cannot communicate.

If you execute # rabbitmqctl stop_ The app command reports an error: it needs to be executed

#If implemented# rabbitmqctl stop_ The app command reports an error: it needs to be executed
#chmod 400 .erlang.cookie
#chown rabbitmq.rabbitmq .erlang.cookie

(the official documents about clusters mentioned that. Erlang. Cookies generally exist in these two addresses: the first is home/.erlang.cookie; the second is / var/lib/rabbitmq/.erlang.cookie. If we use decompression to install the deployed rabbitmq, this file will be in {home} Directory, which is $home / erlang.cookie. If we use rpm and other installation packages for installation, this file will be in the / var/lib/rabbitmq directory.)

[root@rabbitmq-1 ~]# cat /var/lib/rabbitmq/.erlang.cookie
HOUCUGJDZYTFZDSWXTHJ
scp The way will be rabbitmq-1 Nodal.erlang.cookie The value of is copied to the other two nodes.
[root@rabbitmq-1 ~]# scp /var/lib/rabbitmq/.erlang.cookie root@192.168.50.139:/var/lib/rabbitmq/
[root@rabbitmq-1 ~]# scp /var/lib/rabbitmq/.erlang.cookie root@192.168.50.140:/var/lib/rabbitmq/

3. Add mq-2 and mq-3 as memory nodes to the mq-1 node cluster

stay mq-2,mq-3 Execute the following command:
[root@rabbitmq-2 ~]# rabbitmqctl stop_app  #Stop the node and remember not to stop the service

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-6My5Ykhe-1621564294629)(assets/1585815681534.png)]

[root@rabbitmq-2 ~]# rabbitmqctl reset   #If there is data to be reset, do not use it if there is no data

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-vNms6jJ6-1621564294630)(assets/1585815744200.png)]

Pay attention to check the echo, if not above. Is wrong; If an error is reported, restart the rabbitmq service

Join two nodes into the cluster and specify roles
[root@rabbitmq-2 ~]# rabbitmqctl join_cluster --ram rabbit@rabbitmq-1  #Add to memory node
Clustering node 'rabbit@rabbitmq-2' with 'rabbit@rabbitmq-1' ...
[root@rabbitmq-2 ~]# rabbitmqctl start_app  #Start node
Starting node 'rabbit@rabbitmq-2' ...
 completed with 3 plugins.
===============================================================================
[root@rabbitmq-3 ~]# rabbitmqctl stop_app
Stopping node 'rabbit@rabbitmq-3' ...
[root@rabbitmq-3 ~]# rabbitmqctl reset
Resetting node 'rabbit@rabbitmq-3' ...
[root@rabbitmq-3 ~]# rabbitmqctl join_cluster --ram rabbit@rabbitmq-1
Clustering node 'rabbit@rabbitmq-3' with 'rabbit@rabbitmq-1' ...
[root@rabbitmq-3 ~]# rabbitmqctl start_app
Starting node 'rabbit@rabbitmq-3' ...
 completed with 3 plugins.

(1)default rabbitmq After startup is the disk node, in this cluster On command, mq-2 and mq-3 Is a memory node,
mq-1 Is a disk node.
(2)If you want to make mq-2,mq-3 They are all disk nodes. Remove them--ram Parameters.
(3)If you want to change the node type, you can use the command rabbitmqctl change_cluster_node_type
disc(ram),The premise is that it must stop rabbitmq application
 notes:
#If necessary, use disk nodes to join the cluster
 [root@rabbitmq-2 ~]# rabbitmqctl join_cluster  rabbit@rabbitmq-1
 [root@rabbitmq-3 ~]# rabbitmqctl join_cluster  rabbit@rabbitmq-1

4. View cluster status

stay RabbitMQ Execute on any node of the cluster rabbitmqctl cluster_status To check whether the cluster configuration is successful.
stay mq-1 View on disk node
[root@rabbitmq-1 ~]# rabbitmqctl cluster_status

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-d8e8ozih-1621564294632) (assets / 15641558874154. PNG)]

Each machine displays three nodes, indicating that it has been added successfully!

5. Log in to rabbitmq web management console and create a new queue

Open browser input http://192.168.50.138:15672 ,

Enter the default Username: guest

Enter the default Password:guest

After logging in, the interface shown in the figure appears.

[the external link image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-K6DBggHI-1621564294633)(assets/1564162424739.png)]

Create a queue according to the interface prompt

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-Nu2a8hpS-1621564294634)(assets/1564159056655.png)]

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-oeft7PBp-1621564294636)(assets/1564159160374.png)]

RabbitMQ mirror cluster configuration

The default cluster mode of RabbitMQ has been completed above, but the high availability of the queue is not guaranteed, and the contents of the queue will not be copied. If the queue cannot be applied due to the downtime of the queue node, it can only wait for restart. Therefore, in order to apply normally in case of the downtime or failure of the queue node, it is necessary to copy the contents of the queue to each node in the cluster and create a mirror queue.

The mirror queue is based on the normal cluster mode.

Create a mirror cluster: operate on any machine

rabbitmq set_policy: set policy

[root@rabbitmq-1 ~]# rabbitmqctl set_policy  ha-all "^" '{"ha-mode":"all"}'
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...

 "^"Match all queues, ha-all Policy name is ha-all, '{"ha-mode":"all"}' The policy mode is all Copy to all nodes, including new nodes.

Check again that the queue has been synchronized to the other two nodes:

[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-6dhgajtq-1621564294637) (assets / 156415940040. PNG)]

"^" matches all queues. The ha all policy name is ha all, and the '{"Ha mode": "all"} "policy mode is all, that is, it is copied to all nodes, including new nodes.

Introduction to setting policy:
rabbitmqctl set_policy [-p Vhost] Name Pattern Definition
-p Vhost:  Optional parameter for specified vhost Lower queue Make settings
Name: policy The name of the can be defined
Pattern: queue Matching pattern of(regular expression ),That is, it will match a group.
Definition: Image definition, including three parts ha-mode, ha-params, ha-sync-mode
    ha-mode:Indicates the mode of the mirror queue. Valid values are all/exactly/nodes
        all: Indicates that mirroring is performed on all nodes in the cluster
        exactly: Indicates that mirroring is performed on a specified number of nodes. The number of nodes is determined by ha-params appoint
        nodes: Indicates that the image is mirrored on the specified node, and the node name is passed ha-params appoint
    ha-params: ha-mode Parameters required by the mode
    ha-sync-mode: The synchronization method of messages in the queue. The valid values are automatic and manual
 case:
For example, the queue name is hello Mirror all the queues at the beginning and complete the mirroring on the two nodes of the cluster, policy The setting commands are: 
rabbitmqctl set_policy hello-ha "^hello" '{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}'

Then the mirror queue is set successfully.

Deployment completed

Set all queues as mirror queues, that is, the queues will be copied to each node, and the status of each node will be consistent.

3, Frequently asked questions

Common errors:

1. When starting rabbitmq with rabbitmq server - detached command, the following prompt is displayed: PID file not written- Detached was passed. At this time, rabbitmqctl status is used to prompt that the service has been started. It can be seen that this problem does not need to be solved.

2. Because the hostname file is changed, every time rabbitmqctl stop or rabbitmqctl cluster_status, etc. as long as it is rabbitmq command, an error will be reported, and the prompt is roughly as follows

Cluster status of node rabbit@web2 ...
Error: unable to connect to node rabbit@web2: nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@web2]

rabbit@web2:
  * connected to epmd (port 4369) on web2
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed

  * Hostname mismatch: node "rabbit@mq2" believes its host is different. Please ensure that hostnames resolve the same way locally and on "rabbit@mq2"


current node details:
- node name: 'rabbitmq-cli-11@web2'
- home dir: /root
- cookie hash: SGwxMdJ3PjEXG1asIEFpBg==

At this point, first ps aux | grep mq, then kill -9 the process, and then rabbitmq server - detached. (i.e. kill first and then restart)

3. After adding the virtual host - mbadmin, mbabsystem, etc., you need to restart after adding the original host - mbadmin, mbabserver, etc.

Keywords: Linux CentOS kafka RabbitMQ message queue

Added by notsleepy on Sat, 12 Feb 2022 03:53:51 +0200

Popular Keywords

©2024 Programming VIP