Optimizing rabbitmq

In this paper, we will mainly talk about some optimization of RabbitMQ cluster, because there are many problems in the business system of the company because the MQ cluster can not be connected recently. This paper mainly summarizes some solutions:

1. Architecture of RabbitMQ Cluster:

_Our company's RabbitMQ cluster architecture was designed and implemented by me, and at that time it also referenced a lot of architecture documents on the Internet; 2 LVS servers, 3 MQ servers.Two are memory nodes and one is disk nodes; the LVS server implements master-slave through keepalived.VIP addresses are on the primary server, and LVS distributes client requests and loads them through Round-Robin round-robin policy.Load balancers can be either LVS or haproxy. I used Haproxy for load balancing when I first started.Later, because Haproxy does not support source Ip display, it does not know which client connected to each MQ management page, so it changed to LVS.

2. RabbitMQ cluster optimization:

1. Modify the heartbeat heartbeat timeout:

_Heatbeat is often used to detect whether the opposite end of the communication is alive (crash is abnormal because the socket connection is not properly closed).The basic principle is to detect whether the data on the corresponding socket connection is sent or received properly. If there is no data sent or received for a period of time, a heartbeat detection package is sent to the opposite side. If there is no response for a period of time, the heartbeat timeout is considered, that is, the opposite side may be abnormal crash.rabbitmq is no exception, heatbeat is used between client and server to detect whether the other end is normal, that is, whether the tcp link between client and server is normal.

1. Settings on the server side:

_Heatbeat is often used to detect whether the opposite end of the communication is alive (crash is abnormal because the socket connection is not properly closed).The basic principle is to detect whether the data on the corresponding socket connection is sent or received properly. If there is no data sent or received for a period of time, a heartbeat detection package is sent to the opposite side. If there is no response for a period of time, the heartbeat timeout is considered, that is, the opposite side may be abnormal crash.rabbitmq is no exception, heatbeat is used between client and server to detect whether the other end is normal, that is, whether the tcp link between client and server is normal.
_Heatbeat detection interval can be configured by adding configuration item {heartbeat,Timeout} to the configuration file rabbitmq.config, where Timeout specifies the interval in seconds.

[
{rabbit, [{tcp_listeners, [56720]}, {heartbeat,300}, {loopback_users, []}, {vm_memory_high_watermark, 0.6}]},
{rabbitmq_management, [{listener, [{port, 56721}]},{loopback_users, []}]},
{rabbitmq_tracing, [{username, "guest"}, {password, "guest111"}]}
].

2. JAVA client settings:

ConnectionFactory cf = new ConnectionFactory();
// set the heartbeat timeout to 60 seconds
cf.setRequestedHeartbeat(60);

2. Configure log tracking for rabbitmq:

_Sometimes when we connect rabbitmq, there are some problems; for example, the producer sends a message to the mq cluster, but the consumer does not receive the message.Then the producer project group said there was no problem with its configuration, and the consumer said there was no problem with its configuration.At this time, you need to turn on the log tracking of rabbitmq, real-time view the specific log sending content, and troubleshoot business problems; because the default application log of rabbitmq is very simple, only simple connection IP, connection of that vhost information, etc. Open as follows:

rabbitmq-plugins enable rabbitmq_tracing

3. Migration of MQ messages:

_This is the case when we use rabbitmq.Because we have multiple data centers, which are in different cities.To reduce network latency for dedicated lines, we deployed a set of MQ clusters in each data center.For some reasons, some businesses need to migrate from data center A to data center B. I need to create MQ accounts and Vhost s in data center B. When migrating, if there are unexpended messages in the queue, I also need to migrate them together.How to migrate messages? You still need a plugin for rabbitmq.

# Start by opening the plug-in in the MQ cluster of the source
rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_shovel_management
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management

4. Backup and restore of MQ configuration:

_MQ backup can be done in a variety of ways, you can backup the information of a Vhost (including queue, EXCHANGE, Bingding Key, etc.) through the web management page, backup Vhost can restore the individual Vhost configuration only if the target MQ cluster needs to create users, vhost, authorization, etc. in advance; you can also backup the configuration trust of the entire MQInformation, used to restore an empty MQ cluster; if there is already a configuration within the target MQ cluster, it is appended rather than overwritten;

_But this is manual. We should backup the configuration information of MQ on a daily basis, because you do not know which day the MQ cluster will fail. I wrote a shell script to backup the configuration information of MQ cluster on a daily basis.The main purpose of the script is to backup the configuration of MQ every day, and when the backup is complete, an email will be sent to me to tell me if the backup is successful; the script is as follows:

#!/bin/bash
#
RABBITMQ_HOME=/usr/local/rabbitmq_server-3.6.9
PATH=$PATH:$RABBITMQ_HOME/sbin
export RABBITMQ_HOME PATH
ERLANG_HOME=/usr/local/erlang
PATH=$PATH:$ERLANG_HOME/bin
export ERLANG_HOME PATH
RABBITMQADMIN_HOME=/sbin
PATH=$PATH:$RABBITMQADMIN_HOME
export RABBITMQADMIN_HOME PATH

env_ip() {
array2=(
192.168.20.232
192.168.22.94
192.168.0.242
192.168.1.251
192.168.0.21
192.168.0.210
192.168.74.200
)
if [ $(ifconfig |grep eno16777984|wc -l) -eq 1 ];then
    mq_server_ip=${array2[1]}
    env_name=uat
    env="Small and Small Finance UAT Environmental Science MQ colony"
elif [ $(ifconfig eth1|awk 'NR==2 {print $2}'|awk -F ":" '{print $2}'|grep 192.168.20|wc -l) -eq 1 ];then
    mq_server_ip=${array2[0]}
    env_name=sit
    env="Small and Small Finance SIT Environmental Science MQ colony"
elif [ $(ifconfig eth1|awk 'NR==2 {print $2}'|awk -F ":" '{print $2}'|grep 192.168.0|wc -l) -eq 1 ];then
    mq_server_ip=${array2[2]}
    env_name=WHPRD01
    env="Small and Small Finance Wuhan PRD Environment First Set MQ colony"
elif [ $(ifconfig eth1|awk 'NR==2 {print $2}'|awk -F ":" '{print $2}'|grep 192.168.1|wc -l) -eq 1 ];then
    mq_server_ip=${array2[3]}
    env_name=WHPRD02
    env="Small and Small Finance Wuhan PRD Environment Set 2 MQ colony"
elif [ $(ifconfig eth1|awk 'NR==2 {print $2}'|awk -F ":" '{print $2}'|grep 192.168.0|egrep -v '[0-9][0-9][0-9]'|wc -l) -eq 1 ];then
    mq_server_ip=${array2[4]}
    env_name=XXPRD
    env="Small and Small Finance Shenzhen PRD Environmental Science MQ colony"
elif [ $(ifconfig eth1|awk 'NR==2 {print $2}'|awk -F ":" '{print $2}'|grep 192.168.0|egrep  '[0-9][0-9][0-9]'|wc -l) -eq 1 ];then
    mq_server_ip=${array2[5]}
    env_name=XXTTPRD
    env="Small and Small Finance Shenzhen Small and Independent PRD Environmental Science MQ colony"
elif [ $(ifconfig -a|grep Bcast|awk -F "[ :]+" '{print $4}'|grep 192.168.74|wc -l) -eq 1 ];then
    mq_server_ip=${array2[6]}
    env_name=XXTTUAT
    env="Small and Small Finance Shenzhen Small and Independent UAT Environmental Science MQ colony"

fi
}
env_ip
mqadmin_user="admin"
mqadmin_pw="admin"
mqadmin_port="56721"
mq_config_backup_file="/data/mq_config_bak/${mq_server_ip}_mqconfigbak_$(date +%Y%m%d%H%M%S)"
if [ ! -d /data/mq_config_bak ];then
mkdir -p /data/mq_config_bak
fi
if [ "`ls -A /data/mq_config_bak`" != "" ];then
   rm -rf /data/mq_config_bak/*
fi

/sbin/rabbitmqadmin -H ${mq_server_ip} -P ${mqadmin_port} -u ${mqadmin_user} -p ${mqadmin_pw} export ${mq_config_backup_file} > /dev/null 2>&1
if [ "`ls -A /data/mq_config_bak`" != "" ];then
   scp -P 10022 -r /data/mq_config_bak/* root@192.168.20.28:/data/mq_config_bak/ &>/dev/null
   ssh -p 10022 root@192.168.20.28 /bin/sh /.scripts/send_mqconfigbak.sh ${mq_server_ip} ${env}  &>/dev/null

fi

#!/bin/bash
#
echo > /tmp/queues.txt
echo > /tmp/queues2.txt
VHOST2=$(rabbitmqctl list_vhosts|grep -v "Listing vhosts")
for i in ${VHOST2[@]}
        do
          rabbitmqctl list_queues -p $i|grep -v "Listing queues"|awk '{$1="'${i}' "$1;print}' >> /tmp/queues.txt
        done
cat /tmp/queues.txt|sort -rn -k3|sed '/^$/d' > /tmp/queues2.txt
file2="/tmp/queues2.txt"
cat $file2|while read line
    do
             queues_number=$(echo $line|awk '{print $NF}')
         if [ $queues_number -gt 5000 ];then
             queues_name=$(echo $line|awk '{print $2}')
             vhost_name=$(echo $line|awk '{print $1}')
             echo "$(date +%F-%H:%M:%S) the vhost ${vhost_name} include the queues ${queues_name} message number is ${queues_number} ,Because the number of messages exceeds 5000, the messages will be cleared" >> /tmp/purge_message.txt
             /usr/local/rabbitmq_server-3.6.9/sbin/rabbitmqctl purge_queue -p ${vhost_name} ${queues_name} >/dev/null 2>&1
         fi
    done

7. Provide a single-machine version of the MQ installation script:

#!/bin/bash
#
yum  -y install make gcc gcc-c++ kernel-devel m4 unixODBC unixODBC-devel openssl openssl-deetc
yum  -y install ncurses-devel
XXet -O /software/otp_src_19.0.tar.gz ftp://xxjrftp:Pass123$%^@192.168.20.27:9020/software/rabbitmq/otp_src_19.0.tar.gz
XXet -O /software/wxWidgets-3.0.2.tar.bz2 ftp://xxjrftp:Pass123$%^@192.168.20.27:9020/software/rabbitmq/wxWidgets-3.0.2.tar.bz2
XXet -O /software/rabbitmq-server-generic-unix-3.6.9.tar ftp://xxjrftp:Pass123$%^@192.168.20.27:9020/software/rabbitmq/rabbitmq-server-generic-unix-3.6.9.tar
cd /software
tar xzvf otp_src_19.0.tar.gz
cd otp_src_19.0
./configure --prefix=/usr/local/erlang --without-javac
value=$(echo $?)
if [ ${value} -eq 0 ];then
    make && make install
fi
echo -e "ERLANG_HOME=/usr/local/erlang\nPATH=\$PATH:\$ERLANG_HOME/bin\nexport ERLANG_HOME PATH\n" >>/etc/profile.d/erlang.sh
source /etc/profile.d/erlang.sh
yum -y install gtk2-devel binuti-devel mesa-libGL-devel mesa-libGLU-devel
cd /software && tar jxvf wxWidgets-3.0.2.tar.bz2  && cd wxWidgets-3.0.2
./configure --with-opengl --enable-debug --enable-unicode
value=$(echo $?)
if [ ${value} -eq 0 ];then
    make && make install
fi
cd /software
tar xf rabbitmq-server-generic-unix-3.6.9.tar
mv /software/rabbitmq_server-3.6.9 /usr/local
cd /usr/local
ln -sv rabbitmq_server-3.6.9   rabbitmq_server
echo -e "RABBITMQ_HOME=/usr/local/rabbitmq_server\nPATH=\$PATH:\$RABBITMQ_HOME/sbin\nexport RABBITMQ_HOME PATH\n" >>/etc/profile.d/rabbitmq.sh
source /etc/profile.d/rabbitmq.sh
XXet -O /etc/init.d/rabbitmq-server ftp://xxjrftp:Pass123$%^@192.168.20.27:9020/software/rabbitmq/rabbitmq-server
chmod +x /etc/init.d/rabbitmq-server
cd /etc/init.d
chkconfig --add rabbitmq-server
chkconfig rabbitmq-server on
mkdir -p /logs/rabbitmq
mkdir -p /var/run/rabbitmq
ln -s /usr/local/erlang/bin/erl /usr/bin/erl
XXet -O /usr/local/rabbitmq_server-3.6.9/etc/rabbitmq/enabled_plugins ftp://xxjrftp:Pass123$%^@192.168.20.27:9020/software/rabbitmq/enabled_plugins
XXet -O /usr/local/rabbitmq_server-3.6.9/etc/rabbitmq/rabbitmq.config ftp://xxjrftp:Pass123$%^@192.168.20.27:9020/software/rabbitmq/rabbitmq.config
service rabbitmq-server start
sleep 15
source /etc/profile
rabbitmqctl add_user admin 11111
rabbitmqctl set_user_tags admin administrator
rabbitmqctl change_password guest 222222
systemctl stop iptables
systemctl disable iptables

For more details, please pay attention to my personal WeChat public number "IT Operations and Maintenance in the Cloud Age". This public number is designed to share new technologies and trends in Internet Operations and Maintenance, including consulting from the IT Operations and Maintenance industry and sharing of Operations and Maintenance technical documents.Focus on devops, jenkins, zabbix monitoring, kubernetes, ELK, the use of various middleware, such as redis, MQ; shell and python, and other operations and maintenance programming languages; I have been engaged in IT operations and maintenance related work for more than 10 years.In 2008, I started working on Linux/Unix system operation and maintenance; I have a certain understanding of operation and maintenance related technology.All posts on this public number are summaries of my actual work experience and are basically original posts.I would like to share my experience, experience and technology with you!I hope to grow and progress with you along the IT operation and maintenance career path.