RocketMQ Series Quick Start

brief introduction

Apache RocketMQ is a high-performance and high-throughput distributed messaging middleware open source by Alibaba, After Ali donated it to the Apache Software Foundation, it hatched into a Top-Level Project of Apache (top level project, TLP). RocketMQ is implemented based on Ali's closed source MetaQ kernel. It serves Ali's various systems, especially in the field of e-commerce. RocketMQ is developed in Java language, so it is compared with other popular message middleware in the industry, such as RabbitMQ (developed using Erlang) and Kafka (developed using Scala), it is more friendly to Java programmers and more conducive to developers to encapsulate and expand on its underlying basis.
The official website of RocketMQ is , according to the introduction on the official website, Apache RocketMQ ™ It is a standardized message engine and lightweight data processing platform. It has the following advantages:

  • Low latency: under high concurrency pressure, more than 99.6% of the response latency is within 1 millisecond
  • Financial oriented: the tracking and audit functions supported by the system are highly available
  • Industry developability: it ensures trillion level message capacity
  • Vendor neutral: a new distributed messaging and streaming standard has been opened in the latest version 4.1
  • Big data friendly: batch transmission with general integration function supports massive throughput
  • Massive accumulation: as long as enough disk space is given, messages can be accumulated without performance loss

RocketMQ birth background

In the early days, Alibaba was based on ActiveMQ 5.0 X (lower than 5.3) constructs distributed message middleware and uses it for asynchronous communication, search, social network activity flow, data pipeline and even transaction process of their multinational companies. With the growth of trade business throughput, the pressure from message cluster has become an urgent problem to be solved.

Why design RocketMQ?

With the growth of queues and the use of virtual topics, the ActiveMQ IO module has encountered a bottleneck. Although Ali tries its best to solve this problem through current limiting, fusing or degradation, the effect is poor. Therefore, at that time, they began to focus on the popular messaging solution Kafka. Unfortunately, Kafka cannot meet Ali's requirements, especially in terms of low latency and high reliability. See here For more information.
In this case, they decided to invent a new messaging engine to deal with a wider set of use cases - from the traditional publish / subscribe scheme to the mass real-time zero loss tolerant trading system.

RocketMQ vs. ActiveMQ vs. Kafka

The following table shows RocketMQ, ActiveMQ, and Kafka (according to awesome-java For example, Apache RocketMQ is the most popular messaging solution:

Message Oriented MiddlewareActiveMQKafkaRocketMQ
CLIENT SDKJava,. NET, C + +, etcJava, Scala, etcJava, C++, Go
Agreements and specificationsPush model, support OpenWire, STOMP, AMQP, MQTT, JMSPull model, support TCPPull model, support TCP, JMS, OpenMessaging
Sequential messageIndependent consumers or queues can ensure orderEnsure the order of messages within the partitionIt can ensure that the messages are strictly sorted and can be well extended
Timing messagesupportI won't support itsupport
Batch messageI won't support itSupport with asynchronous producerYes, use asynchronous mode to avoid message loss
Broadcast messagesupportI won't support itsupport
Message filteringsupportYes, you can use Kafka flow to filter messagesSupport, attribute filter expression based on SQL92
Server triggered message redeliveryI won't support itI won't support itsupport
Message storageJDBC and high-performance logs (such as level dB and kahaDB) are used to support very fast persistence operationsHigh performance file storageHigh performance and low latency file storage
Message traceabilitysupportThis feature is supported by an offset indicatorTwo indicators, timestamp and offset, are supported
Message Priority supportI won't support itI won't support it
High availability and failoverYes, depending on the storage. If you use kahadb, you need a ZooKeeper serverYes, a ZooKeeper server is requiredSupport, master-slave mode, no other tools are required
Message trackingI won't support itI won't support itsupport
to configureThe default configuration is low-level, and the user needs to optimize the configuration parametersKafka uses key values to configure the format. These values can be provided from a file or programmaticallyOut of the box, users only need to pay attention to a small number of configurations
Management and operation toolssupportSupport, use terminal commands to display core indicatorsSupport, rich Web and terminal commands can display core indicators

Quick Start

This section will introduce how to quickly install a RocketMQ on the local machine for sending and receiving messages. More details can be accessed View.

Pre requirements

Before installing RokcetMQ, you need to install the following software:

  1. 64 bit operating system, Linux/Unix/Mac operating system is recommended
  2. 64 bit JDK 1.8 and above
  3. Maven 3.2.x
  4. Git
  5. Reserve at least 4G disk space for the Broker server

Download, install and use


1. Install jdk
  1. First in oracle official website Download the compressed package for Linux jdk 8

  2. Then use the ftp tool to upload the downloaded compressed package to the Linux server

  3. Use tar -zxvf jdk-8u261-linux-x64 tar. GZ command to extract the file

  4. Configure system environment variables

[root@node01 opt]# vi /etc/profile

Add the following at the end of the file:

# jdk drop directory
# jre drop directory
# Configure the path environment variable to separate:
# Configure the classpath environment variable to separate:
# Setting environment variables
  1. Make changes to the configuration file effective
[root@node01 opt]# source /etc/profile
  1. Check the jdk version. So far, the jdk has been installed
[root@node01 opt]# java -version
java version "1.8.0_226"
Java(TM) SE Runtime Environment (build 1.8.0_226-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.226-b11, mixed mode)
2. Install maven
  1. Download maven using wget
[root@node01 opt]# wget
  1. decompression
[root@node01 opt]# tar -zxvf apache-maven-3.6.3-bin.tar.gz 
  1. Add alicloud image
[root@node01 opt]# cd apache-maven-3.6.3/conf/
[root@node01 conf]# ll
 Total consumption 20
drwxr-xr-x. 2 root root  4096 11 July 2019 logging
-rw-r--r--. 1 root root 10468 11 July 2019 settings.xml
-rw-r--r--. 1 root root  3747 11 July 2019 toolchains.xml
[root@node01 conf]# vi settings.xml 

In the configuration file settings Add the following code to XML:

    <name>aliyun maven</name> 
  1. Configure environment variables
[root@node01 conf]# vi /etc/profile

Add the following configuration to the file:

# maven installation directory
export M2_HOME
[root@node01 opt]# source /etc/profile
  1. Check the maven version. So far, maven has been installed
[root@node01 conf]# mvn -v
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /opt/apache-maven-3.6.3
Java version: 1.8.0_226, vendor: Oracle Corporation, runtime: /opt/jdk1.8.0_226/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1127.el7.x86_64", arch: "amd64", family: "unix"
3. Install rocketmq

1. Download the rocketmq source package

[root@node01 opt]# wget

2. Decompression

[root@node01 opt]# yum install -y unzip
[root@node01 opt]# unzip 

3. Go to the unzipped file directory for compilation

[root@node01 rocketmq-all-4.7.1-source-release]# mvn -Prelease-all -DskipTests clean install -U

4. Move the compiled file to another location

[root@node01 rocketmq-all-4.7.1-source-release]# cd distribution/target/rocketmq-4.7.1
[root@node01 rocketmq-4.7.1]# ll
 Total consumption 4
drwxr-xr-x. 6 root root 4096 8 September 29-15:42 rocketmq-4.7.1
[root@node01 rocketmq-4.7.1]# mv rocketmq-4.7.1/ /opt

5. Start the Name Server. If the following information is displayed on the terminal, it indicates that the startup is successful:

[root@node01 opt]# cd rocketmq-4.7.1/bin/
[root@node01 bin]# ls       dledger      mqbroker            mqbroker.numanode1  mqnamesrv      mqshutdown.cmd     mqadmin      mqbroker.cmd        mqbroker.numanode2  mqnamesrv.cmd       runserver.cmd  mqadmin.cmd  mqbroker.numanode0  mqbroker.numanode3  mqshutdown     play.cmd        runbroker.cmd   tools.cmd
[root@node01 bin]# ./mqnamesrv
Java HotSpot(TM) 64-Bit Server VM warning: Using the DefNew young collector with the CMS collector is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
The Name Server boot success. serializeType=JSON

6. Start the broker
When you start the broker with the command, you will find the following error:

[root@node01 bin]# ./mqbroker -n localhost:9876
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000005c0000000, 8589934592, 0) failed; error='Cannot allocate memory' (errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/rocketmq-4.7.1/bin/hs_err_pid9514.log

From the error message, we can know that the reason is that the jvm startup initialization memory allocation is greater than the physical memory.
Therefore, we can modify the jvm parameters in the startup script to solve this problem. First, let's modify the startup script of nameserver:

[root@node01 bin]# vi 

Locate the configuration line that allocates jvm memory:

JAVA_OPT="${JAVA_OPT} -server -Xms4g -Xmx4g -Xmn2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

Modify it to the following configuration (the specific configuration value is adjusted according to the actual production scenario):

JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

Then modify the startup script of the broker:

[root@node01 bin]# vi 

Similarly, find the configuration line that allocates jvm memory:

JAVA_OPT="${JAVA_OPT} -server -Xms8g -Xmx8g -Xmn4g"

Modify it to the following configuration (the specific configuration value is adjusted according to the actual production scenario):

JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

Restart nameserver and broker after modification. When the following message appears, the broker has been started successfully:

[root@node01 bin]# ./mqbroker -n localhost:9876
The broker[node01,] boot success. serializeType=JSON and name server is localhost:9876

7. Execute the test program to test message sending and receiving
Before sending / receiving messages, we need to tell the client the location of the Name Server. RocketMQ provides a variety of ways to achieve this goal. For simplicity, we set the environment variable NAMESRV_ADDR. Add the following configuration in / etc/profile:

export NAMESRV_ADDR=localhost:9876

Send a message using the test program:

./ org.apache.rocketmq.example.quickstart.Producer

If the following information is displayed, the message is sent successfully:

Start another session and use the test program to receive the message:

./ org.apache.rocketmq.example.quickstart.Consumer

If the following information is displayed, the message is received successfully:

8. Close
Close broker:

[root@node01 bin]# ./mqshutdown broker
The mqbroker(1502) is running...
Send shutdown request to mqbroker(1502) OK

Close name server:

[root@node01 bin]# ./mqshutdown namesrv
The mqnamesrv(1444) is running...
Send shutdown request to mqnamesrv(1444) OK


This article mainly describes the installation of RocketMQ under windows 10 operating system. Please ensure that PowerShell is installed in your operating system. Like Linux, you need to install jdk and maven before installing RocketMQ on windows. This article will not describe how to install jdk and maven in detail. Please refer to the installation materials yourself.

1. Install RocketMQ
  1. First, in Official website Download the binary compressed package of RocketMQ, and then select a local directory for decompression
  2. Configure environment variables and add the following two environment variables:

    Open runbroker CMD, modify the following configuration line:

The modified configuration line is as follows:

rem set "JAVA_OPT=%JAVA_OPT% -cp "%CLASSPATH%"
  1. Start name server:
  2. Start broker:When the console displays the following information, it indicates that the startup is successful:
  3. Use the test program to send / receive messages
    Send message:Receive message:
  4. close
    Just close cmd/powershell directly (do not do this in the production environment)

Problems that may be encountered when the installation starts

1. When compiling, the package cannot find the prompt 502 error on the mirror

Cause: bad network or maven warehouse server error
Try again, or change to another image warehouse

2. Failed to start the broker with Cannot allocate memory error

This problem has been described above and will not be repeated here

3. Start the broker successfully, but prompt: Failed to obtain the host name

Cause: the current host name cannot be resolved
Add a map in / etc/hosts

# Configure ip address to hostname mapping node-01
4. Date calibration

Install ntpdate

yum install ntpdate

Console rocketmq console

Compile and install

1. Download

Download address of compiled source package:
Chinese guide

2. Upload the source package to the server and unzip it
[root@node01 opt]# unzip 
3. Compilation
[root@node01 opt]# cd rocketmq-externals/rocketmq-console/
[root@node01 rocketmq-console]# mvn clean package -Dmaven.test.skip=true
4. Start

After successful compilation, execute rocketmq-console-ng-2.0 in the rocketmq-console/target directory 0.jar
Start up (make sure your name server has been started). You can add the nameserver address dynamically directly, or you can configure the nameserver address by editing the configuration file in the rocketmq console / SRC / main / resources directory and adding the rocketmq.config.namesrvAddr property.

[root@node01 rocketmq-console]# ll
 Total dosage 68
drwxr-xr-x. 3 root root  4096 8 October 30:50 doc
-rw-r--r--. 1 root root 30422 8 October 30:50 LICENSE
-rw-r--r--. 1 root root   180 8 October 30:50 NOTICE
-rw-r--r--. 1 root root 10593 8 October 30:50 pom.xml
-rw-r--r--. 1 root root  2390 8 October 30:50
drwxr-xr-x. 4 root root  4096 8 October 30:50 src
drwxr-xr-x. 3 root root  4096 8 October 30:50 style
drwxr-xr-x. 7 root root  4096 8 October 30:59 target
[root@node01 rocketmq-console]# cd target/
[root@node01 target]# java -jar rocketmq-console-ng-2.0.0.jar --rocketmq.config.namesrvAddr=localhost:9876 

The startup is successful when the following interface appears:

After successful startup, you can access the server port 8080.

Interface composition

RocketMQ is cluster oriented, which can be reflected in the interface of RocketMQ console.

Operation and maintenance

  • NameSvrAddrList: configure the address of the name server, which is a list. Therefore, you can configure the addresses of multiple name servers, which indicates that the name server can be deployed in a cluster
  • IsUseVIPChannel: whether to use VIPChannel

  • Broker TOP 10: view the message volume (total) of the top 10 brokers with the largest message volume
  • Broker 5min trend: view the 5-minute trend of broker message volume
  • Topic TOP 10: view the message volume (total amount) of 10 single topics with the largest message volume
  • Topic 5min trend: view the trend of topic message volume for 5 minutes

  • View the distribution of clusters
    • Relationship between cluster and broker
    • Brokers included in the cluster
  • View broker specific information / operation information / status information
  • View broker configuration information

  • All topics are displayed and can be filtered by topic name
  • Filter general / retry / dead letter / system topics
  • New / updated topics
    • clusterName: cluster name
    • brokerName: host name
    • topicName: topic name
    • Writequeuenum: number of write queues
    • readQueueNums: number of read queues
    • perm: 2 is write, 4 is read, and 6 is read and write
  • Status query message delivery status (which broker s / which queue s / how many messages are delivered, etc.)
  • Route view the route of the message (which brokers will you send the message of this topic to and the queue information of the corresponding broker)
  • CONSUMER management (which group s have subscribed to and consumed this topic, and what is the consumption status)
  • Topic configuration (view / change the current topic configuration)
  • Send a message (send a test message to this topic)
  • Reset the consumption location (online and offline, but check whether the reset is successful)
  • Delete the topic (it will delete the topic configuration and routing information on all broker s and namesrv. Please do this carefully in the production environment)

  • All consumption groups are displayed and can be filtered by group name
  • Refresh page / refresh page every five seconds
  • Sort by subscription group / quantity / TPS / delay
  • Add / update consumption group
    • clusterName: cluster name
    • brokerName: host name
    • groupName: the name of the consumption group
    • consumeEnable: whether consumption is allowed. If it is set to FALSE, consumption will not be allowed
    • consumeBroadcastEnable: can I broadcast consumption
    • Retryqueuenum: the size of the retry queue
    • brokerId: where is the normal consumption
    • Which broker when consumption slowly: what's the problem
  • Terminal: online consumption client viewing, including version subscription information and consumption mode
  • Consumption details: view the consumption details of the corresponding consumption group, the consumption of all topics subscribed by this consumption group, and the consumption client corresponding to each queue (including Retry messages)
  • Configuration: view / change the configuration of the consumption group
  • Delete: delete the consumption group on the specified broker (caution)

  • Show the online message producer client (host, version, address and other information), which can be filtered by group name
  • Filter by topic

  • Query according to Topic and time interval (due to the large amount of data, only 2000 items will be displayed at most, and more will be ignored)
  • Query according to Message Key and Topic
    • Only 64 will be displayed at most
  • Query messages according to Message Id and Topic
  • Message details: display the details of this message and view the consumption of the message corresponding to the specific consumption group (if abnormal, you can view the specific exception information). You can resend the message to the specified consumption group.
Message track

  • Query according to Message Key and Topic
    • Only 64 will be displayed at most
  • Query messages according to Message Id and Topic
  • Message track details: displays the details of the message track of this message

reference material

Keywords: Java Back-end

Added by codeDV on Sat, 18 Dec 2021 07:28:00 +0200