What is canal?

canal [k ə' n æ l], which means waterway / pipeline / ditch. It is mainly used for incremental log analysis based on MySQL database to provide incremental data subscription and consumption

In the early days, Alibaba had the business requirement of cross machine room synchronization due to the deployment of dual machine rooms in Hangzhou and the United States. The implementation method was mainly to obtain incremental changes based on the business trigger. Since 2010, the business has gradually tried to obtain incremental changes through database log parsing for synchronization, resulting in a large number of database incremental subscriptions and consumption services.

Services based on log incremental subscription and consumption include

• database mirroring
• Real time database backup
• Index construction and real-time maintenance (splitting heterogeneous indexes, inverted indexes, etc.)
• Service cache refresh
• Incremental data processing with business logic

The current version of canal supporting source side MySQL includes 5.1 x , 5.5. x , 5.6. x , 5.7. x , 8.0. x

canal obtains database logs through binlog through a slave server disguised as mysql, and then forwards them

Deployment preparation

Prepare a centos server

Prepare a mysql server (or together)

Install java

yum list java-1.8*
yum install java-1.8.0-openjdk* -y

mysql server configuration

mysql needs to be configured as the main server, add server ID, and establish database users

Modify my cnf 

server-id=1  ##Server id, which is used for master-slave synchronization
log-bin=mysql-bin  ##Enable binlog
binlog_format=row  ##Set binlog mode
binlog-do-db=##Your database name is separated by commas. You can delete this line

Execute sql statements to add users

-- Log in using the command: mysql -u root -p
-- Create user user name: canal password: Canal@123456
create user 'canal'@'%' identified by 'Canal@123456';
-- to grant authorization *.*Represents all libraries
grant SELECT, REPLICATION SLAVE, REPLICATION CLIENT on *.* to 'canal'@'%' identified by 'Canal@123456';

Download canal Download address: https://github.com/alibaba/canal/releases

download

cd /tmp/canal-deployer  
wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz
tar -zvxf canal.deployer-1.1.5.tar.gz 
cd ../ 
mv canal-deployer/ /usr/local/  ##Move directory
cd /usr/local/canal-deployer/   ##Enter directory

Deploy and install canal

This article only explains the single machine deployment of canal

Configuration Item Description:

sh-4.4# tree conf/
conf/
|-- canal.properties    (System root profile)
|-- canal_local.properties  (instance Level profile, each instance A copy)
|-- example  (Database listening instance directory)
|   |-- h2.mv.db
|   |-- instance.properties 
|   `-- meta.dat
|-- logback.xml
|-- metrics
|   `-- Canal_instances_tmpl.json
`-- spring
    |-- base-instance.xml
    |-- default-instance.xml
    |-- file-instance.xml
    |-- group-instance.xml
    |-- memory-instance.xml
    `-- tsdb
        |-- h2-tsdb.xml
        |-- mysql-tsdb.xml
        |-- sql
        |   `-- create_table.sql
        `-- sql-map
            |-- sqlmap-config.xml
            |-- sqlmap_history.xml
            `-- sqlmap_snapshot.xml

6 directories, 18 files
sh-4.4#

We just need to focus on instance. Under example Properties:

# position info
canal.instance.master.address=mysql-test:3306 #mysql connection address
canal.instance.master.journal.name=   #mysql binlog start file
canal.instance.master.position=  #Location of binlog +
canal.instance.master.timestamp=
canal.instance.master.gtid=
.
.Omit content
.

# username/password
canal.instance.dbUsername=canal
canal.instance.dbPassword=123456
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

The binlog log file configuration above may not be filled in. After startup, canal will automatically obtain the final data configuration, or you can customize the starting position and then obtain the old data

Locate / bin / start. In the project SH file, execute to start the daemon:

./bin/startup.sh 
./bin/restart.sh 
./bin/stop.sh

After successful startup, you can find the logs in the logs directory to view the startup status:

sh-4.4# tree logs
logs
|-- canal canal Master service log
|   |-- canal.log  
|   `-- canal_stdout.log
`-- example  mysql Instance connection log
    |-- example.log
    `-- meta.log

2 directories, 4 files
sh-4.4#

go client connection

By introducing a small canal client, you can connect data. See the specific tutorial:

https://github.com/withlin/canal-go

Operation screenshot:

This article is an original article of xianshike. You don't need to contact me for reprint, but please indicate that it comes from xianshike blog www.php20.com cn