1, Background

ES is a distributed search framework. It can quickly query data, mainly including elastic search, kibana (a web interface used to query es data)

Where does the data come from? Of course, it comes from the database or inserts into ES one by one. So this time, the requirement is to import data from mysql database into ES.

2, Software

Version 1:
java =1.8
Elasticsearch=6.2.4
Kibana=6.2.4
Canal=1.1.4 #A component of Alibaba that synchronizes mysql database to elastic search (some are called plug-ins, actually including server and customer service, and middleware is more appropriate)
sing1ee:elasticsearch-jieba-plugin=6.4.0   # This is an ES word segmentation plug-in that needs to be modified to 6.2.4 and then packaged with gradle.


//Version 2:
java =1.8
Elasticsearch=7.7.0 
Kibana=7.7.0
Canal=1.1.4  #A component of Alibaba that synchronizes mysql database to elastic search (some are called plug-ins, actually including server and customer service, and middleware is more appropriate)
sing1ee:elasticsearch-jieba-plugin=7.7.0  # This is an ES word segmentation plug-in, which also needs to be packaged with gradel

3, Steps

1. Download the configuration Elasticsearch and run

The official website is very slow. Let's use the ladder to get down, or someone else to send it.

# Modify the file elasticsearch-7.7.0/config/elasticsearch.yml
# There are all these parameters in this file, only need to modify
cluster.name: my-application
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]

./elasticsearch-7.7.0/bin/elasticsearch    # Direct operation
./elasticsearch-7.7.0/bin/elasticsearch -d  # Background operation

2. Download and configure Kibana to run, which is essentially a web server

# This configuration file kibana-7.7.0-linux-x86_64/config/kibana.yml 
# Change to
server.host: "0.0.0.0"

#function
./bin/kibana

3. Test whether the configuration is successful. If it is unsuccessful, it depends on the log. Generally, it will succeed

http://10.90.0.10:5601/

Click a tool like button
GET /_cat/indices

4. Installation and use of canal

In essence, it is a server (deployer) and a client (adapter). That is to start the deployer and then start the adapter. Use the adapter to import mysql data.

Can al = 1.1.4 download address:

https://github.com/alibaba/canal/releases

1) Configure depolyer

# Modify the file deployer/conf/example/instance.properties
canal.instance.master.address=10.90.0.11:3306  #ip:port where your mysql database is located
canal.instance.dbUsername=root  #Users of your mysql database
canal.instance.dbPassword=123   #Users of your mysql database

2) Configure adapter

# 1. Modify the file adapter/conf/application.yml

canal.conf:
  mode: tcp # kafka rocketMQ
  canalServerHost: 127.0.0.1:11111
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://10.90.0.11: 3306 / book? Useunicode = true 񖓿 10.90.0.11 is the address book database name where mysql is located
      username: root
      password: 123
# ......
      - name: es
        hosts: 127.0.0.1:9300 # 127.0.0.1:9200 for rest mode
        properties:
          mode: rest  # It must be set to rest here, otherwise it will report something like Transform
          # security.auth: test:123456 #  only used for rest mode
          cluster.name: my-application  #elasticsearch

# Modify adapter/conf/es/mytest_user.yml
 esMapping:
  _index: index_name #The index name of your Elasticsearch. You need to build an index first to get mysql data
  _type: _doc
  _id: _id
  upsert: true
  sql: "select a.id as _id, a.title, a.content from book a"


# How to build index, on the web side of kibana

PUT /index_name
{       "mappings": {
            "_doc": {
                "properties": {
                    "title": {
                        "type": "text"
                    },
                    "content": {
                        "type": "text"
                    }
                
                }
            }
        }
}

3) Start

Enter the bin folder and run/ startup.sh  

Note: Elasticsearch6.2.4 and Elasticsearch7.7.0 are different_ doc this field

4) We've got data

Using postman to send a request, you can import the data of the book table in the book database of mysql into the name and content.

http://10.90.0.10:8081/etl/es/mytest_user.yml

4, Summary

In a word, we need to understand that elasticsearch is a similar database. canal is a CS (client server) middleware.

Mainly configure the application.yml , I understand how to link with deployer and mysql. Next, adapter/config/es/mytest_user.yml Is a task file.

Attach a problem solving website

https://www.jianshu.com/p/041038a0f916