In ES, we can set Index Template and Dynamic Template to better manage and set indexes and mapping for us.

1, Index Template

For example, we need to use es for log management. We all know that the amount of log data is very large. If a single index is used to save all log data, there may be some performance problems. We can automatically generate the index by day or month. At this time, we can use the IndexTemplate to provide better performance for the index and ES cluster.

Index Template: it can help you set Mapping and Setting, and automatically match to the newly created index according to certain rules.

• Template is only used when an index is created. Modifying the template will not affect the created index.
• You can set multiple index templates, and the properties of multiple templates will be merged together
• You can specify the value of order to control the merging process.

When an index is created, the referenced template rule is as follows:

1. Apply ES default settings and mappings
2. First apply the settings in the index template with a low order value
3. If different index templates have the same attribute settings, the index template with high order will be applied, that is, the index template with low order will be overwritten
4. When creating an index, if the user specifies some properties of settings and mappings, these properties will be subject to those set by the user, and the properties not assigned to will be subject to the matching template.

Example 1

For example, let's create and give the default template now. If the created index does not specify fragmentation, the current template configuration shall prevail.

put _template/template_default
{
  "index_patterns":["*"],
  "order":0,
  "version":1,
  "settings":{
    "number_of_shards":1,
    "number_of_replicas":1
  }
}

Then we create an index without specifying fragment information and obtain the information of the index.

put template_index
--------------------
get template_index
--------------------
{
  "template_index" : {
    "aliases" : { },
    "mappings" : { },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "template_index",
        "creation_date" : "1640524317827",
        "number_of_replicas" : "1",
        "uuid" : "e5zbL0_lQUyYWFkDqmx5xw",
        "version" : {
          "created" : "7160099"
        }
      }
    }
  }
}

You can see that the index has automatically generated the partition configuration according to the default template we set.

Example 2

Create another template. The matching rule is: if the name of the index starts with template, set the number of fragments to 1 and the number of copies to 3.
date_detection: mapping setting: if the string matches the date type, it will be automatically matched as the date. true is on and false is off
numeric_detection: mapping setting. If the string is a number, the number type will be mapped automatically.

put _template/template_test
{
  "index_patterns":["template*"],
  "order":1,
  "settings":{
    "number_of_shards":1,
    "number_of_replicas":3
  },
  "mappings":{
    "date_detection":false,
    "numeric_detection":true
  }
}

We create an index that matches the wildcard rules to get information. You can see that the template is matched as follows_ Test, because template_ The order of default is 0, so use template_ The split copy of test shall prevail.

put template_index2
---------------------
get template_index2
---------------------
{
  "template_index2" : {
    "aliases" : { },
    "mappings" : {
      "date_detection" : false,
      "numeric_detection" : true
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "template_index2",
        "creation_date" : "1640524532090",
        "number_of_replicas" : "3",
        "uuid" : "QsP_eZuhQKWiaGvF_6ux4A",
        "version" : {
          "created" : "7160099"
        }
      }
    }
  }
}

Let's insert a document and get mapping to see if ES automatically converts dates and numbers.

put template_index2/_doc/1
{
  "date":"2021-12-01T00:00:00.000Z",
  "num":"10000"
}
-----------------------------------
{
  "template_index2" : {
    "mappings" : {
      "date_detection" : false,
      "numeric_detection" : true,
      "properties" : {
        "date" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "num" : {
          "type" : "long"
        }
      }
    }
  }
}

You can see that because we set "date_detection":false,"numeric_detection":true. Therefore, even if the date type is matched, the text field type will be given, and the numeric type will be matched to the long type.

Example 3

Let's create an index matching the template, specify the number of slices and copies of setting, and see if it will be subject to our setting.

put template_index3
{
    "settings":{
    "number_of_shards":2,
    "number_of_replicas":2
  }
}
-----------------
{
  "template_index3" : {
    "aliases" : { },
    "mappings" : {
      "date_detection" : false,
      "numeric_detection" : true
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "2",
        "provided_name" : "template_index3",
        "creation_date" : "1640524899092",
        "number_of_replicas" : "2",
        "uuid" : "nW7wNjmmQaqjgasWG8_7zQ",
        "version" : {
          "created" : "7160099"
        }
      }
    }
  }
}

From the mapping information of the index, you can see that the current index matches the template_test is the template, but since we specified the information of fragmentation and replica when creating the index, the information specified by us shall prevail.

2, Dynamic Template

We talked about the Index Template above, which is mainly used to determine whether the currently created index conforms to the unified configuration rules of the template when creating the index, so as to apply the configuration information in the template to our newly created index, which is usually applied to the index scenarios where the same configuration is to be generated, such as log data management, unified index management, etc.

The Dynamic Template is mainly applied to specific indexes. In the mapping setting of an index, the field type will be dynamically set by matching some set rules according to the data type we set.

For example, the string matches all, and the fields beginning with Keyword and is are set to boolean and long_ The fields at the beginning are set to long... We can set the template dynamic matching, and set our custom field types for the matched fields.

Example 1

For example, when we create an index, we need to give the subfield of the field name in the index to copy_to a new field full_name for query, but the address attribute in name is not required.

put template_mapping
{
  "mappings":{
    "dynamic_templates":[
      {
        "full_name":{
          "path_match":"name.*",
          "path_unmatch":"*.address",
          "mapping":{
            "type":"text",
            "copy_to":"full_name"
          }
        }
      }
    ]
  }
}

Then we insert a document

put template_mapping/_doc/1
{
  "name":{
    "first":"Course",
    "middle":"large",
    "last":"Handsome",
    "address":"Shanghai Tangchen Yipin"
  }
}

Get the mapping automatically generated by ES according to the Dynamic Template we set.

get template_mapping/_mapping
---------------------------
{
  "template_mapping" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "full_name" : {
            "path_match" : "name.*",
            "path_unmatch" : "*.address",
            "mapping" : {
              "copy_to" : "full_name",
              "type" : "text"
            }
          }
        }
      ],
      "date_detection" : false,
      "numeric_detection" : true,
      "properties" : {
        "full_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "name" : {
          "properties" : {
            "address" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "first" : {
              "type" : "text",
              "copy_to" : [
                "full_name"
              ]
            },
            "last" : {
              "type" : "text",
              "copy_to" : [
                "full_name"
              ]
            },
            "middle" : {
              "type" : "text",
              "copy_to" : [
                "full_name"
              ]
            }
          }
        }
      }
    }
  }
}

And because of fullill_ Name is by name* Data copy in_ To, so we can search it, but in the_ This field does not exist in srouce.

get template_mapping/_search
{
  "query":{
    "match":{
      "full_name": "Course"
    }
  }
}
------------------------
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "template_mapping",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.2876821,
        "_source" : {
          "name" : {
            "first" : "Course",
            "middle" : "large",
            "last" : "Handsome",
            "address" : "Shanghai Tangchen Yipin"
          }
        }
      }
    ]
  }
}