Filebeat custom index name

1. Load external profile

1.1 Input config

filebeat.config.inputs:
  enabled: true
  path: inputs.d/*.yml

inputs. Example of configuration file in D Directory:

- type: log
  paths:
    - /var/log/mysql.log
  scan_frequency: 10s

- type: log
  paths:
    - /var/log/apache.log
  scan_frequency: 5s

1.2 Module config

filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml

Example of external profile:

- module: apache
  access:
    enabled: true
    var.paths: [/var/log/apache2/access.log*]
  error:
    enabled: true
    var.paths: [/var/log/apache2/error.log*]

2,Elasticsearch output

1.2 template loading

By default, if Elasticsearch output is enabled, Filebeat will automatically load the recommended template file fields YML, if you want to use the default index template, no other configuration is required. Otherwise, you can use Filebeat Change the default value in the YML configuration file to:

Load different templates
```
setup.template.name: "your_template_name"
setup.template.fields: "path/to/fields.yml"
```
If the template already exists, it will not be overwritten unless you configure Filebeat.
Overwrite existing template
```
setup.template.overwrite: true
```
Disable automatic template loading
```
setup.template.enabled: false
```
If automatic template loading is disabled, you need to load the template manually.
Change index name

If you are sending events to a cluster that supports index lifecycle management, see configuring index lifecycle management to learn how to change the index name.

When the index name is fileyy-y7.0 or filebey-y7.0 by default, it is not supported MM. DD, where yyyy MM. DD is the date of the event index. To use a different name, set the index option in the Elasticsearch output. The value you specify should include the root name of the index, version and date information. You also need to configure setup template. Name and setup template. Pattern option to match the new name, for example:
```
output.elasticsearch.index: "customname-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "customname"
setup.template.pattern: "customname-*"
```
If you are using a pre built Kibana dashboard, please also set up setup dashboards. Index option, for example:
```
setup.dashboards.index: "customname-*"
```

2.2 custom index name

2.2.1 configuring filebeat

If we do not configure it, the index in the following format will be generated by default, and if it is detected, it will always use this date by default

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  
# Index format for output to es
filebeat-8.0.0-rc1-2022-01-20

From the above configuration, we can see that the data will be sent to es, but only to the index of filebeat - * of ES, which is obviously not what we want.

The index name of the event to be written when using the default index. The default is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}", for example, "filebeat-8.0.0-rc1-2022-01-20". If you change this setting, you also need to configure setup template. Name and setup template. Pattern option.

To send to the specified index, we use filebeat The changed configuration of YML is as follows:

# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access*
  fields:
    source: access
- type: log
  paths:
    - /var/log/nginx/error*
  fields:
    source: error
# ============================== Filebeat modules ==============================
filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
setup.template.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: true
setup.ilm.enabled: false
#  ============================== Elasticsearch Output ==============================
output.elasticsearch:
  hosts: ["192.168.197.130:9200"]
  # The index prefix nginx here matches the pattern of the template, and the middle string is set to field Source variable. If the following indexes format does not match, the index format will be used
  index: "nginx-%{[fields.source]}-*"  
  indices:
  # The prefix nginx here matches the pattern of the template, with field in the middle The specific value of source, which is the field of the previous input When the source value matches here, the index is set to the defined format
    - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}" 
      when.equals:                                            
      fields:
          source: "access"
    - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.equals:
        fields:
          source: "error"
# ============================== Processors ==============================
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

when.contains: contain
when.equals: be equal to

# Meaning of relevant template fields:
setup.template.name: "nginx"      # Set a new template, the name of the template
setup.template.pattern: "nginx-*" # The template matches those indexes. Here, all indexes starting with nginx are represented
setup.template.enabled: false     # Turn off the default template configuration
setup.template.overwrite: true    # Open the newly set template
setup.ilm.enabled: false      # The ILM function of index lifecycle management is enabled by default. When it is enabled, the index name can only be filebeat - *, through setup ilm. Enabled: false to close; If you want to use a custom index name and need to enable ILM, you can modify the template of filebeat

2.2.2 check es whether a new index has been added

2.2.3 associate es index on kibana

Click [management] - [Stack Management]

The index name is in the form of wildcards, so that the logs collected by the index library beginning with nginx access can be aggregated and displayed in the future. Otherwise, the index needs to be created every month or even every day

It can be seen that there are fewer fields to select than the original one, because it is a custom template, so the fields only have the contents defined by us. The default template will add all supported fields