filebeat log collection

#(1) Download filebeat
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.0-linux-x86_64.tar.gz

#(2) build filebeat image

1) prepare the dockerfile file

# cat dockerfile
FROM  docker.io/centos
WORKDIR /usr/local
COPY filebeat-5.4.0-linux-x86_64.tar.gz  /usr/local
RUN cd /usr/local && \
        tar xvf filebeat-5.4.0-linux-x86_64.tar.gz && \
        rm -f filebeat-5.4.0-linux-x86_64.tar.gz && \
        ln -s /usr/local/filebeat-5.4.0-linux-x86_64 /usr/local/filebeat && \
        chmod +x /usr/local/filebeat/filebeat && \
        mkdir -p /etc/filebeat
ADD ./docker-entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["/usr/local/filebeat/filebeat","-e","-c","/etc/filebeat/filebeat.yml"]

2) prepare docker-entrypoint.sh

# cat docker-entrypoint.sh 
#!/bin/bash
config=/etc/filebeat/filebeat.yml
env
echo 'Filebeat init process done. Ready for start up.'
echo "Using the following configuration:"
cat /etc/filebeat/filebeat.yml
exec "$@"
[root@fs02 logs]# cat docker-entrypoint.sh 
#!/bin/bash
config=/etc/filebeat/filebeat.yml
env
echo 'Filebeat init process done. Ready for start up.'
echo "Using the following configuration:"
cat /etc/filebeat/filebeat.yml
exec "$@"

3) construction
docker build -t registry.cn-hangzhou.aliyuncs.com/wangfang-k8s/filebeat-v5.4.0:latest .
At the same time, I upload the image to alicloud
docker push registry.cn-hangzhou.aliyuncs.com/wangfang-k8s/filebeat-v5.4.0:latest
#(3) prepare resource list

1) prepare the filebeat configuration file,
This configuration file means to collect all files under the / log / directory and output them to redis

# cat configmap.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
    name: filebeat-config
data:
    filebeat.yml: |
        filebeat.prospectors:
        - type: log
            paths:
                - "/log/*"
        output.redis:
            hosts: ["192.168.1.51:6379"]
            key: "nginx-log"
            db: 2
            tag: "nginx-log"
            password: "redhat"
            port: 6379

2) prepare the deployment configuration list file of NGINX
The nginx container and the filebeat container run in a pod. The container logs of nginx are in the / var/log/nginx directory of, and the filebeat is mounted in the / log directory of, so you can see the logs of nginx in the / log directory of filebeat

# cat nginx-deployment.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
    name: nginx-test
    namespace: default
spec:
    replicas: 1
    template:
        metadata:
            labels:
                k8s-app: nginx-test
        spec:
            containers:
            - image: registry.cn-hangzhou.aliyuncs.com/wangfang-k8s/filebeat-v5.4.0
                imagePullPolicy: Always
                name: filebeat
                volumeMounts:
                - name: app-logs
                    mountPath: /log
                - name: filebeat-config
                    mountPath: /etc/filebeat/
            - image: nginx:1.11
                name : nginx-test
                imagePullPolicy: Always
                ports:
                - containerPort: 80
                volumeMounts:
                - name: app-logs
                    mountPath: /var/log/nginx
            volumes:
            - name: app-logs
                emptyDir: {}
            - name: filebeat-config
                configMap:
                    name: filebeat-config

3) prepare the resource configuration list file of nginx service

# cat service.yaml 
apiVersion: v1
kind: Service
metadata:
    name: nginx-test
spec:
    selector:
        k8s-app: nginx-test
    ports:
    - name: http
        nodePort: 38888
        port: 89
        protocol: TCP
        targetPort: 80
    type: NodePort

4) deployment
kubectl apply -f .
Normal operation of container

5) visit nginx
curl 172.30.5.3:80

6) log in to redis to check whether there is a log
Indicate if there is any problem with the log

Keywords: Linux Nginx Docker Redis

Added by simcoweb on Mon, 02 Dec 2019 07:22:28 +0200