Log management

1. What does the log component do

• Logs can do a lot of things. For engineers learning programs and testing, logs can locate and solve problems. They are the biggest function point.
  
• Record all logs to help us record what the program function has done. Whether it is normal input and output or abnormal, you can use logs to record
  
• Locating problem logs can help programmers debug problems and help testers locate problems
  
• Record and analyze user behavior statistical analysts are used to record user behavior and analyze user habits and business value
  
• Backup and restore real-time data database engineers use it as a special database
• ...

2. Common log framework

• Log4j Apache Log4j is a Java based logging tool. It was pioneered by Ceki G ü lc ü and is now a project of the Apache Software Foundation. Log4j is one of several Java logging frameworks
• Log4j 2 Apache Log4j 2 is a log4j upgrade product developed by apache
• Commons Logging is a project of Apache foundation. It is a set of Java logging interfaces, formerly called Jakarta Commons Logging, and later renamed Commons Logging
• Slf4j is similar to common logging. It is a set of simple java logging facade. There is no logging implementation. (Simple Logging Facade for Java, abbreviated as slf4j)
• Logback is the implementation of a set of log components (slf4j camp).
• Jul (Java Util Logging), the official log implementation since Java 1.4.

Common log framework: logback+sl4j

By adding the corresponding libraries to the classpath, you can activate various logging systems, and then provide appropriate configuration files in the root directory of the classpath to further customize the logging system. The configuration files can also be specified through the logging.config attribute of the Spring Environment.

The log framework in Java is mainly divided into two categories: log facade and log implementation.

Log facade: the log facade defines a set of log interface specifications, which does not provide the underlying specific implementation logic. Apache Commons Logging and Slf4j fall into this category.

Log implementation: log implementation is the specific implementation of log, including log level control, log print format and log output form (output to database, output to file, output to console, etc.). Log4j, Log4j2, Logback and Java Util Logging fall into this category.

Separating the log facade from the log implementation is actually a typical facade mode. This way allows specific businesses to switch freely between different log implementation frameworks without changing any code. Developers only need to master the API of the log facade.

3. Configuration file

Log system	Custom configuration
Logback	logback-spring.xml,logback-spring.groovy,logback.xml,logback.groovy
Log4j	log4j.properties or log4j.xml
Log4j2	log4j2-spring.xml or log4j2.xml
JDK	(Java Util Logging) logging.properties

1. If possible, it is recommended that you use the - Spring variant to define the log configuration (for example, use logback-spring.xml instead of logback.xml). If you use the standard configuration path, Spring may not have complete control over log initialization.

2.Java Util Logging will cause some known class loading problems when running from executable jar. We recommend not using it as much as possible.

3. The God combination of lombok uses lombok's @Slf4j annotation to save the tedious task of configuring the declaration log and improve the development efficiency.

4. Log level and usage

The log level from low to high is: trace < debug < info < warn < error < fatal

The default level is INFO, so logs below INFO level will not be printed

<!--In most cases, it is cooperation Slf4j use-->
//Logger and LoggerFactory import the org.slf4j package
private final static Logger logger = LoggerFactory.getLogger(Xxxx.class);
//Logger and LogManager import the org.apache.logging package
private static final Logger LOG = LogManager.getLogger(Xxxx.class); 
   

5. Log output

Usually, the log is stored on disk in the form of text stream, or it can be stored in relational database or No Sql

• text
  
• Relational database
  

<!--logbakck Add configuration to file-->
<appender name="mysql" class="ch.qos.logback.classic.db.DBAppender">
        <connectionSource class="ch.qos.logback.core.db.DataSourceConnectionSource">
            <dataSource class="com.alibaba.druid.pool.DruidDataSource">
                <url>jdbc:mysql://localhost:3306/log?useUnicode=true&amp;characterEncoding=utf-8&amp;useSSL=false&amp;serverTimezone=GMT%2B8</url>
                <driverClassName>com.mysql.cj.jdbc.Driver</driverClassName>
                <username>root</username>
                <password>xxxxxx</password>
            </dataSource>
        </connectionSource>
</appender>

<!--database ddl sql-->
BEGIN;
DROP TABLE IF EXISTS logging_event_property;
DROP TABLE IF EXISTS logging_event_exception;
DROP TABLE IF EXISTS logging_event;
COMMIT;
 
BEGIN;
CREATE TABLE logging_event 
  (
    timestmp         BIGINT NOT NULL,
    formatted_message  TEXT NOT NULL,
    logger_name       VARCHAR(254) NOT NULL,
    level_string      VARCHAR(254) NOT NULL,
    thread_name       VARCHAR(254),
    reference_flag    SMALLINT,
    arg0              VARCHAR(254),
    arg1              VARCHAR(254),
    arg2              VARCHAR(254),
    arg3              VARCHAR(254),
    caller_filename   VARCHAR(254) NOT NULL,
    caller_class      VARCHAR(254) NOT NULL,
    caller_method     VARCHAR(254) NOT NULL,
    caller_line       CHAR(4) NOT NULL,
    event_id          BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY
  );
COMMIT;


 
 
BEGIN;
CREATE TABLE logging_event_property
  (
    event_id       BIGINT NOT NULL,
    mapped_key        VARCHAR(254) NOT NULL,
    mapped_value      TEXT,
    PRIMARY KEY(event_id, mapped_key),
    FOREIGN KEY (event_id) REFERENCES logging_event(event_id)
  );
COMMIT;
 
 
BEGIN;
CREATE TABLE logging_event_exception
  (
    event_id         BIGINT NOT NULL,
    i                SMALLINT NOT NULL,
    trace_line       VARCHAR(254) NOT NULL,
    PRIMARY KEY(event_id, i),
    FOREIGN KEY (event_id) REFERENCES logging_event(event_id)
  );
COMMIT;

• No Sql
  
• Console console
  
• General log components can customize the output format
  

6. Log configuration

Create the logback-spring.xm file. It is generally placed in the resources directory

to configure:
error level is reserved for three days, and other levels are reserved for one day

Log file format: yyyy MM DD - ${I} - ${level}. Log

The default level is info. If you want to modify it, configure it in the configuration file:

logging:
    level:
        org.springframework.web: debug

<?xml version="1.0" encoding="UTF-8"?>
<!--  The log level is from low to high TRACE < DEBUG < INFO < WARN < ERROR < FATAL，If set to WARN，Is lower than WARN No information will be output  -->
<!--  scan:When this property is set to true If the configuration document changes, it will be reloaded. The default value is true  -->
<!--  scanPeriod:Set the time interval for monitoring whether the configuration document is modified. If no time unit is given, the default unit is milliseconds.
       When scan by true This property takes effect when. The default interval is 1 minute.  -->
<!--  debug:When this property is set to true When, it will be printed out logback Internal log information, real-time viewing logback Running status. The default value is false.   -->
<configuration scan="true" scanPeriod="60 seconds" debug="false">
    <contextName>logback</contextName>
    <!--  name The value of is the name of the variable, value The value defined by the variable. Values defined by are inserted into the logger In context. After definition, you can make“ ${}"To use variables.  -->
    <springProperty scope="context" name="log.path" source="log.path"/>
    <!-- 0. Log format and color rendering  -->
    <!--  Rendering classes dependent on color logs  -->
    <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter"/>
    <conversionRule conversionWord="wex"
                    converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter"/>
    <conversionRule conversionWord="wEx"
                    converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter"/>
    <!--  Color log format console log format: with color  -->
    <property name="CONSOLE_LOG_PATTERN1"
              value="${CONSOLE_LOG_PATTERN:-%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>
    <!--Normal log format-->
    <property name="CONSOLE_LOG_PATTERN2" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n"/>
    <!--dev Environment configuration-->
    <springProfile name="dev">
        <!--Output to console-->
        <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
            <!--This log appender It is used for development. Only the lowest level is configured. The log level output from the console is log information greater than or equal to this level-->
            <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
                <level>debug</level>
            </filter>
            <encoder>
                <Pattern>${CONSOLE_LOG_PATTERN1}</Pattern>
                <charset>utf-8</charset>
            </encoder>
        </appender>
        <root level="info">
            <appender-ref ref="console"/>
        </root>
    </springProfile>


    <!--Output to debug-->
    <appender name="debug" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!--  The rolling strategy of the logger, recording by date and by size  -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log.path}/%d{yyyy-MM-dd}-%i-debug.log</fileNamePattern>
            <cleanHistoryOnStart>true</cleanHistoryOnStart>
            <!-- Log document retention days -->
            <maxHistory>1</maxHistory>
            <!--
                      When the log file exceeds maxFileSize The specified size is. Scroll according to the log files above. Note the configuration here SizeBasedTriggeringPolicy Scrolling by file size is not possible and must be configured timeBasedFileNamingAndTriggeringPolicy
                      -->
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>100MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
        <append>true</append>
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN2}</pattern>
            <charset>utf-8</charset>
        </encoder>
        <filter class="ch.qos.logback.classic.filter.LevelFilter"><!-- Print only DEBUG journal -->
            <level>DEBUG</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
    </appender>

    <!--Output to info-->
    <appender name="info" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!--  The rolling strategy of the logger, recording by date and by size  -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log.path}/%d{yyyy-MM-dd}-%i-info.log</fileNamePattern>
            <cleanHistoryOnStart>true</cleanHistoryOnStart>
            <maxHistory>1</maxHistory>
            <!--
                      When the log file exceeds maxFileSize The specified size is. Scroll according to the log files above. Note the configuration here SizeBasedTriggeringPolicy Scrolling by file size is not possible and must be configured timeBasedFileNamingAndTriggeringPolicy
                      -->
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>10MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
        <append>true</append>
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN2}</pattern>
            <charset>utf-8</charset>
        </encoder>
        <filter class="ch.qos.logback.classic.filter.LevelFilter"><!-- Print only INFO journal -->
            <level>INFO</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
    </appender>
    <!--Output to warn-->
    <appender name="warn" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log.path}/%d{yyyy-MM-dd}-%i-warn.log</fileNamePattern>
            <cleanHistoryOnStart>true</cleanHistoryOnStart>
            <maxHistory>1</maxHistory>
            <!--
                      When the log file exceeds maxFileSize The specified size is. Scroll according to the log files above. Note the configuration here SizeBasedTriggeringPolicy Scrolling by file size is not possible and must be configured timeBasedFileNamingAndTriggeringPolicy
                      -->
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>20MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
        <append>true</append>
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN2}</pattern>
            <charset>utf-8</charset>
        </encoder>
        <filter class="ch.qos.logback.classic.filter.LevelFilter"><!-- Print only WARN journal -->
            <level>WARN</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
    </appender>
    <!--Output to error-->
    <appender name="error" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log.path}/%d{yyyy-MM-dd}-%i-error.log</fileNamePattern>
            <cleanHistoryOnStart>true</cleanHistoryOnStart>
            <maxHistory>3</maxHistory>
            <!--
                      When the log file exceeds maxFileSize The specified size is. Scroll according to the log files above. Note the configuration here SizeBasedTriggeringPolicy Scrolling by file size is not possible and must be configured timeBasedFileNamingAndTriggeringPolicy
                      -->
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>10MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
        <append>true</append>
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN2}</pattern>
            <charset>utf-8</charset>
        </encoder>
        <filter class="ch.qos.logback.classic.filter.LevelFilter"><!-- Print only ERROR journal -->
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
    </appender>


    <!--
      root Node is a required node. It is used to specify the most basic log output level. There is only one node level attribute
      level:Used to set the print level, regardless of case: TRACE, DEBUG, INFO, WARN, ERROR, ALL and OFF，
      Cannot be set to INHERITED Or synonyms NULL. The default is DEBUG
      Can contain zero or more elements that identify this appender Will be added to this logger. 
     -->
    <root level="info">
        <appender-ref ref="debug"/>
        <appender-ref ref="info"/>
        <appender-ref ref="warn"/>
        <appender-ref ref="error"/>
    </root>
    <!--test pre Environments: exporting to documents-->
    <springProfile name="pre">
        <root level="info">
            <appender-ref ref="debug"/>
            <appender-ref ref="info"/>
            <appender-ref ref="warn"/>
            <appender-ref ref="error"/>
        </root>
    </springProfile>
    <!--4.2 production environment :Output to document -->
    <springProfile name="pro">
        <root level="info">
            <appender-ref ref="debug"/>
            <appender-ref ref="info"/>
            <appender-ref ref="warn"/>
            <appender-ref ref="error"/>
        </root>
    </springProfile>

</configuration>

7. Usage

1. Copy the logback-spring.xml file above to the resource folder
   
2. Modify the startup script start.sh and specify the log.path directory

nohup java  -Dlog.path="${logs_dir}" -Dserver.port="${port}" -Dlog.home="${logs_dir}" -Dapp.name="${appname}" -Dspring.profiles.active="${profile}" -Dspring.datasource.default.password="${pwd_ds1}" -Xms2g -Xmx2g -jar ${lib_dir}/${jarname} ${classname} > "${logs_dir}/${appname}.out" 2>&1 &