1. Log related concepts

1.1 function of log

• Program debugging
• Know whether the program runs normally
• Fault analysis and problem location
• User behavior analysis

1.2 log level

Grade	meaning
DEBUG	The most detailed log information. The typical application scenario is problem diagnosis
INFO	The level of detail of information is second only to DEBUG, which usually only records the information of key nodes to confirm that everything is working as expected
WARNING	Information recorded when something unexpected happens (for example, low disk free space), but the application is still running normally
ERROR	A message recorded when some functions cannot function properly due to a more serious problem
CRITICAL	Information recorded when a serious error occurs that prevents the application from continuing to run

By default, the logging module prints the log information with the level of WARNING and above to the console

1.3 there are two ways to use the logging module

The logging module can be used in two ways

• The first way is to use the module level functions provided by logging
• The second way is to use the four components of the Logging system

2 use the module level functions provided by logging

2.1 the logging module defines common functions

function	explain
logging.debug(msg,*args,**kwargs)	Create a log record with a severity of DEBUG
logging.info(msg,*args,*kwargs)	Create a log record with severity INFO
logging.warning(msg,*args,*kwargs)	Create a log record with severity WARNING
logging.error(msg,*args,*kwargs)	Create a log record with severity ERROR
logging.critical(msg,*args,**kwargs)	Create a log record with CRITICAL severity
logging.log(level,*args,*kwargs)	Create a log record with severity level
logging.basicConfig(**kwargs)	One time configuration of root logger

Here is a demonstration:

2.2 usage 1: simple configuration

import logging

logging.debug("debug message")
logging.info("info message")
logging.warning("warning message")
logging.error("error message")
logging.critical("critical message")
logging.log(level=logging.ERROR, msg = "error in logging.log function")

Output result:

WARNING:root:warning message
ERROR:root:error message
CRITICAL:root:critical message
ERROR:root:error in logging.log function

By default, the logging module prints logs to the standard output and only displays logs with WARNING level greater than or equal to, which indicates that the default log level is set to WARNING (log level critical > error > WARNING > info > debug)

2.3 usage 2: using logging Basicconfig() function

Use logging Basicconfig() function can adjust log level, output format, etc

logging.basicConfig() function description

Parameter name	describe
filename	Specify the file name of the log output target file. After specifying this setting, the log information will not be output to the console
format	Specify the log format string, that is, specify the field information contained in the log output and their order. The format fields defined by the logging module are listed below.
datefmt	Specify the date / time format. It should be noted that this option is only valid when the format contains the time field% (actime) s
level	Specifies the log level of the logger
stream	Specify the log output target stream, such as sys stdout,sys.stderr and network stream. It should be noted that stream and filename cannot be provided at the same time, otherwise ValueError exception will be thrown
style	Python3.2 newly added configuration item. Specify the style of format string. The values can be '%', '{' and '$', and the default is'% '
handlers	New configuration item added in Python 3.3. If this option is specified, it should be an iterative object that creates multiple handlers, which will be added to the rootlogger. It should be noted that only one of the three configuration items filename, stream and handlers can exist, and two or three cannot occur at the same time. Otherwise, ValueError exception will be thrown.

Format string of logging module

Field / attribute name	Use format	describe
asctime	%(asctime)s	Time of log event – human readable time, such as: 2003-07-08 16:49:45896
created	%(created)f	The time when the log event occurs – timestamp, which is the time when it is called The value returned by the time() function
relativeCreated	%(relativeCreated)d	The number of milliseconds relative to the loading time of the logging module when the log event occurs (I don't know why to use it at present)
msecs	%(msecs)d	The millisecond portion of the log event occurrence event
levelname	%(levelname)s	The log level in text form of the log record ('DEBUG ',' INFO ',' WARNING ',' ERROR ',' CRITICAL ')
levelno	%(levelno)s	The log level (10, 20, 30, 40, 50) of the digital form of the log record
name	%(name)s	The name of the logger used is "root" by default, because rootLogger is used by default
message	%(message)s	The text content of the log record is calculated by MSG% args
pathname	%(pathname)s	The full path of the source file that calls the logging function
filename	%(filename)s	The file name part of pathname, including the file suffix
module	%(module)s	The name part of filename, excluding the suffix
lineno	%(lineno)d	The line number of the source code that calls the logging function
funcName	%(funcName)s	The name of the function that called the logging function
process	%(process)d	Process ID
processName	%(processName)s	Process name, Python 3.1 NEW
thread	%(thread)d	Thread ID
threadName	%(thread)s	Thread name

# coding=utf-8
import logging

MY_FORMAT = "%(asctime)s %(name)s %(levelname)s %(pathname)s %(lineno)d %(message)s"  # Configure output log format
DATE_FORMAT = '%Y-%m-%d  %H:%M:%S %a '  #Configure the format of the output time

logging.basicConfig(
    filename="my.log",  # Specifies that the log is written to a file
    level=logging.INFO,
    datefmt=DATE_FORMAT,
    format=MY_FORMAT,
)

logging.debug("debug")
logging.info("info")
logging.warning("warning")
logging.error("error")
logging.critical("critical")

Open the file my Log, as follows:

2020-11-22  19:18:58 Sun  root INFO E:/prapy/python_project/testcase/test1.py 15 info
2020-11-22  19:18:58 Sun  root WARNING E:/prapy/python_project/testcase/test1.py 16 warning
2020-11-22  19:18:58 Sun  root ERROR E:/prapy/python_project/testcase/test1.py 17 error
2020-11-22  19:18:58 Sun  root CRITICAL E:/prapy/python_project/testcase/test1.py 18 critical

explain:

• logging.basicConfig() function is a one-time simple configuration tool, which means that it will work only when the function is called for the first time, and no operation will be generated when the function is called again later. The setting of multiple calls is not an accumulation operation.
• The Logger has a hierarchical relationship. The Logger used by the logging module level function called above is an instance of RootLogger class, whose name is' root '. It is the Logger at the top level of the Logger hierarchical relationship, and the instance exists in singleton mode.
• If the log to be recorded contains variable data, you can use a format string as the description message of the event (the first parameter of logging.debug, logging.info and other functions), and then pass the variable data as the value of the second parameter * args,

>>> import logging
>>> logging.warning('%s is %d years old.', 'Tom', 10)
WARNING:root:Tom is 10 years old.

3. Use the four components of the Logging system

Above, we learned about logging debug(),logging.info(),logging.warning(),logging.error(),logging.critical() (used to record different levels of log information respectively), logging Basicconfig() (use the default log format to establish a default stream handler for the log system, set the basic configuration (such as log level) and add it to the root logger) functions at the logging module level.

The following describes the second method of printing logs, log stream processing, using the function logging Getlogger ([name]) (returns a logger object. If no name is specified, it will return root logger).

Before introducing the log stream processing flow of the logging module, let's introduce the four components of the logging module:

Component name	Corresponding class name	Function description
Logger	Logger	Provides an interface that applications can always use
processor	Handler	Send the log record created by logger to the appropriate destination for output
filter	Filter	It provides finer grained control tools to determine which log records are output and which log records are discarded
Formatter	Formatter	Determines the final output format for logging

Description of the relationship between these components:

• The logger needs to output the log information to the target location through the handler, such as file and sys Stdout, network, etc;
• Different handler s can output logs to different locations;
• The logger can set multiple handler s to output the same log record to different locations;
• Each handler can set its own filter to filter logs, so that only interested logs can be retained;
• Each handler can set its own formatter to output the same log to different places in different formats.

To put it simply: the logger is the entry, and the real work is the handler. The handler can also filter and format the log content to be output through the filter and formatter.

Introduction to related classes and common methods of logging module

Classes related to the four components of logging: logger, handler, filter and formatter.

3.1 Logger class

The Logger object has three tasks to do:

1. Expose several methods to the application code so that the application can record log messages at run time;
2. Decide which logs to perform subsequent processing based on the log severity level (default filtering facility) or filter object;
3. Send log messages to all interested log handlers.

The most commonly used methods for Logger objects are divided into two categories: configuration methods and message sending methods

Logger class related methods

method	describe
Logger.setLevel()	Sets the minimum severity level of log messages that the logger will process
Logger.addHandler() and logger removeHandler()	Add and remove a handler object for the logger object
Logger.addFilter() and logger removeFilter()	Add and remove a filter object for the logger object

After the logger object is configured, you can use the following method to create log records:

method	describe
Logger.debug(), Logger.info(), Logger.warning(), Logger.error(), Logger.critical()	Create a log record corresponding to the level of their method name
Logger.exception()	Similar to creating a logger Log message for error()
Logger.log()	You need to get an explicit log level parameter to create a log record

What about a Logger object? One way is to create an instance of the Logger class through the instantiation method of the Logger class, but we usually use the second way - logging Getlogger () method.

logging. The getLogger() method has an optional parameter name, which indicates the name and identification of the logger to be returned. If this parameter is not provided, its value is' root '. If the getLogger() method is called multiple times with the same name parameter value, a reference to the same logger object will be returned.

Note that multiple logger s cannot be created after multiple use, otherwise the log will be output repeatedly.

Description of the hierarchy and effective level of the logger:

• The name of the logger is one with '.' Split hierarchy, each ' The following loggers are '. " The children of the previous logger, for example, have a logger named foo, and the other names are foo bar, foo. bar. Baz and foo BAM is the offspring of foo.
• Logger has a concept of "effective level". If a level is not explicitly set on a logger, the logger uses the level of its parent; If the level of its parent is not explicitly set, continue to look up the valid level of its parent, and so on until an ancestor with level explicitly set is found. It should be noted that the root logger always has a clear level setting (WARNING by default). When deciding whether to handle an event that has occurred, the validity level of the logger will be used to decide whether to pass the event to the handlers of the logger for processing.
• After processing the log messages, child loggers will pass the log messages to the handlers related to their ancestor loggers by default. Therefore, we do not need to define and configure handlers for all loggers used in an application. We only need to configure handlers for a top-level logger and then create child loggers as needed. We can also turn off this delivery mechanism by setting the propagate property of a logger to False.

3.2 Handler class

The handler object is used to distribute messages (based on the level of log messages) to the location specified by the handler (file, network, mail, etc.). The Logger object can add 0 or more handler objects for itself through the addHandler() method. For example, an application may want to implement the following logging requirements:

method	describe
Handler.setLevel(lel)	Specify the information level to be processed. Information below lel level will be ignored
Handler.setFormatter()	Choose a format for this handler
Handler.addFilter(filt),Handler.removeFilter(filt)	Add or delete a filter object

It should be noted that the application code should not directly instantiate and use the Handler instance. Because Handler is a base class, it only defines the interfaces that all handlers should have, and provides some default behaviors that subclasses can directly use or override. Here are some commonly used handlers:

Handler	describe
logging.StreamHandler	Send log messages to the output Stream, such as std.out, std.err or any file like object.
logging.FileHandler	Send log messages to disk files. By default, the file size will grow indefinitely
logging.handlers.RotatingFileHandler	Send log messages to disk files and support cutting log files by size
logging.hanlders.TimedRotatingFileHandler	Send log messages to disk files and support log file cutting by time
logging.handlers.HTTPHandler	Send log messages to an HTTP server in the form of GET or POST
logging.handlers.SMTPHandler	Send the log message to a specified email address
logging.NullHandler	This Handler instance ignores error messages, which is usually used by library developers who want to use logging to avoid the message "No handlers could be found for logger XXX".

3.3 Formater class

The Formater object is used to configure the final order, structure and content of log information. And logging Unlike the handler base class, the application code can directly instantiate the Formatter class. In addition, if your application needs some special processing behavior, you can also implement a subclass of Formatter.

The construction method of Formatter class is defined as follows:

logging.Formatter.__init__(fmt=None, datefmt=None, style='%')

It can be seen that the construction method receives three optional parameters:

• fmt: Specifies the message format string. If this parameter is not specified, the original value of message will be used by default
• datefmt: Specifies the date format string. If this parameter is not specified, "% Y -% m -% d% H:% m:% s" will be used by default
• style: a new parameter in Python 3.2, which can be taken as'% ',' {'and' $'. If this parameter is not specified,'% 'will be used by default

Generally, you can directly use logging Formatter(fmt, datefmt)

3.4 Filter class (just understand)

Filter can be used by Handler and logger to perform finer grained and more complex filtering functions than level. Filter is a filter base class, which only allows log events under a logger level to be filtered. This category is defined as follows:

class logging.Filter(name='')
    filter(record)

For example, if the value of the name parameter passed during the instantiation of a filter is' A.B ', the filter instance will only allow the log records generated by loggers with names similar to the following rules to be filtered:' A.B ',' A.B,C ',' A.B.C.D ',' A.B.D ', while the logs generated by loggers with names of' A.BB 'and' B.A.B 'will be filtered out. If the value of name is an empty string, all log events are allowed to pass the filter.

The filter method is used to specifically control whether the transmitted record can pass the filtering. If the return value of this method is 0, it means it cannot pass the filtering, and if the return value is non-0, it means it can pass the filtering.

3.5 brief process of log stream processing

1. Create a logger
2. Set the log level of the logger
3. Create an appropriate Handler(FileHandler should have a path)
4. Set the log level of each Handler
5. Create the format of the next log
6. Add the format created above to the Handler
7. Add the Handler created above to the logger
8. Printout logger debuglogger. infologger. warninglogger. errorlogger. critical

# coding=utf-8
import logging

# Create a logger. If the parameter is empty, the root logger is returned
logger = logging.getLogger("mylogger")
logger.setLevel(logging.DEBUG)  # Set logger log level

# Create handler
fh = logging.FileHandler("test.log", encoding="utf-8")
ch = logging.StreamHandler()

# Set the output log format and pay attention to logging Case of formatter
formatter = logging.Formatter(
    fmt="%(asctime)s %(name)s %(filename)s %(message)s",
    datefmt="%Y/%m/%d %X"
)

# Specify the output format for the handler, paying attention to case
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# Log processor added for logger
logger.addHandler(fh)
logger.addHandler(ch)

# Output log s of different levels
logger.warning("warning message")
logger.info("info message")
logger.error("error message")

Operation results

2020/11/22 21:00:24 mylogger test3.py warning message
2020/11/22 21:00:24 mylogger test3.py info message
2020/11/22 21:00:24 mylogger test3.py error message

python logging repeated log writing

When logging with Python's logging module, you may encounter the problem of repeatedly logging. The first record is written once, the second record is written twice, and the third record is written three times

Cause: the handler was not removed. Solution: removeHandler was removed after logging

# coding=utf-8
import logging

def log(msg):
    #Create a logger. If the parameter is empty, the root logger is returned
    logger = logging.getLogger("mylogger")
    logger.setLevel(logging.DEBUG)  #Set logger log level

    #Create handler
    fh = logging.FileHandler("test.log",encoding="utf-8")
    ch = logging.StreamHandler()

    #Set output log format
    formatter = logging.Formatter(
        fmt="%(asctime)s %(name)s %(filename)s %(message)s",
        datefmt="%Y/%m/%d %X"
        )

    #Specify the output format for the handler
    fh.setFormatter(formatter)
    ch.setFormatter(formatter)

    #Log processor added for logger
    logger.addHandler(fh)
    logger.addHandler(ch)

    # Output log s of different levels
    logger.info(msg)

# Output log s of different levels
log("message1")
log("message2")
log("message3")

Operation results

2020/11/22 21:08:04 mylogger test3.py message1
2020/11/22 21:08:04 mylogger test3.py message2
2020/11/22 21:08:04 mylogger test3.py message2
2020/11/22 21:08:04 mylogger test3.py message3
2020/11/22 21:08:04 mylogger test3.py message3
2020/11/22 21:08:04 mylogger test3.py message3

Analysis: you can see that the output results are printed repeatedly

Reason: when calling log for the second time, get the same logger according to the name in getLogger(name), and there is already a handler you added for the first time in this logger, and a handler is added for the second call. Therefore, there are two identical handlers in this logger, and so on. There will be several handlers after calling several times.

Solution 1: add removeHandler statement

# coding=utf-8
import logging


def log(msg):
    # Create a logger. If the parameter is empty, the root logger is returned
    logger = logging.getLogger("mylogger")
    logger.setLevel(logging.DEBUG)  # Set logger log level

    # Create handler
    fh = logging.FileHandler("test.log", encoding="utf-8")
    ch = logging.StreamHandler()

    # Set output log format
    formatter = logging.Formatter(
        fmt="%(asctime)s %(name)s %(filename)s %(message)s",
        datefmt="%Y/%m/%d %X"
    )

    # Specify the output format for the handler
    fh.setFormatter(formatter)
    ch.setFormatter(formatter)

    # Log processor added for logger
    logger.addHandler(fh)
    logger.addHandler(ch)

    # Output log s of different levels
    logger.info(msg)

    # Solution 1: add a removeHandler statement and remove the Handler after each use
    logger.removeHandler(fh)
    logger.removeHandler(ch)


# Output log s of different levels
log("message1")
log("message2")
log("message3")

Solution 2: make a judgment in the log method. If the logger already has a handler, no handler will be added.

# coding=utf-8
import logging


def log(msg):
    # Create a logger. If the parameter is empty, the root logger is returned
    logger = logging.getLogger("mylogger")
    logger.setLevel(logging.DEBUG)  # Set logger log level

    if not logger.handlers:
        # Create handler
        fh = logging.FileHandler("test.log", encoding="utf-8")
        ch = logging.StreamHandler()

        # Set output log format
        formatter = logging.Formatter(
            fmt="%(asctime)s %(name)s %(filename)s %(message)s",
            datefmt="%Y/%m/%d %X"
        )

        # Specify the output format for the handler
        fh.setFormatter(formatter)
        ch.setFormatter(formatter)

        # Log processor added for logger
        logger.addHandler(fh)
        logger.addHandler(ch)

    # Output log s of different levels
    logger.info(msg)


# Output log s of different levels
log("message1")
log("message2")
log("message3")

Example of logger calling method

# coding=utf-8
import logging.handlers
import datetime


def get_logger():
    logger = logging.getLogger('mylogger')  # mylogger is the name and ID of the logger. If this parameter is not provided, it defaults to 'root'
    logger.setLevel(logging.DEBUG)  # Set logger processing level

    # Judge here if logger If the handlers list is empty, add it. Otherwise, write the log directly
    if not logger.handlers:
        # rf_handler writes all log information to all Log
        # when: string, which defines the interval time unit of log segmentation
        # Interval: the number of interval time units. It refers to the number of when time to wait. The Logger will automatically rebuild the news and continue logging
        # backupCount is the number of log files to keep. The maximum number of log files is backupCount. Redundant files are deleted. The default is 0, which means they will not be deleted automatically
        rf_handler = logging.handlers.TimedRotatingFileHandler('all.log', when='midnight', interval=1, backupCount=7,
                                                               atTime=datetime.time(0, 0, 0, 0))

        # Set output log format
        rf_formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
        # Specify the output format for the handler
        rf_handler.setFormatter(rf_formatter)

        # f_ The handler writes information with a level greater than or equal to error to error Log file
        f_handler = logging.FileHandler('error.log')
        f_handler.setLevel(logging.ERROR)

        # Set output log format
        f_formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(filename)s[:%(lineno)d] - %(message)s")
        # Specify the output format for the handler
        f_handler.setFormatter(f_formatter)

        # Log processor added for logger
        logger.addHandler(rf_handler)
        logger.addHandler(f_handler)

    return logger


logger = get_logger()
logger.debug('debug message')
logger.info('info message')
logger.warning('warning message')
logger.error('error message')
logger.critical('critical message')
logger.log(level=logging.ERROR, msg="logger.log message")

reference resources: https://www.cnblogs.com/Nicholas0707/p/9021672.html#_label1_1