如何扩展python的logging组件支持json日志输出

这两天在优化公司一个python的项目,顺便研究了一下如何将python日志转成json格式,原来在Java的项目中搞过类似的事情,知道日志转成json之后有很多便利的之处,最常见的就是可以直接对接各种日志分析系统,如开源的ELK,将数据导入之后就能快速的进行查询和分析,方便做各种统计,监控或报警等。

python里面的logging组件,其实已经是组件化了,有Logger组件,Handler组件,Fomatter组件,如下图所示:

logger=>handler=>formatter分别是一对多的关系,日志的格式其实是由formatter决定的,所以想要扩展成你想要的各种格式,就重写定制formatter组件就可以了,它实际上和Java里面Log4j的LayOut组件类似。

核心代码如下:

REMOVE_ATTR = ["filename", "module", "exc_text", "stack_info", "created", "msecs", "relativeCreated", "exc_info", "msg"]


class JSONFormatter(logging.Formatter):
    host_name, host_ip = HostIp.get_host_ip()

    def format(self, record):
        extra = self.build_record(record)
        self.set_format_time(extra)  # set time
        self.set_host_ip(extra)  # set host name and host ip
        extra['message'] = record.msg  # set message
        if record.exc_info:
            extra['exc_info'] = self.formatException(record.exc_info)
        if self._fmt == 'pretty':
            return json.dumps(extra, indent=1, ensure_ascii=False)
        else:
            return json.dumps(extra, ensure_ascii=False)

    @classmethod
    def build_record(cls, record):
        return {
            attr_name: record.__dict__[attr_name]
            for attr_name in record.__dict__
            if attr_name not in REMOVE_ATTR
        }

    @classmethod
    def set_format_time(cls, extra):
        now = datetime.datetime.utcnow()
        format_time = now.strftime("%Y-%m-%dT%H:%M:%S" + ".%03d" % (now.microsecond / 1000) + "Z")
        extra['@timestamp'] = format_time
        return format_time

    @classmethod
    def set_host_ip(cls, extra):
        extra['host_name'] = JSONFormatter.host_name
        extra['host_ip'] = JSONFormatter.host_ip

使用的时候,可以通过简单配置即可:

[loggers]
keys=root

[handlers]
keys=console,file

[formatters]
keys=json,json_pretty

[logger_root]
level=DEBUG
;handlers=console,file,rotate
handlers=console,file

[handler_console]
class=StreamHandler
level=INFO
formatter=json_pretty
args=(sys.stderr,)


[handler_file]
class=FileHandler
level=INFO
formatter=json
args=('log/base_conf.log','a','utf-8')

[handler_rotate]
class=handlers.TimedRotatingFileHandler
level=INFO
formatter=json
args=('log/rotate.log', 'D',1,0,'utf-8')

[formatter_json]
class=format.json_formatter.JSONFormatter

[formatter_json_pretty]
format=pretty
class=format.json_formatter.JSONFormatter

如下的一段异常代码:

fileConfig('log_conf.ini')
    log = logging.getLogger(__name__)
    try:
        a = 1 / 0
    except Exception:
        log.exception(" occurred exception ")

输出结果如下:

{
 "name": "__main__",
 "args": [],
 "levelname": "ERROR",
 "levelno": 40,
 "pathname": "C:/texx.py",
 "lineno": 17,
 "funcName": "base_configuration",
 "thread": 10608,
 "threadName": "MainThread",
 "processName": "MainProcess",
 "process": 11916,
 "@timestamp": "2019-01-10T12:50:20.392Z",
 "host_name": "your-PC",
 "host_ip": "192.168.10.11",
 "message": " occurred exception ",
 "exc_info": "Traceback (most recent call last):\n  File \"C:/txxx.py\", line 14, in base_configuration\n    a = 1 / 0\nZeroDivisionError: division by zero"
}

可以看到内容非常详细,并且组件还支持自定义字段的加入,在收集到日志系统上,可以非常的方便检索统计。

详细的解释和代码,可以fork我的github:https://github.com/qindongliang/python_log_json

原文发布于微信公众号 - 我是攻城师(woshigcs)

原文发表时间:2019-01-10

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

扫码关注云+社区