经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 程序设计 » Python » 查看文章
Python 正则表达式实战之Java日志解析
来源：cnblogs　　作者：授客　　时间：2022/1/17 11:11:22　　对本文有异议

需求描述

基于生产监控告警需求，需要对Java日志进行解析，提取相关信息，作为告警通知消息的内容部分。

提取思路

具体怎么提取，提取哪些内容呢？这里笔者分析了大量不同形态的生产日志，最后总结出4种形态，如下，制定出以下提取逻辑。

形态1

上图中，款选部分即为要提取的主要内容，即异常发生时所在文件，代码行，自定义异常相关描述，异常类型，异常描述，这里提取的相关说明和异常描述将统一作为异常的详细描述

形态2

类似形态1，如果没有独占一行的“异常类型”，那就取最后Caused by:后面的异常类型，及其描述

形态3

形态1，形态2不匹配的情况下，匹配形态3，该形态中，异常类型和描述是包含在自定义异常相关描述里面的

形态4

前三者都不匹配的情况下，匹配最后这种形态。没有异常类型，仅日志级别“ERROR”可以标识它是条异常日志。

代码实现

  1. #!/usr/bin/env python
  2. #-*- coding:utf-8 -*-
  5. import re
  8. log_list = [
  9. '''
  10. 2021-10-18 09:22:41,079:ERROR http-nio-9330-exec-4 (DirectJDKLog.java:181) - Servlet.service() for servlet [dispatcherServlet] in context with path [/finance] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause
  11. java.lang.NullPointerException
  12. 	at java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:469) ~[?:1.8.0_202]
  13. 	at java.util.TreeMap.put(TreeMap.java:552) ~[?:1.8.0_202]
  14. ''',
  15. '''
  16. 2021-10-18 09:22:55,222:WARN kafka-async-consumer-2 (FeignClientsErrorDecoder.java:43) - read Exception failed!
  17. com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
  18.     at [Source: java.io.InputStreamReader@743333a3; line: 1, column: 0]
  19. 	at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270) ~[jackson-databind-2.8.4.jar!/:2.8.4]
  20. ''',
  21. '''
  22. 2021-10-18 09:22:52,975:ERROR [,] parallel-2 (AccessLogWebFilter.java:60) - [accessId=616ccc49ff642e00010a4e8c] 发生网关内部错误
  23. org.springframework.web.server.ResponseStatusException: 504 GATEWAY_TIMEOUT "Response took longer than timeout: PT35S"; nested exception is org.springframework.cloud.gateway.support.TimeoutException: Response took longer than timeout: PT35S
  24. 	at org.springframework.cloud.gateway.filter.NettyRoutingFilter.lambda$filter$5(NettyRoutingFilter.java:211) ~[spring-cloud-gateway-core-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
  25. 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
  26. 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
  27. Caused by: org.springframework.cloud.gateway.support.TimeoutException: Response took longer than timeout: PT35S
  28.     at
  29. ''',
  30. '''
  31. 2021-10-18 09:22:41,905:WARN http-nio-8080-exec-60 (VehicleOeImpl.java:1000) - 批量更新第三方价格失败1---->
  32. org.springframework.jdbc.BadSqlGrammarException:
  33. ### Error updating database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Query was empty
  34. ### The error may involve com.cmall.ec.webapp.maindata.web.dao.vehicleOe.ThirdPartyOeMapper.updateList-Inline
  35. ### The error may involve com.cmall.ec.webapp.maindata.web.dao.vehicleOe.ThirdPartyOeMapper.updateList-Inline
  36. ### The error occurred while setting parameters
  37. ### SQL:
  38. ### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Query was empty
  39. ; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Query was empty
  40. 	at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:91)
  41. 	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
  42. 	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  43. 	at java.lang.Thread.run(Thread.java:748)
  44. Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Query was empty
  45. 	at java.lang.reflect.Method.invoke(Method.java:498)
  46. 	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:434)
  47. 	... 70 more
  48. ''',
  49. '''
  50. 2021-10-17 18:39:33,066:ERROR http-nio-10062-exec-34 TID: 962118fb93d345bc92af98499ad0f771.3235.16344671730621817 (DirectJDKLog.java:181) - Servlet.service() for servlet [dispatcherServlet] in context with path [/orders/seller] threw exception [Request processing failed; nested exception is com.cmall.commons.service.exception.HttpMessageException: [400]标名为空] with root cause
  51. Exception: 标名为空
  52. 	at com.cmall.commons.utils.Assert.fail(Assert.java:553) ~[icec-cloud-commons-0.4.5.jar!/:?]
  53. 	at com.cmall.commons.utils.Assert.notBlank(Assert.java:112) ~[icec-cloud-commons-0.4.5.jar!/:?]
  54. ''',
  55. '''
  56. 2021-10-18 09:22:23,849:ERROR http-nio-10030-exec-2 TID: ed41cdfb8d5d4953a713285802c56032.80.16345201436864709 (DicountAssembler.java:266) - 查询商品类优惠结果失败:DiscountProductRequest(companyId=IYl6MgdkiG9KoBMYUJo, userLoginId=5c385563ad996c47bf5f7ccd, provinceGeoId=CN-43, cityGeoId=284)
  57. feign.FeignException: status 500 reading DiscountProductClient#listDiscountProducts(DiscountProductRequest); content:
  58. {"timestamp":1634520143844,"status":500,"error":"Internal Server Error","exception":"com.cmall.commons.service.exception.HttpMessageException","message":"优惠前的不含税价格不能为空","path":"/discountPromotion/listDiscountProducts"}
  59. 	at feign.FeignException.errorStatus(FeignException.java:62) ~[feign-core-9.3.1.jar!/:?]
  60. 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
  61. ''',
  62. '''2021-10-18 09:22:23,849:ERROR [-]  (DicountAssembler.java:266) - task supervisor threw an exception ....''',
  63. '''
  64. 2021-10-18 09:13:13,940:ERROR kafka-async-consumer-9 (ConsumeSupport.java:104) - kafka消费失败, cid:616cca245cc0a90001ead690, message:{"jsonMessageType":"com.cmall.ec.cloud.scheduletask.values.kafka.command.DistributedDelayCommand","id":"616cca24d52d1d00010531dd","usage":"EVENT","service":"schedule-task-service","topic":"prod-quote-command-delay","timeStamp":1634519588985,"delayTimes":16983,"producerTaskId":"B21101807668","inquiryId":"B21101807668","resolveBatchId":"616cca019cf7b70001c35477","resolveIds":["616cca00d52d1d000138be9a"],"type":"AUTO","retryCount":2,"producer":"DistributedDelayCommand"}, cause:java.lang.reflect.InvocationTargetException
  65. 	at sun.reflect.GeneratedMethodAccessor4946.invoke(Unknown Source)
  66. 	at java.lang.reflect.Method.invoke(Method.java:498)
  67. 	at java.lang.Thread.run(Thread.java:748)
  68. Caused by: com.cmall.messagebus.exception.MessageBaseException: 系统自动报价失败-->org.springframework.dao.DuplicateKeyException:
  69. ### Error updating database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'
  70. ### The error may involve defaultParameterMap
  71. ### The error occurred while setting parameters
  72. ### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'
  73. ; SQL []; Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'
  74. 	at com.cmall.ec.cloud.quotation.handler.QuotationAllocationHandler.intelligentAutoQuoteDispatcher(QuotationAllocationHandler.java:80)
  75. 	... 9 more
  76. ''',
  77. '''
  78. 2021-10-16 14:37:19,951:ERROR DiscoveryClient-1 (TimedSupervisorTask.java:79) - task supervisor threw an exception
  79. java.lang.OutOfMemoryError: Java heap space
  80. ''',
  81. '''2021-10-16 14:37:19,951:ERROR DiscoveryClient-1 (TimedSupervisorTask.java:79) - task supervisor threw an exception
  82. java.lang.OutOfMemoryError: Java heap space
  83. at''',
  84. '''
  85. 2021-10-23 14:03:00,785:ERROR kafka-async-consumer-1 (ConsumeSupport.java:104) - kafka消费失败, cid:6173a593f90d4200010ce3fe, message:{"jsonMessageType":"com.cmall.ec.cloud.events.order.OrderSent","id":"6173a593cb769e0001402a3b","usage":"EVENT","service":"orders-service","topic":"prod-order","timeStamp":1634968979938,"orderId":"S2110230001835","shipmentType":"LOGISTICS","logisticsCompany":"快送","shipmentNum":"","shipGroupId":"6173a59355c35d00014c390c","remark":"","productStoreId":"SZQD0001","userLoginId":"5e0aa6649cf5260001336437","username":"许庆杰","items":[{"productId":"00001","quantity":1}],"isLogisticsFeePayOnLine":true}, cause:java.lang.reflect.InvocationTargetException
  86. 	at sun.reflect.GeneratedMethodAccessor1909.invoke(Unknown Source)
  87. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  88. 	at java.lang.reflect.Method.invoke(Method.java:498)
  89. 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  90. 	at java.lang.Thread.run(Thread.java:748)
  91. Caused by: com.baomidou.mybatisplus.exceptions.MybatisPlusException: Error: Cannot execute insertBatch Method. Cause
  92. 	at com.baomidou.mybatisplus.service.impl.ServiceImpl.insertBatch(ServiceImpl.java:137)
  93. 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
  94. 	... 9 more
  95. Caused by: org.apache.ibatis.exceptions.PersistenceException:
  96. ### Error flushing statements.  Cause: org.apache.ibatis.executor.BatchExecutorException: com.cmall.ec.cloud.service.dao.mapper.ShipFeeMapper.insert (batch index #1) failed. Cause: java.sql.BatchUpdateException: Duplicate entry 'SF2110230005145' for key 'PRIMARY'
  97. ### Cause: org.apache.ibatis.executor.BatchExecutorException: com.cmall.ec.cloud.service.dao.mapper.ShipFeeMapper.insert (batch index #1) failed. Cause: java.sql.BatchUpdateException: Duplicate entry 'SF2110230005145' for key 'PRIMARY'
  98. 	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
  99. 	... 51 more
  100. Caused by: org.apache.ibatis.executor.BatchExecutorException: com.cmall.ec.cloud.service.dao.mapper.ShipFeeMapper.insert (batch index #1) failed. Cause: java.sql.BatchUpdateException: Duplicate entry 'SF2110230005145' for key 'PRIMARY'
  101. 	at java.lang.reflect.Method.invoke(Method.java:498)
  102. 	... 52 more
  103. Caused by: java.sql.BatchUpdateException: Duplicate entry 'SF2110230005145' for key 'PRIMARY'
  104. 	at sun.reflect.GeneratedConstructorAccessor1540.newInstance(Unknown Source)
  105. 	... 61 more
  106. Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'SF2110230005145' for key 'PRIMARY'
  107. 	at sun.reflect.GeneratedConstructorAccessor1537.newInstance(Unknown Source)
  108. 	... 71 more
  109. ''',
  110. '''
  111. 2021-10-25 16:29:15,853:ERROR reactor-http-epoll-3 (CompositeLog.java:122) - 500 Server Error for HTTP POST "/job-service/api/registry"
  112. reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
  113. ''',
  114. '''2021-11-06 07:09:04,781:WARN http-nio-10011-exec-85 (AdminService.java:97) - 任务执行失败, code:500, reason:java.lang.OutOfMemoryError: Java heap space''',
  115. '''2022-01-08 13:46:30,668:ERROR http-nio-9524-exec-5 (WaitSettleDealServiceImpl.java:147) - 添加退订单到账单失败com.cmall.commons.service.exception.HttpMessageException: [411]退货单添加失败，该分组已经结束对账，无法添加退货单'''
  116. ]
  119. exception_match_pattern_list = [
  120.     ':(ERROR|WARN) .+\s\(([^\s]+?\.java):(\d+)\)(.*)\n([^:\s>\u4e00-\u9fa5]*Exception|[^:\s>\u4e00-\u9fa5]*Error)(.*?)(\s+at\s|$)',
  121.     ':(ERROR|WARN) .+\s\(([^\s]+?\.java):(\d+)\).*([^\n]*Caused by: )([^:\s>]*?Exception|[^:\s]*?Error)(.*?)(\s+at\s*|$)',
  122.     ':(ERROR|WARN) .+\s\(([^\s]+?\.java):(\d+)\)([^\n]*?)([^:\s>\u4e00-\u9fa5]*Exception|[^:\s>\u4e00-\u9fa5]*Error)([^\n]*?)(\s+at\s|$)',
  123.     ':(ERROR|WARN) .+\s\(([^\s]+?\.java):(\d+)\)(.*?)\n*?\s*?([^:\s>\u4e00-\u9fa5]*Exception|[^:\s>\u4e00-\u9fa5]*Error)*?(.*?)(\s+at\s|$)'
  124. ]
  126. for log_index, log in enumerate(log_list):
  127.     flag = 0
  128.     for pattern_index, flag_pattern in enumerate(exception_match_pattern_list):
  129.         match_result = re.findall(flag_pattern, log, re.DOTALL)
  130.         if match_result:
  131.             print('匹配第%s个Pattern' % (pattern_index+1), '匹配结果：', match_result[0])
  132.             flag = 1
  133.             break
  134.     if not flag:
  135.         print('第%s条日志，不匹配任何正则表达式' % (log_index + 1))

提取效果

  1. 匹配第1个Pattern 匹配结果： ('ERROR', 'DirectJDKLog.java', '181', ' - Servlet.service() for servlet [dispatcherServlet] in context with path [/finance] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause', 'java.lang.NullPointerException', '', '\n\tat ')
  2. 匹配第1个Pattern 匹配结果： ('WARN', 'FeignClientsErrorDecoder.java', '43', ' - read Exception failed!', 'com.fasterxml.jackson.databind.JsonMappingException', ': No content to map due to end-of-input', '\n    at ')
  3. 匹配第1个Pattern 匹配结果： ('ERROR', 'AccessLogWebFilter.java', '60', ' - [accessId=616ccc49ff642e00010a4e8c] 发生网关内部错误', 'org.springframework.web.server.ResponseStatusException', ': 504 GATEWAY_TIMEOUT "Response took longer than timeout: PT35S"; nested exception is org.springframework.cloud.gateway.support.TimeoutException: Response took longer than timeout: PT35S', '\n\tat ')
  4. 匹配第1个Pattern 匹配结果： ('WARN', 'VehicleOeImpl.java', '1000', ' - 批量更新第三方价格失败1---->', 'org.springframework.jdbc.BadSqlGrammarException', ':\n### Error updating database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Query was empty\n### The error may involve com.cmall.ec.webapp.maindata.web.dao.vehicleOe.ThirdPartyOeMapper.updateList-Inline\n### The error may involve com.cmall.ec.webapp.maindata.web.dao.vehicleOe.ThirdPartyOeMapper.updateList-Inline\n### The error occurred while setting parameters\n### SQL:\n### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Query was empty\n; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Query was empty', '\n\tat ')
  5. 匹配第1个Pattern 匹配结果： ('ERROR', 'DirectJDKLog.java', '181', ' - Servlet.service() for servlet [dispatcherServlet] in context with path [/orders/seller] threw exception [Request processing failed; nested exception is com.cmall.commons.service.exception.HttpMessageException: [400]标名为空] with root cause', 'Exception', ': 标名为空', '\n\tat ')
  6. 匹配第1个Pattern 匹配结果： ('ERROR', 'DicountAssembler.java', '266', ' - 查询商品类优惠结果失败:DiscountProductRequest(companyId=IYl6MgdkiG9KoBMYUJo, userLoginId=5c385563ad996c47bf5f7ccd, provinceGeoId=CN-43, cityGeoId=284)', 'feign.FeignException', ': status 500 reading DiscountProductClient#listDiscountProducts(DiscountProductRequest); content:\n{"timestamp":1634520143844,"status":500,"error":"Internal Server Error","exception":"com.cmall.commons.service.exception.HttpMessageException","message":"优惠前的不含税价格不能为空","path":"/discountPromotion/listDiscountProducts"}', '\n\tat ')
  7. 匹配第4个Pattern 匹配结果： ('ERROR', 'DicountAssembler.java', '266', '', '', ' - task supervisor threw an exception ....', '')
  8. 匹配第2个Pattern 匹配结果： ('ERROR', 'ConsumeSupport.java', '104', 'Caused by: ', 'com.cmall.messagebus.exception.MessageBaseException', ": 系统自动报价失败-->org.springframework.dao.DuplicateKeyException:\n### Error updating database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'\n### The error may involve defaultParameterMap\n### The error occurred while setting parameters\n### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'\n; SQL []; Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '616cca225cc0a90001e1d0d2' for key 'PRIMARY'", '\n\tat ')
  9. 匹配第1个Pattern 匹配结果： ('ERROR', 'TimedSupervisorTask.java', '79', ' - task supervisor threw an exception', 'java.lang.OutOfMemoryError', ': Java heap space', '')
  10. 匹配第1个Pattern 匹配结果： ('ERROR', 'TimedSupervisorTask.java', '79', ' - task supervisor threw an exception', 'java.lang.OutOfMemoryError', ': Java heap space\nat', '')
  11. 匹配第2个Pattern 匹配结果： ('ERROR', 'ConsumeSupport.java', '104', 'Caused by: ', 'com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException', ": Duplicate entry 'SF2110230005145' for key 'PRIMARY'", '\n\tat ')
  12. 匹配第1个Pattern 匹配结果： ('ERROR', 'CompositeLog.java', '122', ' - 500 Server Error for HTTP POST "/job-service/api/registry"', 'reactor.netty.http.client.PrematureCloseException', ': Connection prematurely closed BEFORE response', '')
  13. 匹配第3个Pattern 匹配结果： ('WARN', 'AdminService.java', '97', ' - 任务执行失败, code:500, reason:', 'java.lang.OutOfMemoryError', ': Java heap space', '')
  14. 匹配第3个Pattern 匹配结果： ('ERROR', 'WaitSettleDealServiceImpl.java', '147', ' - 添加退订单到账单失败', 'com.cmall.commons.service.exception.HttpMessageException', ': [411]退货单添加失败，该分组已经结束对账，无法添加退货单', '')

作者：授客
QQ：1033553122
全国软件测试QQ交流群：7156436
Git地址：https://gitee.com/ishouke
友情提示：限于时间仓促，文中可能存在错误，欢迎指正、评论！
作者五行缺钱，如果觉得文章对您有帮助，请扫描下边的二维码打赏作者，金额随意，您的支持将是我继续创作的源动力，打赏后如有任何疑问，请联系我!!!
           微信打赏                        支付宝打赏                  全国软件测试交流QQ群  
              

原文链接：http://www.cnblogs.com/shouke/p/15807375.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Python热门文章

  • Python | 新冠肺炎疫情数据的爬取与可视化分析
  • Python爬小草1024图片，盖达尔的诱惑（urllib.request）
  • pytorch如何获得模型的计算量和参数量
  • 计算曲线与坐标轴的面积
  • 基于Python实现语音合成小工具
  • PyQt5笔记之弹出窗口大全
  • python反编译学习之字节码详解
  • Python爬虫基础讲解（二十）：多进程——multiprocessing模块的使用
  • python?OpenCV计算图片相似度的5种算法
  • matplotlib 设置图形大小时 figsize 与 dpi 的关系

    • Python推荐文章

  • python 音频处理（2)——提取PPG特征之whisper库的使用（2.1）
  • 《最新出炉》系列初窥篇-Python+Playwright自动化测试-60 - 判断元素是否显示 - 下篇 - 北京-宏哥
  • [python] 启发式算法库scikit-opt使用指北
  • 《最新出炉》系列初窥篇-Python+Playwright自动化测试-59 - 判断元素是否显示 - 上篇 - 北京-宏哥
  • 界面自动化测试录制工具，让python selenium自动化测试脚本开发更加方便
  • 《最新出炉》系列入门篇-Python+Playwright自动化测试-56- 多文件上传 - 下篇 - 北京-宏哥
  • 音频文件降噪及python示例
  • 全网最适合入门的面向对象编程教程：20 类和对象的 Python 实现-组合关系的实现与 CSV 文件保存
  • 张高兴的 MicroPython 入门指南：（一）环境配置、Blink、部署
  • Python——比 Seaborn 更好的相关性热力图：Biokit Corrplot

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号