Logstash 处理 Mongod Log2

franket

发布于 2022-01-19 11:25:40

2470

发布于 2022-01-19 11:25:40

文章被收录于专栏：技术杂记

关注信息

从下面实例的格式中可以看到

2014-11-03T18:28:32.450-0500 I NETWORK  [initandlisten] waiting for connections on port 27017
2015-12-25T18:41:47.683+0800 I CONTROL  [signalProcessingThread] pid=37405 port=27017 64-bit host=mongodb-server
2015-12-25T18:51:43.858+0800 I QUERY    [conn425412] query local.oplog.rs query: { ts: { $gte: Timestamp 1450975902000|10 } } planSummary: COLLSCAN cursorid:400229983803 ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:102 keyUpdates:0 writeConflicts:0 numYields:11609 nreturned:101 reslen:18110 locks:{ Global: { acquireCount: { r: 11610 } }, MMAPV1Journal: { acquireCount: { r: 11611 } }, Database: { acquireCount: { r: 11610 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 165 } }, oplog: { acquireCount: { R: 11610 } } } 1211ms
2015-12-25T20:54:11.336+0800 I JOURNAL  [journal writer] old journal file will be removed: /var/lib/mongo/journal/j._177
2015-12-26T00:46:36.512+0800 I COMMAND  [conn424487] command feed_test_repo.$cmd command: geoNear { geoNear: "users", near: [ 88.598884, 44.102866 ], query: {}, num: 30, maxDistance: 10 } keyUpdates:0 writeConflicts:0 numYields:399 reslen:37700 locks:{ Global: { acquireCount: { r: 400 } }, MMAPV1Journal: { acquireCount: { r: 400 } }, Database: { acquireCount: { r: 400 } }, Collection: { acquireCount: { R: 400 } } } 2584ms
2015-12-26T02:15:02.218+0800 I QUERY    [conn429640] assertion 13435 not master and slaveOk=false ns:feed_test_repo.notifications query:{ query: {}, orderby: { _id: 1.0 } }
2015-12-26T13:50:20.755+0800 I REPL     [ReplicationExecutor] Member 192.168.100.123:27017 is now in state ARBITER
2015-12-29T01:45:40.781+0800 I STORAGE  [FileAllocator] allocating new datafile /var/lib/mongo/feed_test_repo.107, filling with zeroes...

参考

<timestamp> <severity> <component> [<context>] <message>

前四部分(<timestamp> <severity> <component> [<context>])的内容相对固定
最后一部分 (<message>) 内部比较多变

我们比较关心操作时长，希望可以将这个信息收集进来，这个信息在最后一部分包含，有些内容包含，有些不包含

logstash配置

[root@h102 etc]# cat logstash-for-mongo.conf  
input {
  stdin {}
  file {
	    type=>"mongolog"
	    path=>"/tmp/xyz.log"
	    start_position => beginning
       }
}

filter {
  grok {
       match => ["message","%{TIMESTAMP_ISO8601:timestamp}\s+%{MONGO3_SEVERITY:severity}\s+%{MONGO3_COMPONENT:component}%{SPACE}(?:\[%{DATA:context}\])?\s+%{GREEDYDATA:body}"]
  } 
  if [body] =~ "ms$"  {  
       grok {
	match => ["body",".*\}(\s+%{NUMBER:spend_time:int}ms$)?"]
       }
 }
 date {
   match => [ "timestamp", "ISO8601" ]
   #remove_field => [ "timestamp" ]
  }
}

output {
  elasticsearch { 
  	hosts => ["localhost:9200"] 
        index=>"mongodb-slow-log-%{+YYYY.MM.dd}"
	}
  stdout { codec => rubydebug }
}
[root@h102 etc]#

本文系转载，前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

component