Logstash不会处理跨越多行的日志。
提问于 2015-12-11 05:15:53
我试图解析一些本地日志文件，在我的windows机器上运行ELK堆栈。下面是我试图解析的日志的一个例子。
2015-12-10 13:50:25,487 [http-nio-8080-exec-26] INFO a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- This Message is OK
2015-12-10 13:50:26,487 [http-nio-8080-exec-26] INFO a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- Journe
y road update: <rows>
<row adi="D" date="2015-12-10" garage="TOP">
<codeNum order="1">TP</codeNum>
<number order="1">1001</number>
<journeystatus code="RT">OnRoute</journeystatus>
</row>
</rows>第一条消息在筛选器中运行良好，但第二条消息被分割成带有标记部分中的_grokparsefailure的多个消息。
Logstash Config文件
input {
file {
path => "C:/data/sampleLogs/temp.log"
type => "testlog"
start_position => "beginning"
}
}
filter {
grok {
# Parse timestamp data. We need the "(?m)" so that grok (Oniguruma internally) correctly parses multi-line events
match => [ "message", [
"(?m)%{TIMESTAMP_ISO8601:logTimestamp}[ ;]\[%{DATA:threadId}\][ ;]%{LOGLEVEL:logLevel}[ ;]+%{JAVACLASS:JavaClass}[ ;]%{SYSLOG5424SD:TransactionID}[ ;]*%{GREEDYDATA:LogMessage}",
"(?m)%{TIMESTAMP_ISO8601:logTimestamp}[ ;]\[%{DATA:threadId}\][ ;]%{LOGLEVEL:logLevel}[ ;]+%{JAVAFILE:JavaClass}[ ;]%{SYSLOG5424SD:TransactionID}[ ;]*%{GREEDYDATA:LogMessage}"
]
]
}
# The timestamp may have commas instead of dots. Convert so as to store everything in the same way
mutate {
gsub => [
# replace all commas with dots
"logTimestamp", ",", "."
]
}
mutate {
gsub => [
# make the logTimestamp sortable. With a space, it is not! This does not work that well, in the end
# but somehow apparently makes things easier for the date filter
"logTimestamp", " ", ";"
]
}
date {
locale => "en"
timezone => "UTC"
match => [ "logTimestamp", "YYYY-MM-dd;HH:mm:ss.SSS" ]
target => "@timestamp"
}
mutate {
add_field => { "debug-timestamp" => "timestampMatched"}
}
}
output {
stdout {
codec => rubydebug
}
}当我跑的时候
bin\logstash agent -f \ELK-Stack\logstash\conf\01_input.conf在CMD提示符中，返回的内容如下
io/console not supported; tty will not be manipulated
Default settings used: Filter workers: 4
Logstash startup completed
{
"message" => " <row adi=\"D\" date=\"2015-12-10\" garage=\"TOP\"
>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " <codeNum order=\"1\">TP</codeNum>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " <number order=\"1\">1001</number>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " <journeystatus code=\"RT\">OnRoute</journeys
tatus>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.278Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => " </row>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.278Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => "y road update: <rows>\r",
"@version" => "1",
"@timestamp" => "2015-12-11T12:49:34.268Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"tags" => [
[0] "_grokparsefailure"
],
"debug-timestamp" => "timestampMatched"
}
{
"message" => "2015-12-10 13:50:25,487 [http-nio-8080-exec-26] INFO
a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- Journe\r",
"@version" => "1",
"@timestamp" => "2015-12-10T13:50:25.487Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"logTimestamp" => "2015-12-10;13:50:25.487",
"threadId" => "http-nio-8080-exec-26",
"logLevel" => "INFO",
"JavaClass" => "a.b.c.v1.myTestClass",
"TransactionID" => "[abcde-1234-12345-b425-12ad]",
"LogMessage" => "- Journe\r",
"debug-timestamp" => "timestampMatched"
}
{
"message" => "</rows>2015-12-10 13:50:25,487 [http-nio-8080-exec-26]
INFO a.b.c.v1.myTestClass [abcde-1234-12345-b425-12ad]- This Message is OK\r",
"@version" => "1",
"@timestamp" => "2015-12-10T13:50:25.487Z",
"host" => "GMAN",
"path" => "C:/data/sampleLogs/temp.log",
"type" => "testlog",
"logTimestamp" => "2015-12-10;13:50:25.487",
"threadId" => "http-nio-8080-exec-26",
"logLevel" => "INFO",
"JavaClass" => "a.b.c.v1.myTestClass",
"TransactionID" => "[abcde-1234-12345-b425-12ad]",
"LogMessage" => "- This Message is OK\r",
"debug-timestamp" => "timestampMatched"
}我确实在我的过滤器顶部添加了混线，但是它不起作用，只是给出了下面的错误，就在我的摸索之后。
multiline {
pattern => "^201*-**-**- **:**:"
what => "previous"
negate=> true
}但这并没有帮助我继续给我一条错误信息
Error: Cannot use more than 1 filter worker because the following plugins don't
work with more than one worker: multiline
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.因此，我尝试按建议运行--configtest，并出现新的错误消息。
Error: Cannot use more than 1 filter worker because the following plugins don't
work with more than one worker: multiline有人能帮我解决这个问题吗?能帮我处理多行吗？
我们非常感谢你的帮助。
更新
正如@Alain建议在多线中使用编解码器，下面是我的配置的输入。
input {
file {
path => "C:/data/sampleLogs/mulline.log"
codec => multiline {
# Grok pattern names are valid! :)
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
type => "testlog"
start_position => "beginning"
}
}G
回答 1
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/34224472
复制相关文章
腾讯云开发者
