配置nginx日志
log_format json '{"remote_addr":"$remote_addr" ,"host":"$host" ,"server_addr":"$server_addr" ,"timestamp":"$time_iso8601" ,"request_time":$request_time, "remote_user":"$remote_user", "request":"$request" ,"status":$status, "body_sent":$body_bytes_sent ,"http_referer":"$http_referer" ,"http_user_agent":"$http_user_agent" ,"http_x_forwarded_for":"$http_x_forwarded_for"}';
配置logstash
input {
file {
path => ["/data/logs/nginx/collectd.dev-access.log"]
type => "demo-codec-json-log"
start_position => "beginning"
codec => "json"
}
}
output {
stdout{
codec=>rubydebug
}
}
启动
bin/logstash -f /etc/logstash/conf.d/demo-codec-json.conf
结果
{
"remote_addr" => "192.168.56.1",
"request" => "GET /graph.php?p=load&t=load&h=192.168.56.201&s=86400 HTTP/1.1",
"type" => "demo-codec-json-log",
"server_addr" => "192.168.56.201",
"http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36",
"remote_user" => "-",
"path" => "/data/logs/nginx/collectd.dev-access.log",
"request_time" => 0.026,
"@timestamp" => 2017-06-13T06:31:12.761Z,
"http_referer" => "http://collectd.dev/host.php?h=192.168.56.201&p=load",
"host" => "collectd.dev",
"http_x_forwarded_for" => "-",
"@version" => "1",
"body_sent" => 13863,
"timestamp" => "2017-06-13T06:31:12+00:00",
"status" => 200
}
备注
nginx日志当中部分字段可能会是数字或者-,可以将日志全部转换为字符串,然后通过filter来处理