我在ElasticBeanstalk桶上有AWS的S3实例日志。
日志的路径是:
resources/environments/logs/publish/e-3ykfgdfgmp8/i-cf216955/_var_log_nginx_rotated_access.log1417633261.gz这意味着:
resources/environments/logs/publish/e-random环境id/i-随机实例id/
路径包含多个日志:
_var_log_eb-docker_containers_eb-current-app_rotated_application.log1417586461.gz
_var_log_eb-docker_containers_eb-current-app_rotated_application.log1417597261.gz
_var_log_rotated_docker1417579261.gz
_var_log_rotated_docker1417582862.gz
_var_log_rotated_docker-events.log1417579261.gz
_var_log_nginx_rotated_access.log1417633261.gz注意到有一些随机数(时间戳?)由AWS在文件名中插入".gz“之前
问题是我需要根据日志文件名来设置变量。
这是我的配置:
input {
s3 {
debug => "true"
bucket => "elasticbeanstalk-us-east-1-something"
region => "us-east-1"
region_endpoint => "us-east-1"
credentials => ["..."]
prefix => "resources/environments/logs/publish/"
sincedb_path => "/tmp/s3.sincedb"
backup_to_dir => "/tmp/logstashed/"
tags => ["s3","elastic_beanstalk"]
type => "elastic_beanstalk"
}
}
filter {
if [type] == "elastic_beanstalk" {
grok {
match => [ "@source_path", "resources/environments/logs/publish/%{environment}/%{instance}/%{file}<unnecessary_number>.gz" ]
}
}
}在本例中,我希望从path提取环境、实例和文件名。在文件名中,我需要忽略那个随机数。我这样做对吗?什么是完整的,正确的解决方案?
另一个问题是如何从上面为特定日志文件指定自定义日志格式的字段?
这可能类似于:(元代码)
filter {
if [type] == "elastic_beanstalk" {
if [file_name] BEGINS WITH "application_custom_log" {
grok {
match => [ "message", "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" ]
}
}
if [file_name] BEGINS WITH "some_other_custom_log" {
....
}
}
}如何测试文件名模式?
发布于 2014-12-06 05:22:37
对于第一个问题,并假设@source_path包含完整路径,请尝试:
match => [ "@source_path", "logs/publish/%{NOTSPACE:env}/%{NOTSPACE:instance}/%{NOTSPACE:file}%{NUMBER}%{NOTSPACE:suffix}" ]这将为您创建4个logstash字段:
有关于grok手册页的更多信息,您应该使用grok调试器进行测试。
要测试logstash中的字段,可以使用条件词。
if [field] == "value"
if [field] =~ /regexp/等。
请注意,使用grok并不总是必要的。您可以有多个“匹配”参数,它(默认情况下)将在命中匹配的第一个参数之后停止。如果您的模式是独占的,这应该适用于您。
https://stackoverflow.com/questions/27327493
复制相似问题