扫描日志文件中最近30分钟的数据
扫描日志文件中最近30分钟的数据
提问于 2014-11-20 18:33:49
我必须编写一个shell/perl脚本来扫描日志文件中最近30分钟的数据。要求在Cron中安排此脚本每30分钟运行一次，并查找错误字符串。
OS: Solaris
Shell:Bash我试过下面的脚本，但它变得太长太笨拙了，我们有其他方法可以把它缩短一点吗？
blogs=/opt/docs/datapower/prod/business.log
slogs=/opt/docs/datapower/prod/system.log
starttime=$(date +'%H')
currmin=$(date +'%M')
curdate=`date|cut -d' ' -f5`
echo $(date)
if [ $currmin -le 29 ] && [ $starttime -ne 00 ] ; then
starttime1=`echo "$(date +'%H') - 1" | bc`
logtime="$starttime1"
logtime="$logtime:[3-5][0-9]"
echo $logtime
elif [ $currmin -le 29 ] && [ $starttime -eq 00 ] ; then
logtime="23:[3-5][0-9]"
echo $logtime
else
logtime="$starttime"
logtime="$logtime:[0-2][0-9]"
echo $logtime
fi
if ( grep "$logtime" $slogs | egrep "AAA Authentication Failure|AAA Authorization Failure") > dptest 2>&1;then
Do something
fi 更新:添加示例日志语句。以下是日志语句的示例：Nov 20 06:06:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
回答 2
Stack Overflow用户
发布于 2014-11-20 21:04:08
我认为你在做一些倒退的事情--构建一个RE来从日志文件中获取一个日期。
在perl中，我希望读取整个日志文件，对其进行标记，以提取时间戳，然后根据消息内容发出警报。
Perl为第一部分提供了一个很好的模块-- Time::Piece。它有点像这样：
use strict;
use warnings;
use Time::Piece;
my $HALF_HOUR = 30 * 60;
while (<DATA>) {
#extract timestamp via regular expression
my ( $timestamp, $message ) = (m/\A(\w+\s+\d+\s+\d+:\d+:\d+) (.*)/);
#convert text timestamp to 'unix time'.
#need the year in here because your log doesn't include it.
my $t = localtime();
$t = $t->strptime( $timestamp . " " . $t->year, "%b %d %H:%M:%S %Y" );
#skip if parsed time is more than half an hour ago.
next if ( $t < time() - $HALF_HOUR );
if ( $message =~ m/AAA Authentication failure/i
or $message =~ m/AAA Authorization failure/i )
{
print "Alert: ( $t ) $message\n";
}
}
__DATA__
Nov 20 13:46:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
Nov 20 13:00:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
Nov 20 10:06:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>后续问题：
“你能解释一下这条语句是做什么的吗，my ( $timestamp, $message ) = (m/\A(\w+\s+\d+\s+\d+:\d+:\d+) (.*)/);”
这会做两件事：
- Perl 中的一个技巧是，您可以通过将正则表达式放在括号中来捕获它的一部分。所以
\A(\w+\s+\d+\s+\d+:\d+:\d+)- 将从行首开始匹配 - 一个或多个“单词”字符。
- 一个或多个“数字”
\d+:\d+:\d+将捕获时间。 （任何 3 个冒号分隔的数字）。
当然，另一部分捕获了“其余部分”。
- 然后，我们将模式匹配返回的数组赋值到一个命名变量数组(
$timestamp和$message)中。
最终结果是-给定一行：
Nov 20 13:46:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
(\w+ \d+ \d+:\d+:\d+) (.*)我们的正则表达式分别返回两个“块”，然后将它们放入两个变量中。
Stack Overflow用户
发布于 2014-11-20 23:17:30
你的观点是如何使用sqlite3来做过滤工作-它解析时间的好处对你来说可能是非常方便的。唯一的缺点是你必须对数据进行规范化。
function sqlite-filter-time() {
if [ '0' = "$#" ]; then
echo "Usage: $FUNCNAME <file> <timespan> <where>"
return
fi
local year="$(date '+%Y')"
local ofs='___FS___'
sed "s,^\([^ ]* [^ ]*\) \([^ ]*\),\1 \2$ofs," "$1" | sed "s,Jan ,$year-01-,;s,Feb ,$year-02-,;s,Mar ,$year-03-,;s,Apr ,$year-04-,;s,May ,$year-05-,;s,June ,$year-06-,;s,July ,$year-07-,;s,Aug ,$year-08-,;s,Sep ,$year-09-,;s,Oct ,$year-10-,;s,Nov ,$year-11-,;s,Dec ,$year-12-," > "$1.tmp" # normalize data for sqlite - command to extract the date and the rest of the text
{
echo '.mode csv'
echo 'DROP TABLE IF EXISTS sft;'
echo 'CREATE TEMPORARY TABLE sft ('
echo ' sft_date TEXT,'
echo ' sft_text TEXT'
echo ');'
echo ".headers off"
echo ".nullvalue ''"
echo ".separator '$ofs'"
echo ".import $1.tmp sft"
echo ".separator ' '"
echo "SELECT *"
echo "FROM sft"
echo "WHERE sft_date > datetime('now', '$2')"
echo " AND (sft_text like '%AAA Authentication Failure%'"
echo " OR sft_text like '%AAA Authorization Failure%'"
echo " )"
echo ";"
} | sqlite3
rm "$1.tmp"
}
$ sqlite-filter-time "$slogs" '-30 minutes'
"2014-11-20 16:01:58" " business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>"
$页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/27037369
复制相关文章
相似问题
腾讯云开发者
