blocks|key|296266|text|Perl来拯救我们：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|296267|cat+access.log+%7C+perl+-n+-e'm@\[(\d{1,2})/(\w{3})/(\d{4}):@;+open(LOG,+">>access.apache.$3_$2_$1.log");+print+LOG+$_;'|code-block|syntax|javascript|296268|好吧，它不完全是“标准”的操作程序，但它是为文本操作而设计的。|296269|我还更改了文件名中参数的顺序，这样文件的名称就像access.apache.yyyy_mon_dd.log一样，便于排序。|296270|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Perl came to the rescue:

<pre><code>cat access.log | perl -n -e'm@\[(\d{1,2})/(\w{3})/(\d{4}):@; open(LOG, "&gt;&gt;access.apache.$3_$2_$1.log"); print LOG $_;'
</code></pre>

Well, it's not exactly "standard" manipulation program, but it's made for text manipulation nevertheless.

I've also changed order of arguments in file name, so that files are named like access.apache.yyyy_mon_dd.log for easier sorting.

blocks|key|296352|text|有点丑陋，这是对你的抨击：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|296353|++++for+year+in+2010+2011+2012;+do
+++++++for+month+in+jan+feb+mar+apr+may+jun+jul+aug+sep+oct+nov+dec;+do
+++++++++++for+day+in+1+2+3+4+5+6+7+8+9+10+...+31+;+do
+++++++++++++++cat+access.log+%7C+grep+-i+$day/$month/$year+>+$day-$month-$year.log
++++++++++++done
++++++++done
+++++done|code-block|syntax|javascript|296354|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Kind of ugly, that's bash for you:

<pre><code> for year in 2010 2011 2012; do
 for month in jan feb mar apr may jun jul aug sep oct nov dec; do
 for day in 1 2 3 4 5 6 7 8 9 10 ... 31 ; do
 cat access.log | grep -i $day/$month/$year &gt; $day-$month-$year.log
 done
 done
 done
</code></pre>

blocks|key|296404|text|纯bash，让一个人通过访问日志：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|296405|while+read;+do
++++[[+$REPLY+=~+\[(..)/(...)/(....):+]]

++++d=${BASH_REMATCH[1]}
++++m=${BASH_REMATCH[2]}
++++y=${BASH_REMATCH[3]}

++++#printf+-v+fname+"access.apache.%25s_%25s_%25s.log"+${BASH_REMATCH[@]:1:3}
++++printf+-v+fname+"access.apache.%25s_%25s_%25s.log"+$y+$m+$d

++++echo+"$REPLY"+>>+$fname
done+<+access.log|code-block|syntax|javascript|296406|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Pure bash, making one pass through the access log:

<pre><code>while read; do
 [[ $REPLY =~ \[(..)/(...)/(....): ]]

 d=${BASH_REMATCH[1]}
 m=${BASH_REMATCH[2]}
 y=${BASH_REMATCH[3]}

 #printf -v fname "access.apache.%s_%s_%s.log" ${BASH_REMATCH[@]:1:3}
 printf -v fname "access.apache.%s_%s_%s.log" $y $m $d

 echo "$REPLY" &gt;&gt; $fname
done &lt; access.log
</code></pre>

blocks|key|296446|text|这是一个按词法输出可排序日志文件的awk版本。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|296447|一些效率增强:所有这些都在一遍中完成，只在与以前不同的时候生成fname，在切换到新文件时关闭fname+(否则可能会用完文件描述符)。|296448|awk+-F"[]/:[]"+'
BEGIN+{
++m2n["Jan"]+=++1;++m2n["Feb"]+=++2;+m2n["Mar"]+=++3;+m2n["Apr"]+=++4;
++m2n["May"]+=++5;++m2n["Jun"]+=++6;+m2n["Jul"]+=++7;+m2n["Aug"]+=++8;
++m2n["Sep"]+=++9;++m2n["Oct"]+=+10;+m2n["Nov"]+=+11;+m2n["Dec"]+=+12;
}
{
++if($4+!=+pyear+%7C%7C+$3+!=+pmonth+%7C%7C+$2+!=+pday)+{
++++pyear++=+$4
++++pmonth+=+$3
++++pday+++=+$2

++++if(fname+!=+"")
++++++close(fname)

++++fname++=+sprintf("access_%2504d_%2502d_%2502d.log",+$4,+m2n[$3],+$2)
++}
++print+>+fname
}'+access-log|code-block|syntax|javascript|296449|entityMap^0|H|3|0|V|5|1B|5|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|R|8|@$9|S|A|T|B|C]|$9|U|A|V|B|C]]|D|@]|E|$]]|$1|H|3|I|5|J|7|W|8|@]|D|@]|E|$K|L]]|$1|M|3|-4|5|6|7|X|8|@]|D|@]|E|$]]]|N|$]]

Here is an <code>awk</code> version that outputs lexically sortable log files.

Some efficiency enhancements: all done in one pass, only generate <code>fname</code> when it is not the same as before, close <code>fname</code> when switching to a new file (otherwise you might run out of file descriptors).

<pre><code>awk -F"[]/:[]" '
BEGIN {
 m2n["Jan"] = 1; m2n["Feb"] = 2; m2n["Mar"] = 3; m2n["Apr"] = 4;
 m2n["May"] = 5; m2n["Jun"] = 6; m2n["Jul"] = 7; m2n["Aug"] = 8;
 m2n["Sep"] = 9; m2n["Oct"] = 10; m2n["Nov"] = 11; m2n["Dec"] = 12;
}
{
 if($4 != pyear || $3 != pmonth || $2 != pday) {
 pyear = $4
 pmonth = $3
 pday = $2

 if(fname != "")
 close(fname)

 fname = sprintf("access_%04d_%02d_%02d.log", $4, m2n[$3], $2)
 }
 print &gt; fname
}' access-log
</code></pre>

blocks|key|296493|text|使用awk的一种方式|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|296494|awk+'BEGIN+{
++++split("Jan+Feb+Mar+Apr+May+Jun+Jul+Aug+Sep+Oct+Nov+Dec+",+months,+"+")
++++for+(a+=+1;+a+<=+12;+a%2B%2B)
++++++++m[months[a]]+=+sprintf("%2502d",+a)
}
{
++++split($4,array,"[:/]")
++++year+=+array[3]
++++month+=+m[array[2]]

++++print+>+FILENAME"-"year"_"month".txt"
}'+incendiary.ws-2009|code-block|syntax|javascript|296495|这将输出如下所示的文件：|296496|incendiary.ws-2010-2010_04.txt
incendiary.ws-2010-2010_05.txt
incendiary.ws-2010-2010_06.txt
incendiary.ws-2010-2010_07.txt|296497|与150MB的日志文件相比，chepner在3.4+GHz+8+Core+Xeon+E31270上的回答花费了70秒，而这种方法只需要5秒。|296498|原创灵感："How+to+split+existing+apache+logfile+by+month?“|296499|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/a/11714105/430062^0|2|3|0|0|0|0|0|6|1A|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@$9|11|A|12|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|13|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|14|8|@]|D|@]|E|$]]|$1|M|3|N|5|H|7|15|8|@]|D|@]|E|$I|J]]|$1|O|3|P|5|6|7|16|8|@]|D|@]|E|$]]|$1|Q|3|R|5|6|7|17|8|@]|D|@$9|18|A|19|1|1A]]|E|$]]|$1|S|3|-4|5|6|7|1B|8|@]|D|@]|E|$]]]|T|$U|$5|V|W|X|E|$Y|Z]]]]

One way using <code>awk</code>:

<pre><code>awk 'BEGIN {
 split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec ", months, " ")
 for (a = 1; a &lt;= 12; a++)
 m[months[a]] = sprintf("%02d", a)
}
{
 split($4,array,"[:/]")
 year = array[3]
 month = m[array[2]]

 print &gt; FILENAME"-"year"_"month".txt"
}' incendiary.ws-2009
</code></pre>

This will output files like:

<pre><code>incendiary.ws-2010-2010_04.txt
incendiary.ws-2010-2010_05.txt
incendiary.ws-2010-2010_06.txt
incendiary.ws-2010-2010_07.txt
</code></pre>

Against a 150 MB log file, the answer by chepner took 70 seconds on an 3.4 GHz 8 Core Xeon E31270, while this method took 5 seconds.

Original inspiration: "<a href="https://stackoverflow.com/a/11714105/430062">How to split existing apache logfile by month?</a>"

blocks|key|296522|text|我结合了Theodore和Thor的解决方案，使用了Thor的效率改进和日常文件，但在组合格式文件中保留了对IPv6地址的原始支持。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|296523|awk+'
BEGIN+{
++m2n["Jan"]+=++1;++m2n["Feb"]+=++2;+m2n["Mar"]+=++3;+m2n["Apr"]+=++4;
++m2n["May"]+=++5;++m2n["Jun"]+=++6;+m2n["Jul"]+=++7;+m2n["Aug"]+=++8;
++m2n["Sep"]+=++9;++m2n["Oct"]+=+10;+m2n["Nov"]+=+11;+m2n["Dec"]+=+12;
}
{
++split($4,+a,+"[]/:[]")
++if(a[4]+!=+pyear+%7C%7C+a[3]+!=+pmonth+%7C%7C+a[2]+!=+pday)+{
++++pyear++=+a[4]
++++pmonth+=+a[3]
++++pday+++=+a[2]

++++if(fname+!=+"")
++++++close(fname)

++++fname++=+sprintf("access_%2504d-%2502d-%2502d.log",+a[4],+m2n[a[3]],+a[2])
++}
++print+>>+fname
}'|code-block|syntax|javascript|296524|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

I combined Theodore's and Thor's solutions to use Thor's efficiency improvement and daily files, but retain the original support for IPv6 addresses in combined format file.

<pre><code>awk '
BEGIN {
 m2n["Jan"] = 1; m2n["Feb"] = 2; m2n["Mar"] = 3; m2n["Apr"] = 4;
 m2n["May"] = 5; m2n["Jun"] = 6; m2n["Jul"] = 7; m2n["Aug"] = 8;
 m2n["Sep"] = 9; m2n["Oct"] = 10; m2n["Nov"] = 11; m2n["Dec"] = 12;
}
{
 split($4, a, "[]/:[]")
 if(a[4] != pyear || a[3] != pmonth || a[2] != pday) {
 pyear = a[4]
 pmonth = a[3]
 pday = a[2]

 if(fname != "")
 close(fname)

 fname = sprintf("access_%04d-%02d-%02d.log", a[4], m2n[a[3]], a[2])
 }
 print &gt;&gt; fname
}'
</code></pre>

I have a Apache access.log file, which is around 35GB in size. Grepping through it is not an option any more, without waiting a great deal.

I wanted to split it in many small files, by using date as splitting criteria.

Date is in format <code>[15/Oct/2011:12:02:02 +0000]</code>. Any idea how could I do it using only bash scripting, standard text manipulation programs (grep, awk, sed, and likes), piping and redirection?

Input file name is <code>access.log</code>. I'd like output files to have format such as <code>access.apache.15_Oct_2011.log</code> (that would do the trick, although not nice when sorting.)

Split access.log file by dates using command line tools

日志文件

Perl

我有一个Apache access.log文件，大小约为35 is。无需等待很长时间，就不再是一个选择。我想通过使用日期作为拆分标准，将其拆分成许多小文件。日期采用[15/Oct/2011:12:02:02 +0000]格式。你知道我如何只使用bash脚本，标准的文本操作程序(grep，awk，sed，和like)，管...

问使用命令行工具按日期拆分access.log文件
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用命令行工具按日期拆分access.log文件EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用命令行工具按日期拆分access.log文件
EN