blocks|key|1165602|text|通过首先使用sort对文件进行排序，然后可以应用uniq。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1165603|它似乎可以很好地对文件进行排序：|1165604|$+cat+test.csv
overflow@domain2.com,2009-11-27+00:58:29.793000000,xx3.net,255.255.255.0
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
overflow@domain2.com,2009-11-27+00:58:29.646465785,2x3.net,256.255.255.0+
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1

$+sort+test.csv
overflow@domain2.com,2009-11-27+00:58:29.646465785,2x3.net,256.255.255.0+
overflow@domain2.com,2009-11-27+00:58:29.793000000,xx3.net,255.255.255.0
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1

$+sort+test.csv+%7C+uniq
overflow@domain2.com,2009-11-27+00:58:29.646465785,2x3.net,256.255.255.0+
overflow@domain2.com,2009-11-27+00:58:29.793000000,xx3.net,255.255.255.0
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1|code-block|syntax|javascript|1165605|你也可以做一些AWK魔术：|1165606|$+awk+-F,+'{+lines[$1]+=+$0+}+END+{+for+(l+in+lines)+print+lines[l]+}'+test.csv
stack2@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27+01:05:47.893000000,xx2.net,127.0.0.1
overflow@domain2.com,2009-11-27+00:58:29.646465785,2x3.net,256.255.255.0+|1165607|entityMap^0|6|4|O|4|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]|$9|V|A|W|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|X|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|Y|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|Z|8|@]|D|@]|E|$]]|$1|O|3|P|5|J|7|10|8|@]|D|@]|E|$K|L]]|$1|Q|3|-4|5|6|7|11|8|@]|D|@]|E|$]]]|R|$]]

By sorting the file with <code>sort</code> first, you can then apply <code>uniq</code>.

It seems to sort the file just fine:

<pre><code>$ cat test.csv
overflow@domain2.com,2009-11-27 00:58:29.793000000,xx3.net,255.255.255.0
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
overflow@domain2.com,2009-11-27 00:58:29.646465785,2x3.net,256.255.255.0 
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1

$ sort test.csv
overflow@domain2.com,2009-11-27 00:58:29.646465785,2x3.net,256.255.255.0 
overflow@domain2.com,2009-11-27 00:58:29.793000000,xx3.net,255.255.255.0
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1

$ sort test.csv | uniq
overflow@domain2.com,2009-11-27 00:58:29.646465785,2x3.net,256.255.255.0 
overflow@domain2.com,2009-11-27 00:58:29.793000000,xx3.net,255.255.255.0
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
</code></pre>

You could also do some AWK magic:

<pre><code>$ awk -F, '{ lines[$1] = $0 } END { for (l in lines) print lines[l] }' test.csv
stack2@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack4@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
stack3@domain.com,2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1
overflow@domain2.com,2009-11-27 00:58:29.646465785,2x3.net,256.255.255.0 
</code></pre>

blocks|key|1165664|text|或者如果你想使用uniq：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1165665|<mycvs.cvs+tr+-s+','+'+'+%7C+awk++'{print+$3"+"$2"+"$1}'+%7C+uniq+-c+-f2|offset|length|style|CODE|1165666|提供：|1165667|1+01:05:47.893000000+2009-11-27+tack2@domain.com
2+00:58:29.793000000+2009-11-27+overflow@domain2.com
1|code-block|syntax|javascript|1165668|entityMap^0|0|0|1W|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@$D|S|E|T|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|U|8|@]|9|@]|A|$]]|$1|J|3|K|5|L|7|V|8|@]|9|@]|A|$M|N]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

or if u want to use uniq:

<code>&lt;mycvs.cvs tr -s ',' ' ' | awk '{print $3" "$2" "$1}' | uniq -c -f2</code>

gives:

<pre><code>1 01:05:47.893000000 2009-11-27 tack2@domain.com
2 00:58:29.793000000 2009-11-27 overflow@domain2.com
1
</code></pre>

blocks|key|1380652|text|如果你想保留最后一个副本，你可以使用|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1380653|+tac+a.csv+%7C+sort+-u+-t,+-r+-k1,1+%7Ctac|code-block|syntax|javascript|1380654|这是我的要求|1380655|这里|1380656|tac将逐行反转文件|offset|length|style|CODE|1380657|entityMap^0|0|0|0|0|0|3|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|T|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|U|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|V|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|W|8|@$M|X|N|Y|O|P]]|9|@]|A|$]]|$1|Q|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]]|R|$]]

If you want to retain the last one of the duplicates you could use

<pre><code> tac a.csv | sort -u -t, -r -k1,1 |tac
</code></pre>

Which was my requirement

here 

<code>tac</code> will reverse the file line by line

blocks|key|1380757|text|这是一个非常好的方法。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1380758|首先，格式化内容，以便要比较唯一性的列是固定宽度。一种方法是使用带有字段/列宽度说明符("%2515s")的awk+printf。|1380759|现在，uniq的-f和-w选项可用于跳过前面的字段/列，并指定比较宽度(列宽度)。|1380760|这里有三个例子。|1380761|在第一个例子中...|1380762|1)暂时将感兴趣的列设置为大于或等于字段最大宽度的固定宽度。|1380763|2)使用-f+uniq选项跳过前面的列，使用-w+uniq选项限制tmp_fixed_width的宽度。|1380764|3)从列中删除尾随空格以“恢复”它的宽度(假设事先没有尾随空格)。|1380765|printf+"%25s"+"$str"+\
%7C+awk+'{+tmp_fixed_width=15;+uniq_col=8;+w=tmp_fixed_width-length($uniq_col);+for+(i=0;i<w;i%2B%2B)+{+$uniq_col=$uniq_col"+"};+printf+"%25s\n",+$0+}'+\
%7C+uniq+-f+7+-w+15+\
%7C+awk+'{+uniq_col=8;+gsub(/+*/,+"",+$uniq_col);+printf+"%25s\n",+$0+}'|code-block|syntax|javascript|1380766|在第二个例子中...|1380767|创建一个新的uniq列%25+1。然后在应用uniq筛选器后将其删除。|1380768|printf+"%25s"+"$str"+\
%7C+awk+'{+uniq_col_1=4;+printf+"%2515s+%25s\n",+uniq_col_1,+$0+}'+\
%7C+uniq+-f+0+-w+15+\
%7C+awk+'{+$1="";+gsub(/%5E+*/,+"",+$0);+printf+"%25s\n",+$0+}'|1380769|第三个示例与第二个示例相同，但针对多个列。|1380770|printf+"%25s"+"$str"+\
%7C+awk+'{+uniq_col_1=4;+uniq_col_2=8;+printf+"%255s+%2515s+%25s\n",+uniq_col_1,+uniq_col_2,+$0+}'+\
%7C+uniq+-f+0+-w+5+\
%7C+uniq+-f+1+-w+15+\
%7C+awk+'{+$1=$2="";+gsub(/%5E+*/,+"",+$0);+printf+"%25s\n",+$0+}'|1380771|entityMap^0|0|0|0|0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|16|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|17|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|18|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|19|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|1A|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|1B|8|@]|9|@]|A|$]]|$1|L|3|M|5|6|7|1C|8|@]|9|@]|A|$]]|$1|N|3|O|5|6|7|1D|8|@]|9|@]|A|$]]|$1|P|3|Q|5|R|7|1E|8|@]|9|@]|A|$S|T]]|$1|U|3|V|5|6|7|1F|8|@]|9|@]|A|$]]|$1|W|3|X|5|6|7|1G|8|@]|9|@]|A|$]]|$1|Y|3|Z|5|R|7|1H|8|@]|9|@]|A|$S|T]]|$1|10|3|11|5|6|7|1I|8|@]|9|@]|A|$]]|$1|12|3|13|5|R|7|1J|8|@]|9|@]|A|$S|T]]|$1|14|3|-4|5|6|7|1K|8|@]|9|@]|A|$]]]|15|$]]

Here is a very nifty way.

First format the content such that the column to be compared for uniqueness is a fixed width. One way of doing this is to use awk printf with a field/column width specifier ("%15s").

Now the -f and -w options of uniq can be used to skip preceding fields/columns and to specify the comparison width (column(s) width).

Here are three examples.

In the first example...

1) Temporarily make the column of interest a fixed width greater than or equal to the field's max width.

2) Use -f uniq option to skip the prior columns, and use the -w uniq option to limit the width to the tmp_fixed_width.

3) Remove trailing spaces from the column to "restore" it's width (assuming there were no trailing spaces beforehand).

<pre><code>printf "%s" "$str" \
| awk '{ tmp_fixed_width=15; uniq_col=8; w=tmp_fixed_width-length($uniq_col); for (i=0;i&lt;w;i++) { $uniq_col=$uniq_col" "}; printf "%s\n", $0 }' \
| uniq -f 7 -w 15 \
| awk '{ uniq_col=8; gsub(/ */, "", $uniq_col); printf "%s\n", $0 }'
</code></pre>

In the second example...

Create a new uniq column 1. Then remove it after the uniq filter has been applied.

<pre><code>printf "%s" "$str" \
| awk '{ uniq_col_1=4; printf "%15s %s\n", uniq_col_1, $0 }' \
| uniq -f 0 -w 15 \
| awk '{ $1=""; gsub(/^ */, "", $0); printf "%s\n", $0 }'
</code></pre>

The third example is the same as the second, but for multiple columns.

<pre><code>printf "%s" "$str" \
| awk '{ uniq_col_1=4; uniq_col_2=8; printf "%5s %15s %s\n", uniq_col_1, uniq_col_2, $0 }' \
| uniq -f 0 -w 5 \
| uniq -f 1 -w 15 \
| awk '{ $1=$2=""; gsub(/^ */, "", $0); printf "%s\n", $0 }'
</code></pre>

I have a .csv file like this:

<pre><code>stack2@example.com,2009-11-27 01:05:47.893000000,example.net,127.0.0.1
overflow@example.com,2009-11-27 00:58:29.793000000,example.net,255.255.255.0
overflow@example.com,2009-11-27 00:58:29.646465785,example.net,256.255.255.0
...
</code></pre>

I have to remove duplicate e-mails (the entire line) from the file (i.e. one of the lines containing <code>overflow@example.com</code> in the above example). How do I use <code>uniq</code> on only field 1 (separated by commas)? According to <code>man</code>, <code>uniq</code> doesn't have options for columns.

I tried something with <code>sort | uniq</code> but it doesn't work.

Is there a way to 'uniq' by column?

我有一个这样的.csv文件：stack2@example.com,2009-11-27 01:05:47.893000000,example.net,127.0.0.1overflow@example.com,2009-11-27 00:58:29.793000000,example.net,255.255.255.0...

问有没有办法通过列来“uniq”？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问有没有办法通过列来“uniq”？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问有没有办法通过列来“uniq”？
EN