blocks|key|3125289|text|通过sort和uniq传输它们。这将删除所有重复项。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3125290|uniq+-d仅提供副本，uniq+-u仅提供唯一副本(条带副本)。|3125291|entityMap^0|2|4|7|4|0|0|7|D|7|0^^$0|@$1|2|3|4|5|6|7|J|8|@$9|K|A|L|B|C]|$9|M|A|N|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|O|8|@$9|P|A|Q|B|C]|$9|R|A|S|B|C]]|D|@]|E|$]]|$1|H|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|I|$]]

Pipe them through <code>sort</code> and <code>uniq</code>. This removes all duplicates.

<code>uniq -d</code> gives only the duplicates, <code>uniq -u</code> gives only the unique ones (strips duplicates).

blocks|key|3125302|text|对于可能不需要排序的较大数据集，您还可以使用以下perl脚本：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3125303|./yourscript.ksh+%7C+perl+-ne+'if+(!defined+$x{$_})+{+print+$_;+$x{$_}+=+1;+}'|code-block|syntax|javascript|3125304|这基本上只是记住每一行输出，这样它就不会再次输出它。|3125305|与"sort+%7C+uniq“解决方案相比，它的优势在于不需要预先排序。|offset|length|style|CODE|3125306|entityMap^0|0|0|0|2|B|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|S|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|T|8|@$K|U|L|V|M|N]]|9|@]|A|$]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

For larger data sets where sorting may not be desirable, you can also use the following perl script:

<pre><code>./yourscript.ksh | perl -ne 'if (!defined $x{$_}) { print $_; $x{$_} = 1; }'
</code></pre>

This basically just remembers every line output so that it doesn't output it again.

It has the advantage over the "<code>sort | uniq</code>" solution in that there's no sorting required up front.

blocks|key|3125313|text|使用zsh，您可以执行以下操作：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3125314|%25+cat+infile+
tar
more+than+one+word
gz
java
gz
java
tar
class
class
zsh-5.0.0[t]%25+print+-l+"${(fu)$(<infile)}"
tar
more+than+one+word
gz
java
class|code-block|syntax|javascript|3125315|或者您可以使用AWK：|3125316|%25+awk+'!_[$0]%2B%2B'+infile++++
tar
more+than+one+word
gz
java
class|3125317|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

With zsh you can do this:

<pre><code>% cat infile 
tar
more than one word
gz
java
gz
java
tar
class
class
zsh-5.0.0[t]% print -l "${(fu)$(&lt;infile)}"
tar
more than one word
gz
java
class
</code></pre>

Or you can use AWK:

<pre><code>% awk '!_[$0]++' infile 
tar
more than one word
gz
java
class
</code></pre>

blocks|key|3338243|text|./script.sh+%7C+sort+-u|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|3338244|这与monoxide's+answer相同，但更简洁一些。|unstyled|offset|length|3338245|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/users/15537/monoxide|1|https://stackoverflow.com/questions/618378/select-unique-or-distinct-values-from-a-list-in-unix-shell-script/618381#618381^0|0|2|A|0|D|6|1|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|T|8|@]|9|@$G|U|H|V|1|W]|$G|X|H|Y|1|Z]]|A|$]]|$1|I|3|-4|5|F|7|10|8|@]|9|@]|A|$]]]|J|$K|$5|L|M|N|A|$O|P]]|Q|$5|L|M|N|A|$O|R]]]]

<pre><code>./script.sh | sort -u
</code></pre>

This is the same as <a href="https://stackoverflow.com/users/15537/monoxide">monoxide's</a> <a href="https://stackoverflow.com/questions/618378/select-unique-or-distinct-values-from-a-list-in-unix-shell-script/618381#618381">answer</a>, but a bit more concise.

blocks|key|347784|text|唯一的，如要求的(但不排序)；|type|unstyled|depth|inlineStyleRanges|entityRanges|data|347785|对少于~70个元素使用更少的系统资源(根据时间测试)；|347786|写入以从标准输入中获取输入，|347787|(或修改并包含在另一个脚本中)：|347788|(Bash)|347789|bag2set+()+{
++++#+Reduce+a_bag+to+a_set.
++++local+-i+i+j+n=${#a_bag[@]}
++++for+((i=0;+i+<+n;+i%2B%2B));+do
++++++++if+[[+-n+${a_bag[i]}+]];+then
++++++++++++a_set[i]=${a_bag[i]}
++++++++++++a_bag[i]=$'\0'
++++++++++++for+((j=i%2B1;+j+<+n;+j%2B%2B));+do
++++++++++++++++[[+${a_set[i]}+==+${a_bag[j]}+]]+&&+a_bag[j]=$'\0'
++++++++++++done
++++++++fi
++++done
}
declare+-a+a_bag=()+a_set=()
stdin="$(</dev/stdin)"
declare+-i+i=0
for+e+in+$stdin;+do
++++a_bag[i]=$e
++++i=$i%2B1
done
bag2set
echo+"${a_set[@]}"|code-block|syntax|javascript|347790|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|S|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|T|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|U|8|@]|9|@]|A|$]]|$1|J|3|K|5|L|7|V|8|@]|9|@]|A|$M|N]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

Unique, as requested, (but not sorted); 
uses fewer system resources for less than ~70 elements (as tested with time); 
written to take input from stdin, 
(or modify and include in another script): 
(Bash)

<pre><code>bag2set () {
 # Reduce a_bag to a_set.
 local -i i j n=${#a_bag[@]}
 for ((i=0; i &lt; n; i++)); do
 if [[ -n ${a_bag[i]} ]]; then
 a_set[i]=${a_bag[i]}
 a_bag[i]=$'\0'
 for ((j=i+1; j &lt; n; j++)); do
 [[ ${a_set[i]} == ${a_bag[j]} ]] &amp;&amp; a_bag[j]=$'\0'
 done
 fi
 done
}
declare -a a_bag=() a_set=()
stdin="$(&lt;/dev/stdin)"
declare -i i=0
for e in $stdin; do
 a_bag[i]=$e
 i=$i+1
done
bag2set
echo "${a_set[@]}"
</code></pre>

blocks|key|3338276|text|使用AWK，您可以执行以下操作：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3338277|+./yourscript.ksh+%7C+awk+'!a[$0]%2B%2B'|code-block|syntax|javascript|3338278|我发现它比sort和uniq更快|3338279|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

With AWK you can do:
<pre><code> ./yourscript.ksh | awk '!a[$0]++'
</code></pre>
I find it faster than sort and uniq

blocks|key|348588|text|我得到了在文件中获取非重复条目的更好提示|type|unstyled|depth|inlineStyleRanges|entityRanges|data|348589|awk+'$0+!=+x+":FOO"+&&+NR>1+{print+x}+{x=$0}+END+{print}'+file_name+%7C+uniq+-f1+-u|code-block|syntax|javascript|348590|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

I get a better tips to get non-duplicate entries in a file

<pre class="lang-sh prettyprint-override"><code>awk '$0 != x ":FOO" &amp;&amp; NR&gt;1 {print x} {x=$0} END {print}' file_name | uniq -f1 -u

</code></pre>

I have a ksh script that returns a long list of values, newline separated, and I want to see only the unique/distinct values. It is possible to do this?

For example, say my output is file suffixes in a directory:

<blockquote>
<pre><code>tar
gz
java
gz
java
tar
class
class
</code></pre>
</blockquote>

I want to see a list like:

<blockquote>
<pre><code>tar
gz
java
class
</code></pre>
</blockquote>

Select unique or distinct values from a list in UNIX shell script

我有一个ksh脚本，它返回一个很长的值列表，其中以换行符分隔，并且我只想看到唯一/不同的值。这样做是可能的吗？例如，假设我的输出是目录中的文件后缀： tar gz java gz java tar类我希望看到这样的列表： tar gz java类

问从UNIX shell脚本的列表中选择唯一或不同的值
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问从UNIX shell脚本的列表中选择唯一或不同的值EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问从UNIX shell脚本的列表中选择唯一或不同的值
EN