blocks|key|843660|text|我对python没有经验，但是一种算法是对列表进行排序，然后删除重复项(通过比较列表中的前一项)，最后通过与旧列表的比较来找到新列表中的位置。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|843661|较长答案：http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560|offset|length|843662|entityMap|0|LINK|mutability|MUTABLE|url|http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560^0|0|5|1P|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|O|8|@]|9|@$D|P|E|Q|1|R]]|A|$]]|$1|F|3|-4|5|6|7|S|8|@]|9|@]|A|$]]]|G|$H|$5|I|J|K|A|$L|M]]]]

I have no experience with python, but an algorithm would be to sort the list, then remove duplicates (by comparing to previous items in the list), and finally find the position in the new list by comparing with the old list.

Longer answer: <a href="http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560" rel="nofollow noreferrer">http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560</a>

blocks|key|843666|text|我还没有做过任何测试，但一个可能的算法可能是创建第二个列表，并遍历第一个列表。如果某项不在第二个列表中，则将其添加到第二个列表中。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|843667|x+=+[1,+1,+2,+'a',+'a',+3]
y+=+[]
for+each+in+x:
++++if+each+not+in+y:
++++++++y.append(each)|code-block|syntax|javascript|843668|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

I haven't done any tests, but one possible algorithm might be to create a second list, and iterate through the first list. If an item is not in the second list, add it to the second list.

<pre><code>x = [1, 1, 2, 'a', 'a', 3]
y = []
for each in x:
 if each not in y:
 y.append(each)
</code></pre>

blocks|key|1153315|text|取自http://www.peterbe.com/plog/uniqifiers-benchmark|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1153316|def+f5(seq,+idfun=None):++
++++#+order+preserving+
++++if+idfun+is+None:+
++++++++def+idfun(x):+return+x+
++++seen+=+{}+
++++result+=+[]+
++++for+item+in+seq:+
++++++++marker+=+idfun(item)+
++++++++#+in+old+Python+versions:+
++++++++#+if+seen.has_key(marker)+
++++++++#+but+in+new+ones:+
++++++++if+marker+in+seen:+continue+
++++++++seen[marker]+=+1+
++++++++result.append(item)+
++++return+result|code-block|syntax|javascript|1153317|entityMap|0|LINK|mutability|MUTABLE|url|http://www.peterbe.com/plog/uniqifiers-benchmark^0|2|1C|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

Taken from <a href="http://www.peterbe.com/plog/uniqifiers-benchmark" rel="noreferrer">http://www.peterbe.com/plog/uniqifiers-benchmark</a>

<pre><code>def f5(seq, idfun=None): 
 # order preserving 
 if idfun is None: 
 def idfun(x): return x 
 seen = {} 
 result = [] 
 for item in seq: 
 marker = idfun(item) 
 # in old Python versions: 
 # if seen.has_key(marker) 
 # but in new ones: 
 if marker in seen: continue 
 seen[marker] = 1 
 result.append(item) 
 return result
</code></pre>

blocks|key|1153327|text|>>>+def+unique(list):
...+++y+=+[]
...+++for+x+in+list:
...+++++if+x+not+in+y:
...+++++++y.append(x)
...+++return+y|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1153328|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>&gt;&gt;&gt; def unique(list):
... y = []
... for x in list:
... if x not in y:
... y.append(x)
... return y
</code></pre>

blocks|key|1153377|text|什么是最快的取决于您的列表中的百分比是重复的。如果它几乎是所有重复的，几乎没有唯一的项目，创建一个新的列表可能会更快。如果它主要是唯一的项目，从原始列表(或副本)中删除它们将更快。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1153378|这里有一个用来修改列表的方法：|1153379|def+unique(items):
++seen+=+set()
++for+i+in+xrange(len(items)-1,+-1,+-1):
++++it+=+items[i]
++++if+it+in+seen:
++++++del+items[i]
++++else:
++++++seen.add(it)|code-block|syntax|javascript|1153380|对索引进行反向迭代确保删除项不影响迭代。|1153381|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|N|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|O|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

What's going to be fastest depends on what percentage of your list is duplicates. If it's nearly all duplicates, with few unique items, creating a new list will probably be faster. If it's mostly unique items, removing them from the original list (or a copy) will be faster.

Here's one for modifying the list in place:

<pre><code>def unique(items):
 seen = set()
 for i in xrange(len(items)-1, -1, -1):
 it = items[i]
 if it in seen:
 del items[i]
 else:
 seen.add(it)
</code></pre>

Iterating backwards over the indices ensures that removing items doesn't affect the iteration.

blocks|key|843697|text|O(n)如果dict是散列，则O(nlogn)，如果dict是树，并且简单，固定。谢谢马修的建议。对不起，我不知道潜在的类型。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|843698|def+unique(x):++++
++output+=+[]
++y+=+{}
++for+item+in+x:
++++y[item]+=+""

++for+item+in+x:
++++if+item+in+y:
++++++output.append(item)

++return+output|code-block|syntax|javascript|843699|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

O(n) if dict is hash, O(nlogn) if dict is tree, and simple, fixed. Thanks to Matthew for the suggestion. Sorry I don't know the underlying types.

<pre><code>def unique(x): 
 output = []
 y = {}
 for item in x:
 y[item] = ""

 for item in x:
 if item in y:
 output.append(item)

 return output
</code></pre>

blocks|key|839694|text|def+unique(items):
++++found+=+set()
++++keep+=+[]

++++for+item+in+items:
++++++++if+item+not+in+found:
++++++++++++found.add(item)
++++++++++++keep.append(item)
++++++++++++
++++return+keep

print+unique([1,+1,+2,+'a',+'a',+3])|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|839695|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>def unique(items):
 found = set()
 keep = []

 for item in items:
 if item not in found:
 found.add(item)
 keep.append(item)
 
 return keep

print unique([1, 1, 2, 'a', 'a', 3])
</code></pre>

blocks|key|1153441|text|实际上，您可以在Python中做一些很酷的事情来解决这个问题。您可以创建一个列表理解，以便在构建它时引用它自己。详情如下：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1153442|+++#+remove+duplicates...
+++def+unique(my_list):
+++++++return+[x+for+x+in+my_list+if+x+not+in+locals()['_[1]'].__self__]|code-block|syntax|javascript|1153443|编辑：我删除了"self"，它在Mac上运行，Python2.5.1.|offset|length|style|BOLD|1153444|_1是Python对新列表的“秘密”引用。当然，上面的内容有点混乱，但是您可以根据需要对其进行调整。例如，您实际上可以编写一个返回对理解的引用的函数；它看起来更像：|1153445|return+[x+for+x+in+my_list+if+x+not+in+this_list()]|1153446|entityMap^0|0|0|3|W|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|T|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|U|8|@$I|V|J|W|K|L]]|9|@]|A|$]]|$1|M|3|N|5|6|7|X|8|@]|9|@]|A|$]]|$1|O|3|P|5|D|7|Y|8|@]|9|@]|A|$E|F]]|$1|Q|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]]|R|$]]

You can actually do something really cool in Python to solve this. You can create a list comprehension that would reference itself as it is being built. As follows:

<pre><code> # remove duplicates...
 def unique(my_list):
 return [x for x in my_list if x not in locals()['_[1]'].__self__]
</code></pre>

Edit: I removed the "self", and it works on Mac OS X, Python 2.5.1.

The _[1] is Python's "secret" reference to the new list. The above, of course, is a little messy, but you could adapt it fit your needs as necessary. For example, you can actually write a function that returns a reference to the comprehension; it would look more like:

<pre><code>return [x for x in my_list if x not in this_list()]
</code></pre>

<hr>

blocks|key|1153473|text|>>>+x=[1,1,2,'a','a',3]
>>>+y+=+[+_x+for+_x+in+x+if+not+_x+in+locals()['_[1]']+]
>>>+y
[1,+2,+'a',+3]|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1153474|“局部变量()[‘_1’]”是正在创建的列表的“秘密名称”。|unstyled|1153475|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|J|8|@]|9|@]|A|$]]|$1|G|3|-4|5|F|7|K|8|@]|9|@]|A|$]]]|H|$]]

<pre><code>&gt;&gt;&gt; x=[1,1,2,'a','a',3]
&gt;&gt;&gt; y = [ _x for _x in x if not _x in locals()['_[1]'] ]
&gt;&gt;&gt; y
[1, 2, 'a', 3]
</code></pre>

 
"locals()['_[1]']" is the "secret name" of the list being created.

blocks|key|843755|text|单线：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|843756|new_list+=+reduce(lambda+x,y:+x%2B[y][:1-int(y+in+x)],+my_list,+[])|code-block|syntax|javascript|843757|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

One-liner:

<pre><code>new_list = reduce(lambda x,y: x+[y][:1-int(y in x)], my_list, [])
</code></pre>

blocks|key|1153506|text|复制件首先需要在列表中吗？在查找元素时没有开销，但是添加元素的开销要多一点(虽然开销应该是O(1)+)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1153507|>>>+x++=+[]
>>>+y+=+set()
>>>+def+add_to_x(val):
...+++++if+val+not+in+y:
...+++++++++++++x.append(val)
...+++++++++++++y.add(val)
...+++++print+x
...+++++print+y
...+
>>>+add_to_x(1)
[1]
set([1])
>>>+add_to_x(1)
[1]
set([1])
>>>+add_to_x(1)
[1]
set([1])
>>>+|code-block|syntax|javascript|1153508|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Do the duplicates necessarily need to be in the list in the first place? There's no overhead as far as looking the elements up, but there is a little bit more overhead in adding elements (though the overhead should be O(1) ).

<pre><code>&gt;&gt;&gt; x = []
&gt;&gt;&gt; y = set()
&gt;&gt;&gt; def add_to_x(val):
... if val not in y:
... x.append(val)
... y.add(val)
... print x
... print y
... 
&gt;&gt;&gt; add_to_x(1)
[1]
set([1])
&gt;&gt;&gt; add_to_x(1)
[1]
set([1])
&gt;&gt;&gt; add_to_x(1)
[1]
set([1])
&gt;&gt;&gt; 
</code></pre>

blocks|key|1153572|text|python中的has_key是O(1)。从散列插入和检索也是O(1)。循环遍历n项两次，所以O(n)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1153573|def+unique(list):
++s+=+{}
++output+=+[]
++for+x+in+list:
++++count+=+1
++++if(s.has_key(x)):
++++++count+=+s[x]+%2B+1

++++s[x]+=+count
++for+x+in+list:
++++count+=+s[x]
++++if(count+>+0):
++++++s[x]+=+0
++++++output.append(x)
++return+output|code-block|syntax|javascript|1153574|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

has_key in python is O(1). Insertion and retrieval from a hash is also O(1). Loops through n items twice, so O(n).

<pre><code>def unique(list):
 s = {}
 output = []
 for x in list:
 count = 1
 if(s.has_key(x)):
 count = s[x] + 1

 s[x] = count
 for x in list:
 count = s[x]
 if(count &gt; 0):
 s[x] = 0
 output.append(x)
 return output
</code></pre>

blocks|key|1153598|text|一次传球。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1153599|a+=+[1,1,'a','b','c','c']

new_list+=+[]
prev+=+None

while+1:
++++try:
++++++++i+=+a.pop(0)
++++++++if+i+!=+prev:
++++++++++++new_list.append(i)
++++++++prev+=+i
++++except+IndexError:
++++++++break|code-block|syntax|javascript|1153600|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

One pass.

<pre><code>a = [1,1,'a','b','c','c']

new_list = []
prev = None

while 1:
 try:
 i = a.pop(0)
 if i != prev:
 new_list.append(i)
 prev = i
 except IndexError:
 break
</code></pre>

blocks|key|843894|text|使用：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|843895|lst+=+[8,+8,+9,+9,+7,+15,+15,+2,+20,+13,+2,+24,+6,+11,+7,+12,+4,+10,+18,+13,+23,+11,+3,+11,+12,+10,+4,+5,+4,+22,+6,+3,+19,+14,+21,+11,+1,+5,+14,+8,+0,+1,+16,+5,+10,+13,+17,+1,+16,+17,+12,+6,+10,+0,+3,+9,+9,+3,+7,+7,+6,+6,+7,+5,+14,+18,+12,+19,+2,+8,+9,+0,+8,+4,+5]|code-block|syntax|javascript|843896|并使用timeit模块：|843897|$+python+-m+timeit+-s+'import+uniquetest'+'uniquetest.etchasketch(uniquetest.lst)'|843898|诸如此类的其他功能(我以他们的海报命名)，我有以下结果(关于我的第一代英特尔MacBook+Pro)：|843899|Allen:++++++++++++++++++14.6+µs+per+loop+[1]
Terhorst:+++++++++++++++26.6+µs+per+loop
Tarle:++++++++++++++++++44.7+µs+per+loop
ctcherry:+++++++++++++++44.8+µs+per+loop
Etchasketch+1+(short):++64.6+µs+per+loop
Schinckel:++++++++++++++65.0+µs+per+loop
Etchasketch+2:++++++++++71.6+µs+per+loop
Little:+++++++++++++++++89.4+µs+per+loop
Tyler:+++++++++++++++++179.0+µs+per+loop|843900|注意，艾伦修改了列表--我认为这已经扭曲了时间，因为timeit模块运行了100000次代码，其中99999次使用的是无欺骗列表。|offset|length|style|CODE|843901|汇总：使用集合的直进实现胜于令人困惑的一行程序:-)|BOLD|843902|entityMap^0|0|0|0|0|0|0|Q|6|0|0|2|0^^$0|@$1|2|3|4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|10|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|11|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|12|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|13|8|@]|9|@]|A|$]]|$1|M|3|N|5|D|7|14|8|@]|9|@]|A|$E|F]]|$1|O|3|P|5|6|7|15|8|@$Q|16|R|17|S|T]]|9|@]|A|$]]|$1|U|3|V|5|6|7|18|8|@$Q|19|R|1A|S|W]]|9|@]|A|$]]|$1|X|3|-4|5|6|7|1B|8|@]|9|@]|A|$]]]|Y|$]]

Using:

<pre><code>lst = [8, 8, 9, 9, 7, 15, 15, 2, 20, 13, 2, 24, 6, 11, 7, 12, 4, 10, 18, 13, 23, 11, 3, 11, 12, 10, 4, 5, 4, 22, 6, 3, 19, 14, 21, 11, 1, 5, 14, 8, 0, 1, 16, 5, 10, 13, 17, 1, 16, 17, 12, 6, 10, 0, 3, 9, 9, 3, 7, 7, 6, 6, 7, 5, 14, 18, 12, 19, 2, 8, 9, 0, 8, 4, 5]
</code></pre>

And using the timeit module:

<pre><code>$ python -m timeit -s 'import uniquetest' 'uniquetest.etchasketch(uniquetest.lst)'
</code></pre>

And so on for the various other functions (which I named after their posters), I have the following results (on my first generation Intel MacBook Pro):

<pre><code>Allen: 14.6 µs per loop [1]
Terhorst: 26.6 µs per loop
Tarle: 44.7 µs per loop
ctcherry: 44.8 µs per loop
Etchasketch 1 (short): 64.6 µs per loop
Schinckel: 65.0 µs per loop
Etchasketch 2: 71.6 µs per loop
Little: 89.4 µs per loop
Tyler: 179.0 µs per loop
</code></pre>

[1] Note that Allen modifies the list in place – I believe this has skewed the time, in that the <code>timeit</code> module runs the code 100000 times and 99999 of them are with the dupe-less list.

<hr>

Summary: Straight-forward implementation with sets wins over confusing one-liners :-)

blocks|key|1153667|text|我不知道这个是不是快，但至少是简单的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1153668|简单地说，首先将其转换为一个集合，然后再转换为一个列表。|1153669|def+unique(container):
++return+list(set(container))|code-block|syntax|javascript|1153670|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

I don't know if this one is fast or not, but at least it is simple.

Simply, convert it first to a set and then again to a list

<pre><code>def unique(container):
 return list(set(container))
</code></pre>

blocks|key|154516|text|这是我发现的最快的就地方法(假设有很大比例的重复)：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|154517|def+unique(l):
++++s+=+set();+n+=+0
++++for+x+in+l:
++++++++if+x+not+in+s:+s.add(x);+l[n]+=+x;+n+%2B=+1
++++del+l[n:]|code-block|syntax|javascript|154518|这比艾伦的实现快了10%25，艾伦的实现是基于这个实现的(用timeit.repeat计时，由精神病学编译)。它保留任何副本的第一个实例。|154519|如果你能确认我的时间，我会感兴趣的。|154520|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

This is the fastest in-place method I've found (assuming a large proportion of duplicates):

<pre><code>def unique(l):
 s = set(); n = 0
 for x in l:
 if x not in s: s.add(x); l[n] = x; n += 1
 del l[n:]
</code></pre>

This is 10% faster than Allen's implementation, on which it is based (timed with timeit.repeat, JIT compiled by psyco). It keeps the first instance of any duplicate.

repton-infinity: I'd be interested if you could confirm my timings.

blocks|key|154538|text|这里有一些很好的有效的解决方案。但是，对于不关心绝对最有效的O(n)解决方案的人，我将使用简单的一行O(n%5E2*log(n))解决方案：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|154539|def+unique(xs):
++++return+sorted(set(xs),+key=lambda+x:+xs.index(x))|code-block|syntax|javascript|154540|或者更有效的双线性O(n*log(n))解决方案：|154541|def+unique(xs):
++++positions+=+dict((e,pos)+for+pos,e+in+reversed(list(enumerate(xs))))
++++return+sorted(set(xs),+key=lambda+x:+positions[x])|154542|entityMap^0|U|4|1E|D|0|0|9|B|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@$9|R|A|S|B|C]|$9|T|A|U|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|V|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|W|8|@$9|X|A|Y|B|C]]|D|@]|E|$]]|$1|M|3|N|5|H|7|Z|8|@]|D|@]|E|$I|J]]|$1|O|3|-4|5|6|7|10|8|@]|D|@]|E|$]]]|P|$]]

There are some great, efficient solutions here. However, for anyone not concerned with the absolute most efficient <code>O(n)</code> solution, I'd go with the simple one-liner <code>O(n^2*log(n))</code> solution:

<pre><code>def unique(xs):
 return sorted(set(xs), key=lambda x: xs.index(x))
</code></pre>

or the more efficient two-liner <code>O(n*log(n))</code> solution:

<pre><code>def unique(xs):
 positions = dict((e,pos) for pos,e in reversed(list(enumerate(xs))))
 return sorted(set(xs), key=lambda x: positions[x])
</code></pre>

blocks|key|154586|text|如果您在Terhost的答案中从调用set()中取出空列表，就会得到一些速度提升。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|154587|变更:+found+=+set([])|154588|to:+found+=+set()|154589|然而，你根本不需要这套设备。|154590|def+unique(items):
++++keep+=+[]

++++for+item+in+items:
++++++++if+item+not+in+keep:
++++++++++++keep.append(item)

++++return+keep|code-block|syntax|javascript|154591|使用timeit，我得到了以下结果：|154592|与set([])+-+4.97210427363|154593|与set()+-+4.65712377445|154594|没有设定--+3.44865284975|154595|entityMap^0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|X|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|Y|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|Z|8|@]|9|@]|A|$]]|$1|H|3|I|5|J|7|10|8|@]|9|@]|A|$K|L]]|$1|M|3|N|5|6|7|11|8|@]|9|@]|A|$]]|$1|O|3|P|5|6|7|12|8|@]|9|@]|A|$]]|$1|Q|3|R|5|6|7|13|8|@]|9|@]|A|$]]|$1|S|3|T|5|6|7|14|8|@]|9|@]|A|$]]|$1|U|3|-4|5|6|7|15|8|@]|9|@]|A|$]]]|V|$]]

If you take out the empty list from the call to set() in Terhost's answer, you get a little speed boost.

Change:
 found = set([]) 
to:
 found = set()

However, you don't need the set at all.

<pre><code>def unique(items):
 keep = []

 for item in items:
 if item not in keep:
 keep.append(item)

 return keep
</code></pre>

Using timeit I got these results:

with set([]) -- 4.97210427363 
with set() -- 4.65712377445 
with no set -- 3.44865284975

blocks|key|843960|text|强制性发电机变体：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|843961|def+unique(seq):
++seen+=+set()
++for+x+in+seq:
++++if+x+not+in+seen:
++++++seen.add(x)
++++++yield+x|code-block|syntax|javascript|843962|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Obligatory generator-based variation:

<pre><code>def unique(seq):
 seen = set()
 for x in seq:
 if x not in seen:
 seen.add(x)
 yield x
</code></pre>

blocks|key|1153927|text|a=[1,2,3,4,5,7,7,8,8,9,9,3,45]

def+unique(l):

++++ids={}
++++for+item+in+l:
++++++++if+not+ids.has_key(item):
++++++++++++ids[item]=item
++++return++ids.keys()
print+a

print+unique(a)|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1153928|插入元素将进行θ(N)检索，如果元素存在或不存在，则需要一定的时间来测试所有项也需要θ(N)，因此我们可以看到这个解将采用θ(N)。记住，python中的字典是由哈希表实现的。|unstyled|1153929|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|J|8|@]|9|@]|A|$]]|$1|G|3|-4|5|F|7|K|8|@]|9|@]|A|$]]]|H|$]]

<pre><code>a=[1,2,3,4,5,7,7,8,8,9,9,3,45]

def unique(l):

 ids={}
 for item in l:
 if not ids.has_key(item):
 ids[item]=item
 return ids.keys()
print a

print unique(a)
</code></pre>
Inserting elements will take theta(n)
retrieving if element is exiting or not will take constant time
testing all the items will take also theta(n)
so we can see that this solution will take theta(n).
Bear in mind that dictionary in python implemented by hash table.

blocks|key|154705|text|一条就地的一条衬里：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|154706|>>>+x+=+[1,+1,+2,+'a',+'a',+3]
>>>+[+item+for+pos,item+in+enumerate(x)+if+x.index(item)==pos+]
[1,+2,+'a',+3]|code-block|syntax|javascript|154707|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

An in-place one-liner for this:

<pre><code>&gt;&gt;&gt; x = [1, 1, 2, 'a', 'a', 3]
&gt;&gt;&gt; [ item for pos,item in enumerate(x) if x.index(item)==pos ]
[1, 2, 'a', 3]
</code></pre>

blocks|key|844008|text|移除复本并保持秩序：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|844009|这是一个快速的2线，利用内置功能的列表理解和切分。|844010|x+=+[1,+1,+2,+'a',+'a',+3]

tmpUniq+=+{}+#+temp+variable+used+below+
results+=+[tmpUniq.setdefault(i,i)+for+i+in+x+if+i+not+in+tmpUniq]

print+results
[1,+2,+'a',+3]|code-block|syntax|javascript|844011|dict.setdefaults()函数返回值，并将其直接添加到列表理解中的temp中。使用dict的内置函数和散列将最大限度地提高过程的效率。|844012|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|N|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|O|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Remove duplicates and preserve order:

This is a fast 2-liner that leverages built-in functionality of list comprehensions and dicts.

<pre><code>x = [1, 1, 2, 'a', 'a', 3]

tmpUniq = {} # temp variable used below 
results = [tmpUniq.setdefault(i,i) for i in x if i not in tmpUniq]

print results
[1, 2, 'a', 3]
</code></pre>

The dict.setdefaults() function returns the value as well as adding it to the temp dict directly in the list comprehension. Using the built-in functions and the hashes of the dict will work to maximize efficiency for the process.

blocks|key|844017|text|下面是迭代工具文档中的两个菜谱：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|844018|def+unique_everseen(iterable,+key=None):
++++"List+unique+elements,+preserving+order.+Remember+all+elements+ever+seen."
++++#+unique_everseen('AAAABBBCCDAABBB')+-->+A+B+C+D
++++#+unique_everseen('ABBCcAD',+str.lower)+-->+A+B+C+D
++++seen+=+set()
++++seen_add+=+seen.add
++++if+key+is+None:
++++++++for+element+in+ifilterfalse(seen.__contains__,+iterable):
++++++++++++seen_add(element)
++++++++++++yield+element
++++else:
++++++++for+element+in+iterable:
++++++++++++k+=+key(element)
++++++++++++if+k+not+in+seen:
++++++++++++++++seen_add(k)
++++++++++++++++yield+element

def+unique_justseen(iterable,+key=None):
++++"List+unique+elements,+preserving+order.+Remember+only+the+element+just+seen."
++++#+unique_justseen('AAAABBBCCDAABBB')+-->+A+B+C+D+A+B
++++#+unique_justseen('ABBCcAD',+str.lower)+-->+A+B+C+A+D
++++return+imap(next,+imap(itemgetter(1),+groupby(iterable,+key)))|code-block|syntax|javascript|844019|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Here are two recipes from the itertools documentation:

<pre><code>def unique_everseen(iterable, key=None):
 "List unique elements, preserving order. Remember all elements ever seen."
 # unique_everseen('AAAABBBCCDAABBB') --&gt; A B C D
 # unique_everseen('ABBCcAD', str.lower) --&gt; A B C D
 seen = set()
 seen_add = seen.add
 if key is None:
 for element in ifilterfalse(seen.__contains__, iterable):
 seen_add(element)
 yield element
 else:
 for element in iterable:
 k = key(element)
 if k not in seen:
 seen_add(k)
 yield element

def unique_justseen(iterable, key=None):
 "List unique elements, preserving order. Remember only the element just seen."
 # unique_justseen('AAAABBBCCDAABBB') --&gt; A B C D A B
 # unique_justseen('ABBCcAD', str.lower) --&gt; A B C A D
 return imap(next, imap(itemgetter(1), groupby(iterable, key)))
</code></pre>

blocks|key|1154007|text|这可能是最简单的方法：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1154008|list(OrderedDict.fromkeys(iterable))|code-block|syntax|javascript|1154009|从Python3.5开始，OrderedDict现在是用C实现的，所以它现在是最短、最干净和最快的。|1154010|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

This may be the simplest way:

<pre><code>list(OrderedDict.fromkeys(iterable))
</code></pre>

As of Python 3.5, OrderedDict is now implemented in C, so this was is now the shortest, cleanest, and fastest.

blocks|key|1154056|text|这是最快的一个，比较了来自这个冗长的讨论的所有东西和这里给出的其他答案，引用了这个基准测试。它比讨论中最快的函数f8快25%25。谢谢大卫·柯比的主意。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1154057|def+uniquify(seq):
++++seen+=+set()
++++seen_add+=+seen.add
++++return+[x+for+x+in+seq+if+x+not+in+seen+and+not+seen_add(x)]|code-block|syntax|javascript|1154058|一些时间比较：|1154059|$+python+uniqifiers_benchmark.py+
*+f8_original+3.76
*+uniquify+3.0
*+terhorst+5.44
*+terhorst_localref+4.08
*+del_dups+4.76|1154060|entityMap|0|LINK|mutability|MUTABLE|url|http://www.peterbe.com/plog/uniqifiers-benchmark|1|http://www.peterbe.com/plog/uniqifiers-benchmark/uniqifiers_benchmark.py^0|1K|2|F|5|0|15|4|1|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@$9|Z|A|10|B|C]]|D|@$9|11|A|12|1|13]|$9|14|A|15|1|16]]|E|$]]|$1|F|3|G|5|H|7|17|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|18|8|@]|D|@]|E|$]]|$1|M|3|N|5|H|7|19|8|@]|D|@]|E|$I|J]]|$1|O|3|-4|5|6|7|1A|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]|W|$5|R|S|T|E|$U|X]]]]

This is the fastest one, comparing all the stuff from this <a href="http://www.peterbe.com/plog/uniqifiers-benchmark" rel="nofollow">lengthy discussion</a> and the other answers given here, refering to this <a href="http://www.peterbe.com/plog/uniqifiers-benchmark/uniqifiers_benchmark.py" rel="nofollow">benchmark</a>. It's another 25% faster than the fastest function from the discussion, <code>f8</code>. Thanks to David Kirby for the idea.

<pre><code>def uniquify(seq):
 seen = set()
 seen_add = seen.add
 return [x for x in seq if x not in seen and not seen_add(x)]
</code></pre>

Some time comparison:

<pre><code>$ python uniqifiers_benchmark.py 
* f8_original 3.76
* uniquify 3.0
* terhorst 5.44
* terhorst_localref 4.08
* del_dups 4.76
</code></pre>

blocks|key|154823|text|x+=+[]+#+Your+list++of+items+that+includes+Duplicates

#+Assuming+that+your+list+contains+items+of+only+immutable+data+types

dict_x+=+{}+

dict_x+=+{item+:+item+for+i,+item+in+enumerate(x)+if+item+not+in+dict_x.keys()}
#+Average+t.c.+=+O(n)*+O(1)+;+furthermore+the+dict+comphrehension+and+generator+like+behaviour+of+enumerate+adds+a+certain+efficiency+and+pythonic+feel+to+it.

x+=+dict_x.keys()+#+if+you+want+your+output+in+list+format+|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|154824|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>x = [] # Your list of items that includes Duplicates

# Assuming that your list contains items of only immutable data types

dict_x = {} 

dict_x = {item : item for i, item in enumerate(x) if item not in dict_x.keys()}
# Average t.c. = O(n)* O(1) ; furthermore the dict comphrehension and generator like behaviour of enumerate adds a certain efficiency and pythonic feel to it.

x = dict_x.keys() # if you want your output in list format 
</code></pre>

For example:

<pre><code>&gt;&gt;&gt; x = [1, 1, 2, 'a', 'a', 3]
&gt;&gt;&gt; unique(x)
[1, 2, 'a', 3]
</code></pre>

Assume list elements are hashable.

Clarification: The result should keep the first duplicate in the list. For example, [1, 2, 3, 2, 3, 1] becomes [1, 2, 3].

In Python, what is the fastest algorithm for removing duplicates from a list so that all elements are unique *while preserving order*?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

例如：>>> x = [1, 1, 2, 'a', 'a', 3]>>> unique(x)[1, 2, 'a', 3]假设列表元素是可引用的。Clarification:结果应该保留列表中的第一个副本。例如，1，2，3，2，3，1变成1，2，3。

问在Python中，从列表中删除重复项的最快算法是什么，以便所有元素都是唯一的，同时保持顺序*？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Python中，从列表中删除重复项的最快算法是什么，以便所有元素都是唯一的，同时保持顺序*？EN