blocks|key|754275|text|def+unique_list(l):
++++ulist+=+[]
++++[ulist.append(x)+for+x+in+l+if+x+not+in+ulist]
++++return+ulist

a="calvin+klein+design+dress+calvin+klein"
a='+'.join(unique_list(a.split()))|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|754276|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>def unique_list(l):
 ulist = []
 [ulist.append(x) for x in l if x not in ulist]
 return ulist

a="calvin klein design dress calvin klein"
a=' '.join(unique_list(a.split()))
</code></pre>

blocks|key|537931|text|string1+=+"calvin+klein+design+dress+calvin+klein"
words+=+string1.split()
print+("+".join(sorted(set(words),+key=words.index)))|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|537932|这将根据原始单词列表中的单词索引对字符串中的所有(唯一)单词集进行排序。|unstyled|537933|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|J|8|@]|9|@]|A|$]]|$1|G|3|-4|5|F|7|K|8|@]|9|@]|A|$]]]|H|$]]

<pre><code>string1 = "calvin klein design dress calvin klein"
words = string1.split()
print (" ".join(sorted(set(words), key=words.index)))
</code></pre>

This sorts the set of all the (unique) words in your string by the word's index in the original list of words.

blocks|key|754350|text|在Python+2.7%2B中，您可以使用collections.OrderedDict来执行以下操作：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|754351|from+collections+import+OrderedDict
s+=+"calvin+klein+design+dress+calvin+klein"
print+'+'.join(OrderedDict((w,w)+for+w+in+s.split()).keys())|code-block|syntax|javascript|754352|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python.org/dev/library/collections.html#collections.OrderedDict^0|J|N|J|N|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]]|D|@$9|V|A|W|1|X]]|E|$]]|$1|F|3|G|5|H|7|Y|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Z|8|@]|D|@]|E|$]]]|L|$M|$5|N|O|P|E|$Q|R]]]]

In Python 2.7+, you could use <a href="http://docs.python.org/dev/library/collections.html#collections.OrderedDict" rel="noreferrer"><code>collections.OrderedDict</code></a> for this:

<pre><code>from collections import OrderedDict
s = "calvin klein design dress calvin klein"
print ' '.join(OrderedDict((w,w) for w in s.split()).keys())
</code></pre>

blocks|key|754398|text|从itertools+recipes中剪切和粘贴|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|754399|from+itertools+import+ifilterfalse

def+unique_everseen(iterable,+key=None):
++++"List+unique+elements,+preserving+order.+Remember+all+elements+ever+seen."
++++#+unique_everseen('AAAABBBCCDAABBB')+-->+A+B+C+D
++++#+unique_everseen('ABBCcAD',+str.lower)+-->+A+B+C+D
++++seen+=+set()
++++seen_add+=+seen.add
++++if+key+is+None:
++++++++for+element+in+ifilterfalse(seen.__contains__,+iterable):
++++++++++++seen_add(element)
++++++++++++yield+element
++++else:
++++++++for+element+in+iterable:
++++++++++++k+=+key(element)
++++++++++++if+k+not+in+seen:
++++++++++++++++seen_add(k)
++++++++++++++++yield+element|code-block|syntax|javascript|754400|我真的希望他们能尽快把这些食谱做成一个模块。我非常希望能够做from+itertools_recipes+import+unique_everseen，而不是每次我需要东西的时候都使用剪切粘贴。|style|CODE|754401|像这样使用：|754402|def+unique_words(string,+ignore_case=False):
++++key+=+None
++++if+ignore_case:
++++++++key+=+str.lower
++++return+"+".join(unique_everseen(string.split(),+key=key))

string2+=+unique_words(string1)|754403|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python.org/library/itertools.html#recipes^0|1|H|0|0|0|U|19|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@$A|Z|B|10|1|11]]|C|$]]|$1|D|3|E|5|F|7|12|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|13|8|@$A|14|B|15|K|L]]|9|@]|C|$]]|$1|M|3|N|5|6|7|16|8|@]|9|@]|C|$]]|$1|O|3|P|5|F|7|17|8|@]|9|@]|C|$G|H]]|$1|Q|3|-4|5|6|7|18|8|@]|9|@]|C|$]]]|R|$S|$5|T|U|V|C|$W|X]]]]

Cut and paste from the <a href="http://docs.python.org/library/itertools.html#recipes" rel="noreferrer">itertools recipes</a>

<pre><code>from itertools import ifilterfalse

def unique_everseen(iterable, key=None):
 "List unique elements, preserving order. Remember all elements ever seen."
 # unique_everseen('AAAABBBCCDAABBB') --&gt; A B C D
 # unique_everseen('ABBCcAD', str.lower) --&gt; A B C D
 seen = set()
 seen_add = seen.add
 if key is None:
 for element in ifilterfalse(seen.__contains__, iterable):
 seen_add(element)
 yield element
 else:
 for element in iterable:
 k = key(element)
 if k not in seen:
 seen_add(k)
 yield element
</code></pre>

I really wish they could go ahead and make a module out of those recipes soon. I'd very much like to be able to do <code>from itertools_recipes import unique_everseen</code> instead of using cut-and-paste every time I need something.

Use like this:

<pre><code>def unique_words(string, ignore_case=False):
 key = None
 if ignore_case:
 key = str.lower
 return " ".join(unique_everseen(string.split(), key=key))

string2 = unique_words(string1)
</code></pre>

blocks|key|537893|text|string+=+'calvin+klein+design+dress+calvin+klein'

def+uniquify(string):
++++output+=+[]
++++seen+=+set()
++++for+word+in+string.split():
++++++++if+word+not+in+seen:
++++++++++++output.append(word)
++++++++++++seen.add(word)
++++return+'+'.join(output)

print+uniquify(string)|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|537894|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>string = 'calvin klein design dress calvin klein'

def uniquify(string):
 output = []
 seen = set()
 for word in string.split():
 if word not in seen:
 output.append(word)
 seen.add(word)
 return ' '.join(output)

print uniquify(string)
</code></pre>

blocks|key|537732|text|您可以使用集合来跟踪已处理的单词。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|537733|words+=+set()
result+=+''
for+word+in+string1.split():
++++if+word+not+in+words:
++++++++result+=+result+%2B+word+%2B+'+'
++++++++words.add(word)
print+result|code-block|syntax|javascript|537734|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You can use a set to keep track of already processed words.

<pre><code>words = set()
result = ''
for word in string1.split():
 if word not in words:
 result = result + word + ' '
 words.add(word)
print result
</code></pre>

blocks|key|754490|text|有几个答案非常接近这一点，但并没有完全达到我想要的结果：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|754491|def+uniques(+your_string+):++++
++++seen+=+set()
++++return+'+'.join(+seen.add(i)+or+i+for+i+in+your_string.split()+if+i+not+in+seen+)|code-block|syntax|javascript|754492|当然，如果你想让它更干净或者更快，我们可以重构一下：|754493|def+uniques(+your_string+):++++
++++words+=+your_string.split()

++++seen+=+set()
++++seen_add+=+seen.add

++++def+add(x):
++++++++seen_add(x)++
++++++++return+x

++++return+'+'.join(+add(i)+for+i+in+words+if+i+not+in+seen+)|754494|我认为第二个版本是你在少量代码中所能获得的最好的性能。(可以使用更多的代码在一次扫描输入字符串中完成所有工作，但对于大多数工作负载来说，这应该足够了。)|754495|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

Several answers are pretty close to this but haven't quite ended up where I did:

<pre><code>def uniques( your_string ): 
 seen = set()
 return ' '.join( seen.add(i) or i for i in your_string.split() if i not in seen )
</code></pre>

Of course, if you want it a tiny bit cleaner or faster, we can refactor a bit:

<pre><code>def uniques( your_string ): 
 words = your_string.split()

 seen = set()
 seen_add = seen.add

 def add(x):
 seen_add(x) 
 return x

 return ' '.join( add(i) for i in words if i not in seen )
</code></pre>

I think the second version is about as performant as you can get in a small amount of code. (More code could be used to do all the work in a single scan across the input string but for most workloads, this should be sufficient.)

blocks|key|754833|text|使用numpy函数使导入最好有一个别名作为导入(作为np)|type|unstyled|depth|inlineStyleRanges|entityRanges|data|754834|import+numpy+as+np|code-block|syntax|javascript|754835|然后，您可以像这样使用它，以便从数组中删除重复项|754836|no_duplicates_array+=+np.unique(your_array)|754837|对于您的情况，如果想要字符串形式的结果，可以使用|754838|no_duplicates_string+=+'+'.join(np.unique(your_string.split()))|754839|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|S|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|T|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|U|8|@]|9|@]|A|$]]|$1|M|3|N|5|D|7|V|8|@]|9|@]|A|$E|F]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

Use numpy function
make an import its better to have an alias for the import (as np)

<pre><code>import numpy as np
</code></pre>

and then you can bing it like this
for removing duplicates from array you can use it this way

<pre><code>no_duplicates_array = np.unique(your_array)
</code></pre>

for your case if you want result in string you can use

<pre><code>no_duplicates_string = ' '.join(np.unique(your_string.split()))
</code></pre>

blocks|key|754504|text|11和2完美地工作：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|754505|++++s="the+sky+is+blue+very+blue"
++++s=s.lower()
++++slist+=+s.split()
++++print+"+".join(sorted(set(slist),+key=slist.index))|code-block|syntax|javascript|754506|和2|754507|++++s="the+sky+is+blue+very+blue"
++++s=s.lower()
++++slist+=+s.split()
++++print+"+".join(sorted(set(slist),+key=slist.index))|754508|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

11 and 2 work perfectly:

<pre><code> s="the sky is blue very blue"
 s=s.lower()
 slist = s.split()
 print " ".join(sorted(set(slist), key=slist.index))
</code></pre>

and 2

<pre><code> s="the sky is blue very blue"
 s=s.lower()
 slist = s.split()
 print " ".join(sorted(set(slist), key=slist.index))
</code></pre>

blocks|key|538031|text|问:删除字符串中的重复项|type|unstyled|depth|inlineStyleRanges|entityRanges|data|538032|+from+_collections+import+OrderedDict

++++a+=+"Gina+Gini+Gini+Protijayi"

++++aa+=+OrderedDict().fromkeys(a.split())
++++print('+'.join(aa))
+++#+output+=>+Gina+Gini+Protijayi|code-block|syntax|javascript|538033|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Question: Remove the duplicates in a string

<pre><code> from _collections import OrderedDict

 a = "Gina Gini Gini Protijayi"

 aa = OrderedDict().fromkeys(a.split())
 print(' '.join(aa))
 # output =&gt; Gina Gini Protijayi
</code></pre>

blocks|key|538111|text|您可以使用以下代码从文本文件或字符串中删除重复或重复的单词：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|538112|from+collections+import+Counter
for+lines+in+all_words:

++++line=''.join(lines.lower())
++++new_data1='+'.join(lemmatize_sentence(line))
++++new_data2+=+word_tokenize(new_data1)
++++new_data3=nltk.pos_tag(new_data2)

++++#+below+code+is+for+removal+of+repeated+words

++++for+i+in+range(0,+len(new_data3)):
++++++++new_data3[i]+=+"".join(new_data3[i])
++++UniqW+=+Counter(new_data3)
++++new_data5+=+"+".join(UniqW.keys())
++++print+(new_data5)


++++new_data.append(new_data5)


print+(new_data)|code-block|syntax|javascript|538113|根据要求提供P.S.+-Do标识。希望这能有所帮助！|538114|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

You can remove duplicate or repeated words from a text file or string using following codes - 

<pre><code>from collections import Counter
for lines in all_words:

 line=''.join(lines.lower())
 new_data1=' '.join(lemmatize_sentence(line))
 new_data2 = word_tokenize(new_data1)
 new_data3=nltk.pos_tag(new_data2)

 # below code is for removal of repeated words

 for i in range(0, len(new_data3)):
 new_data3[i] = "".join(new_data3[i])
 UniqW = Counter(new_data3)
 new_data5 = " ".join(UniqW.keys())
 print (new_data5)


 new_data.append(new_data5)


print (new_data)
</code></pre>

P.S. -Do identations as per required.
Hope this helps!!!

blocks|key|754774|text|不使用split函数(在面试中会有所帮助)|type|unstyled|depth|inlineStyleRanges|entityRanges|data|754775|def+unique_words2(a):
++++words+=+[]
++++spaces+=+'+'
++++length+=+len(a)
++++i+=+0
++++while+i+<+length:
++++++++if+a[i]+not+in+spaces:
++++++++++++word_start+=+i
++++++++++++while+i+<+length+and+a[i]+not+in+spaces:
++++++++++++++++i+%2B=+1
++++++++++++words.append(a[word_start:i])
++++++++i+%2B=+1
++++words_stack+=+[]
++++for+val+in+words:++#
++++++++if+val+not+in+words_stack:++#+We+can+replace+these+three+lines+with+this+one+->+[words_stack.append(val)+for+val+in+words+if+val+not+in+words_stack]
++++++++++++words_stack.append(val)++#
++++print('+'.join(words_stack))++#+or+return,+your+choice


unique_words2('calvin+klein+design+dress+calvin+klein')+|code-block|syntax|javascript|754776|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Without using the split function (will help in interviews)

<pre><code>def unique_words2(a):
 words = []
 spaces = ' '
 length = len(a)
 i = 0
 while i &lt; length:
 if a[i] not in spaces:
 word_start = i
 while i &lt; length and a[i] not in spaces:
 i += 1
 words.append(a[word_start:i])
 i += 1
 words_stack = []
 for val in words: #
 if val not in words_stack: # We can replace these three lines with this one -&gt; [words_stack.append(val) for val in words if val not in words_stack]
 words_stack.append(val) #
 print(' '.join(words_stack)) # or return, your choice


unique_words2('calvin klein design dress calvin klein') 
</code></pre>

blocks|key|538345|text|正在初始化列表|type|unstyled|depth|inlineStyleRanges|entityRanges|data|538346|listA+=+[+'xy-xy',+'pq-qr',+'xp-xp-xp',+'dd-ee']

print("Given+list+:+",listA)|code-block|syntax|javascript|538347|使用set()和split()|offset|length|style|CODE|538348|res+=+[set(sub.split('-'))+for+sub+in+listA]|538349|结果|538350|print("List+after+duplicate+removal+:",+res)+|538351|entityMap^0|0|0|2|5|8|7|0|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|V|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|W|8|@$I|X|J|Y|K|L]|$I|Z|J|10|K|L]]|9|@]|A|$]]|$1|M|3|N|5|D|7|11|8|@]|9|@]|A|$E|F]]|$1|O|3|P|5|6|7|12|8|@]|9|@]|A|$]]|$1|Q|3|R|5|D|7|13|8|@]|9|@]|A|$E|F]]|$1|S|3|-4|5|6|7|14|8|@]|9|@]|A|$]]]|T|$]]

<h4>initializing list</h4>
<pre><code>listA = [ 'xy-xy', 'pq-qr', 'xp-xp-xp', 'dd-ee']

print(&quot;Given list : &quot;,listA)
</code></pre>
<h4>using <code>set()</code> and <code>split()</code></h4>
<pre><code>res = [set(sub.split('-')) for sub in listA]
</code></pre>
<h4>Result</h4>
<pre><code>print(&quot;List after duplicate removal :&quot;, res) 
</code></pre>

blocks|key|538199|text|您可以简单地通过获取与字符串相关联的集合来做到这一点，根据定义，该字符串是一个不包含重复元素的数学对象。它可以将集合中的单词重新连接到一个字符串中：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|538200|def+remove_duplicate_words(string):
++++++++x+=+string.split()
++++++++x+=+sorted(set(x),+key+=+x.index)
++++++++return+'+'.join(x)|code-block|syntax|javascript|538201|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You can do that simply by getting the set associated to the string, which is a mathematical object containing no repeated elements by definition. It suffices to join the words in the set back into a string:
<pre><code>def remove_duplicate_words(string):
 x = string.split()
 x = sorted(set(x), key = x.index)
 return ' '.join(x)
</code></pre>

Following example:

<pre><code>string1 = "calvin klein design dress calvin klein"
</code></pre>

How can I remove the second two duplicates <code>"calvin"</code> and <code>"klein"</code>?

The result should look like 

<pre><code>string2 = "calvin klein design dress"
</code></pre>

only the second duplicates should be removed and the sequence of the words should not be changed!

How can I remove duplicate words in a string with Python?

下面是示例：string1 = "calvin klein design dress calvin klein"如何删除第二个重复的"calvin"和"klein"结果应该如下所示string2 = "calvin klein design dress"只应删除第二个重复项，并且不应更改单词的顺序！

问如何使用Python删除字符串中的重复单词？
EN

回答 14

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何使用Python删除字符串中的重复单词？EN

回答 14

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何使用Python删除字符串中的重复单词？
EN