blocks|key|1178376|text|怎么样|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1178377|import+re
splitter+=+re.compile(r'(\s%2B%7C\S%2B)')
splitter.findall(s)|code-block|syntax|javascript|1178378|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

How about

<pre><code>import re
splitter = re.compile(r'(\s+|\S+)')
splitter.findall(s)
</code></pre>

blocks|key|1393927|text|>>>+re.compile(r'(\s%2B)').split("\tthis+is+an++example")
['',+'\t',+'this',+'+',+'is',+'+',+'an',+'++',+'example']|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1393928|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>&gt;&gt;&gt; re.compile(r'(\s+)').split("\tthis is an example")
['', '\t', 'this', ' ', 'is', ' ', 'an', ' ', 'example']
</code></pre>

blocks|key|1393975|text|re模块提供以下功能：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1393976|>>>+import+re
>>>+re.split('(\W%2B)',+'Words,+words,+words.')
['Words',+',+',+'words',+',+',+'words',+'.',+'']|code-block|syntax|javascript|1393977|(引用自Python文档)。|1393978|对于您的示例(以空格拆分)，请使用re.split('(\s%2B)',+'\tThis+is+an+example')。|1393979|关键是要在捕获括号中包含要拆分的正则表达式。这样，分隔符就会添加到结果列表中。|1393980|编辑:如上所述，任何前面/后面的分隔符当然也会添加到列表中。为了避免这种情况，您可以先对输入字符串使用.strip()方法。|1393981|entityMap^0|0|2|0|0|0|H|15|0|0|1F|8|0^^$0|@$1|2|3|4|5|6|7|U|8|@$9|V|A|W|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|X|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|Y|8|@]|D|@]|E|$]]|$1|M|3|N|5|6|7|Z|8|@$9|10|A|11|B|C]]|D|@]|E|$]]|$1|O|3|P|5|6|7|12|8|@]|D|@]|E|$]]|$1|Q|3|R|5|6|7|13|8|@$9|14|A|15|B|C]]|D|@]|E|$]]|$1|S|3|-4|5|6|7|16|8|@]|D|@]|E|$]]]|T|$]]

the <code>re</code> module provides this functionality:

<pre><code>&gt;&gt;&gt; import re
&gt;&gt;&gt; re.split('(\W+)', 'Words, words, words.')
['Words', ', ', 'words', ', ', 'words', '.', '']
</code></pre>

(quoted from the Python documentation).

For your example (split on whitespace), use <code>re.split('(\s+)', '\tThis is an example')</code>.

The key is to enclose the regex on which to split in capturing parentheses. That way, the delimiters are added to the list of results. 

Edit: As pointed out, any preceding/trailing delimiters will of course also be added to the list. To avoid that you can use the <code>.strip()</code> method on your input string first.

blocks|key|1394030|text|感谢大家指向re模块，我仍然在尝试选择是使用我自己的函数返回一个序列……|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1394031|def+split_keep_delimiters(s,+delims="\t\n\r+"):
++++delim_group+=+s[0]+in+delims
++++start+=+0
++++for+index,+char+in+enumerate(s):
++++++++if+delim_group+!=+(char+in+delims):
++++++++++++delim_group+%5E=+True
++++++++++++yield+s[start:index]
++++++++++++start+=+index
++++yield+s[start:index%2B1]|code-block|syntax|javascript|1394032|如果我有时间，我会对它们进行xD基准测试|1394033|entityMap^0|6|2|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|S|8|@]|D|@]|E|$]]|$1|M|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|N|$]]

Thanks guys for pointing for the <code>re</code> module, I'm still trying to decide between that and using my own function that returns a sequence...

<pre><code>def split_keep_delimiters(s, delims="\t\n\r "):
 delim_group = s[0] in delims
 start = 0
 for index, char in enumerate(s):
 if delim_group != (char in delims):
 delim_group ^= True
 yield s[start:index]
 start = index
 yield s[start:index+1]
</code></pre>

If I had time I'd benchmark them xD

blocks|key|1394061|text|你看过pyparsing吗？从the+pyparsing+wiki借用的示例|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1394062|>>>+from+pyparsing+import+Word,+alphas
>>>+greet+=+Word(alphas)+%2B+","+%2B+Word(alphas)+%2B+"!"
>>>+hello1+=+'Hello,+World!'
>>>+hello2+=+'Greetings,+Earthlings!'
>>>+for+hello+in+hello1,+hello2:
...+++++print+(u'%25s+\u2192+%25r'+%25+(hello,+greet.parseString(hello))).encode('utf-8')
...+
Hello,+World!+→+(['Hello',+',',+'World',+'!'],+{})
Greetings,+Earthlings!+→+(['Greetings',+',',+'Earthlings',+'!'],+{})|code-block|syntax|javascript|1394063|entityMap|0|LINK|mutability|MUTABLE|url|http://pyparsing.wikispaces.com/file/view/greeting.py^0|F|I|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

Have you looked at pyparsing? Example borrowed from <a href="http://pyparsing.wikispaces.com/file/view/greeting.py" rel="nofollow noreferrer">the pyparsing wiki</a>:

<pre><code>&gt;&gt;&gt; from pyparsing import Word, alphas
&gt;&gt;&gt; greet = Word(alphas) + "," + Word(alphas) + "!"
&gt;&gt;&gt; hello1 = 'Hello, World!'
&gt;&gt;&gt; hello2 = 'Greetings, Earthlings!'
&gt;&gt;&gt; for hello in hello1, hello2:
... print (u'%s \u2192 %r' % (hello, greet.parseString(hello))).encode('utf-8')
... 
Hello, World! → (['Hello', ',', 'World', '!'], {})
Greetings, Earthlings! → (['Greetings', ',', 'Earthlings', '!'], {})
</code></pre>

Is there any equivalent to <code>str.split</code> in Python that also returns the delimiters?

I need to preserve the whitespace layout for my output after processing some of the tokens.

Example:

<pre><code>&gt;&gt;&gt; s="\tthis is an example"
&gt;&gt;&gt; print s.split()
['this', 'is', 'an', 'example']

&gt;&gt;&gt; print what_I_want(s)
['\t', 'this', ' ', 'is', ' ', 'an', ' ', 'example']
</code></pre>

Thanks!

tokenize a string keeping delimiters in Python

Python

在Python语言中有没有等价物str.split也会返回分隔符？在处理一些标记之后，我需要为输出保留空白布局。示例：>>> s="\tthis is an  example">>> print s.split()['this', 'is', 'an', 'example']>>> print what_I_want(...

问在Python中使用分隔符对字符串进行标记
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Python中使用分隔符对字符串进行标记EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Python中使用分隔符对字符串进行标记
EN