blocks|key|262826|text|首先，您可以尝试使用unix工具sort对数据进行排序(它处理非常大的文件非常好)。然后，具有相同第一和第二字段的行将处于连续块中，这样您就可以使用python脚本迭代它们，并以您喜欢的格式输出它们。下面给出了一个示例，用于不希望在第二列中没有第一、第二或第三列的任何数据。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|262827|#+merger.py
#+this+python+script+takes+in+a+file+with+sorted+lines+and+gives+your+output
import+sys

prevtag+=+None
data+=+['',+'',+'']
printed+=+True
for+line+in+sys.stdin:
++++fields+=+line.strip().split()
++++if+fields[0]+!=+prevtag:
++++++++if+not+printed:
++++++++++++print+prevtag,+'+'.join(data)
++++++++++++printed+=+True
++++++++prevtag+=+fields[0]
++++++++data+=+['',+'',+'']
++++if+fields[1]+==+'<first>':
++++++++data[0]+=+fields[2]
++++++++printed+=+False
++++if+fields[1]+==+'<second>':
++++++++data[1]+=+fields[2]
++++++++printed+=+False
++++if+fields[1]+==+'<third>':
++++++++data[2]+=+fields[2]
++++++++printed+=+False
if+not+printed:
++++print+prevtag,+'+'.join(data)|code-block|syntax|javascript|262828|现在，您可以将sort的输出输送到这个脚本中，以实现您想要的结果。|262829|sort+<inputfile>+%7C+python+merger.py|262830|输入：|262831|<abc>++<anything2>+<anything3>
<_901>++<first>+<something1>
<_905>++<second>+<something2>
<_910>++<anything>+<something3>
<_901>++<second>+<something4>
<_905>++<first>+<something6>
<_901>++<third>+<something5>
<_905>++<third>+<something7>|262832|输出：|262833|<_901>+<something1>+<something4>+<something5>
<_905>+<something6>+<something2>+<something7>|262834|entityMap^0|G|4|0|0|7|4|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@$9|Z|A|10|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|11|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|12|8|@$9|13|A|14|B|C]]|D|@]|E|$]]|$1|M|3|N|5|H|7|15|8|@]|D|@]|E|$I|J]]|$1|O|3|P|5|6|7|16|8|@]|D|@]|E|$]]|$1|Q|3|R|5|H|7|17|8|@]|D|@]|E|$I|J]]|$1|S|3|T|5|6|7|18|8|@]|D|@]|E|$]]|$1|U|3|V|5|H|7|19|8|@]|D|@]|E|$I|J]]|$1|W|3|-4|5|6|7|1A|8|@]|D|@]|E|$]]]|X|$]]

First of all, you can try to sort the data using unix tool <code>sort</code> (it handles very large files pretty nicely). Then lines with same first and second field will be in continuous block, so that you can iterate through them using a python script and output in your favorite format. An example is given below for the case where you don't want any data which do not have first, second or third in their second column.

<pre><code># merger.py
# this python script takes in a file with sorted lines and gives your output
import sys

prevtag = None
data = ['', '', '']
printed = True
for line in sys.stdin:
 fields = line.strip().split()
 if fields[0] != prevtag:
 if not printed:
 print prevtag, ' '.join(data)
 printed = True
 prevtag = fields[0]
 data = ['', '', '']
 if fields[1] == '&lt;first&gt;':
 data[0] = fields[2]
 printed = False
 if fields[1] == '&lt;second&gt;':
 data[1] = fields[2]
 printed = False
 if fields[1] == '&lt;third&gt;':
 data[2] = fields[2]
 printed = False
if not printed:
 print prevtag, ' '.join(data)
</code></pre>

Now you can pipe the output of <code>sort</code> into this script to achieve what you want.

<pre><code>sort &lt;inputfile&gt; | python merger.py
</code></pre>

Input:

<pre><code>&lt;abc&gt; &lt;anything2&gt; &lt;anything3&gt;
&lt;_901&gt; &lt;first&gt; &lt;something1&gt;
&lt;_905&gt; &lt;second&gt; &lt;something2&gt;
&lt;_910&gt; &lt;anything&gt; &lt;something3&gt;
&lt;_901&gt; &lt;second&gt; &lt;something4&gt;
&lt;_905&gt; &lt;first&gt; &lt;something6&gt;
&lt;_901&gt; &lt;third&gt; &lt;something5&gt;
&lt;_905&gt; &lt;third&gt; &lt;something7&gt;
</code></pre>

Output:

<pre><code>&lt;_901&gt; &lt;something1&gt; &lt;something4&gt; &lt;something5&gt;
&lt;_905&gt; &lt;something6&gt; &lt;something2&gt; &lt;something7&gt;
</code></pre>

blocks|key|261791|text|这可能会给你一些想法。我使用了大卫·比兹利(+David+)在系统程序员的生成器技巧上的演讲中的一些东西。由于您的文件很大，所以我将<something>值存储在单独的文件中。你也许可以把它们附加到清单上；我不确定。每次文件的打开、附加和关闭对我来说都是相当低效的，但我想不出任何其他的东西。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|261792|from+__future__+import+division,+print_function
from+StringIO+import+StringIO
import+re
import+os
import+glob

#for+generator+details+see+see+http://www.dabeaz.com/generators/

def+gen_grep_groups(pat,+lines):

++++patc+=+re.compile(pat)
++++for+line+in+lines:
++++++++match+=+patc.search(line)
++++++++if+match:
++++++++++++yield+match.groups()

def+logme(pairs,+ext='.special_list'):
++++for+name,+thing+in+pairs:
++++++++#there+must+be+a+better+way+than+opening+and+closing+each+time
++++++++f+=+open(name+%2B+ext,'a')
++++++++f.writelines(thing+%2B'\n')
++++++++f.close()


data+=+StringIO("""<abc>++<anything2>+<anything3>.
<_901>++<first>+<something1>.
<_905>++<second>+<something2>.
<_910>++<anything>+<something3>.
<_901>++<second>+<something4>.
<_905>++<first>+<something6>.
<_901>++<third>+<something5>.
<_905>++<third>+<something7>.""")

#for+a+file+use+data+=+open(filename)



pat+=+r'<(.%2B?)>\s%2B<(?:first%7Csecond%7Cthird)>\s%2B<(.%2B?)>'
ext+=+".special_list"
#patternc+=+re.compile(pattern)

for+f+in+glob.glob('*'%2B+ext):
++++os.remove(f)+#remove+any+existing+files

pairs+=+gen_grep_groups(pat,data)
logme(pairs,+ext)


for+filename+in+glob.glob('*'%2B+ext):
++++#do+what+you+want+in+here++++
++++name+=+os.path.splitext(filename)[0]
++++print(name)
++++f+=+open(filename,'r')++++
++++print(f.read())
++++f.close()|code-block|syntax|javascript|261793|entityMap|0|LINK|mutability|MUTABLE|url|http://www.dabeaz.com/generators/^0|1U|B|V|B|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]]|D|@$9|V|A|W|1|X]]|E|$]]|$1|F|3|G|5|H|7|Y|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Z|8|@]|D|@]|E|$]]]|L|$M|$5|N|O|P|E|$Q|R]]]]

This might give you some ideas. I've used some things from a David Beazely lecture on <a href="http://www.dabeaz.com/generators/" rel="nofollow">Generator Tricks for Systems Programmers</a>. As your file is large I'm storing the <code>&lt;something&gt;</code> values in separate files. You might be able to append them to a list/dict; I'm not sure. The opening, appending to and closing of files each time seems pretty inefficient to me but I can't think of anything else. 

<pre><code>from __future__ import division, print_function
from StringIO import StringIO
import re
import os
import glob

#for generator details see see http://www.dabeaz.com/generators/

def gen_grep_groups(pat, lines):

 patc = re.compile(pat)
 for line in lines:
 match = patc.search(line)
 if match:
 yield match.groups()

def logme(pairs, ext='.special_list'):
 for name, thing in pairs:
 #there must be a better way than opening and closing each time
 f = open(name + ext,'a')
 f.writelines(thing +'\n')
 f.close()


data = StringIO("""&lt;abc&gt; &lt;anything2&gt; &lt;anything3&gt;.
&lt;_901&gt; &lt;first&gt; &lt;something1&gt;.
&lt;_905&gt; &lt;second&gt; &lt;something2&gt;.
&lt;_910&gt; &lt;anything&gt; &lt;something3&gt;.
&lt;_901&gt; &lt;second&gt; &lt;something4&gt;.
&lt;_905&gt; &lt;first&gt; &lt;something6&gt;.
&lt;_901&gt; &lt;third&gt; &lt;something5&gt;.
&lt;_905&gt; &lt;third&gt; &lt;something7&gt;.""")

#for a file use data = open(filename)



pat = r'&lt;(.+?)&gt;\s+&lt;(?:first|second|third)&gt;\s+&lt;(.+?)&gt;'
ext = ".special_list"
#patternc = re.compile(pattern)

for f in glob.glob('*'+ ext):
 os.remove(f) #remove any existing files

pairs = gen_grep_groups(pat,data)
logme(pairs, ext)


for filename in glob.glob('*'+ ext):
 #do what you want in here 
 name = os.path.splitext(filename)[0]
 print(name)
 f = open(filename,'r') 
 print(f.read())
 f.close()
</code></pre>

blocks|key|2447961|text|对于这么大的文件，可以多次读取。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2447962|首先，读取这些行，并将键字段与数据所在的行号(或者更好的:文件位置)一起存储：|2447963|pos+=+{}
filepos+=+file.tell()
for+line+in+file:
++++sline+=+line.split()
++++pos.setdefault(sline[0],+[]).append(filepos)
++++filepos.file.tell()+#+for+the+next+read|code-block|syntax|javascript|2447964|现在你有了一个像{''：，<_901>：25，87，.}|2447965|现在，您可以在dict上迭代并执行以下操作|2447966|for+key,+fileposes+in+pos.iteritems():
++++if+len(fileposes)+==+1:
++++++++file.seek(fileposes[0])
++++++++print+file.readline()
++++else:
++++++++outlinedata+=+[]
++++++++for+filepos+in+fileposes:
++++++++++++file.seek(fileposes[0])
++++++++++++line+=+file.readline()
++++++++++++sline+=+line.split()
++++++++++++outlinedata.append(sline[2])
++++++++print+key,+"+".join(outlinedata)|2447967|(但我不确定tell/seek和文件的行文读取和缓冲是否会互相干扰……)|offset|length|style|CODE|2447968|entityMap^0|0|0|0|0|0|0|6|9|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|X|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|Y|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|Z|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|10|8|@]|9|@]|A|$]]|$1|M|3|N|5|F|7|11|8|@]|9|@]|A|$G|H]]|$1|O|3|P|5|6|7|12|8|@$Q|13|R|14|S|T]]|9|@]|A|$]]|$1|U|3|-4|5|6|7|15|8|@]|9|@]|A|$]]]|V|$]]

For a file this large, it might be appropriate to read in in multiple passes.

First, read the lines and store the key field together with the line numbers (or maybe better: file positions) where the data live:

<pre><code>pos = {}
filepos = file.tell()
for line in file:
 sline = line.split()
 pos.setdefault(sline[0], []).append(filepos)
 filepos.file.tell() # for the next read
</code></pre>

Now you have a dict like, say, {'': [0], &lt;_901>: [25, 87], ...}

Now you can iterate over the dict and do

<pre><code>for key, fileposes in pos.iteritems():
 if len(fileposes) == 1:
 file.seek(fileposes[0])
 print file.readline()
 else:
 outlinedata = []
 for filepos in fileposes:
 file.seek(fileposes[0])
 line = file.readline()
 sline = line.split()
 outlinedata.append(sline[2])
 print key, " ".join(outlinedata)
</code></pre>

(But I am not sure if <code>tell/seek</code> on one hand and linewise reading and buffering of the file won't interfere with each other...)

I have data in the following format:

<pre><code>&lt;abc&gt; &lt;anything2&gt; &lt;anything3&gt;.
&lt;_901&gt; &lt;first&gt; &lt;something1&gt;.
&lt;_905&gt; &lt;second&gt; &lt;something2&gt;.
&lt;_910&gt; &lt;anything&gt; &lt;something3&gt;.
&lt;_901&gt; &lt;second&gt; &lt;something4&gt;.
&lt;_905&gt; &lt;first&gt; &lt;something6&gt;.
&lt;_901&gt; &lt;third&gt; &lt;something5&gt;.
&lt;_905&gt; &lt;third&gt; &lt;something7&gt;.
</code></pre>

Now I want to group together all the information corresponding to the first column and (first, second, third) values of second column such that I get aggregated information in the following form:

<pre><code> &lt;abc&gt; &lt;anything2&gt; &lt;anything3&gt;.
 &lt;_901&gt; &lt;something1&gt; &lt;something4&gt; &lt;something5&gt;.
 &lt;_905&gt; &lt;something6&gt; &lt;something2&gt; &lt;something7&gt;.
 &lt;_910&gt; &lt;anything&gt; &lt;something3&gt;.
</code></pre>

I tried to achieve this using python dictionary. But since I have a file of 2 Tera bytes. My procedure is running out of memory and is very inefficient. Is there a faster way in python to achieve this. If yes, then can someone please illustrate with an example?

python how to efficiently group together terms

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我有以下格式的数据：<abc> <anything2> <anything3>.<_901> <first> <something1>.<_905> <second> <something2>.<_910> <anything> <something3>.<_901> <second> <something4...

问python如何有效地组合术语
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问python如何有效地组合术语EN