blocks|key|866827|text|如果您使用的是.NET+4.0，请尝试MemoryMappedFile，它是为此场景设计的类。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|866828|否则，您可以使用StreamReader.ReadLine。|style|CODE|866829|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile.aspx^0|J|G|0|0|8|L|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]]|C|$]]|$1|D|3|E|5|6|7|T|8|@$A|U|B|V|F|G]]|9|@]|C|$]]|$1|H|3|-4|5|6|7|W|8|@]|9|@]|C|$]]]|I|$J|$5|K|L|M|C|$N|O]]]]

If you are using .NET 4.0, try <a href="http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile.aspx" rel="noreferrer">MemoryMappedFile</a> which is a designed class for this scenario.

You can use <code>StreamReader.ReadLine</code> otherwise.

blocks|key|1081555|text|您可以使用LINQ|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1081556|int+result+=+File.ReadLines(filePath).Count(line+=>+line.StartsWith(word));|code-block|syntax|javascript|1081557|File.ReadLines返回一个IEnumerable，它懒惰地从文件中读取每一行，而不将整个文件加载到内存中。|1081558|Enumerable.Count计算以单词开头的行数。|1081559|如果从UI线程调用，请使用BackgroundWorker。|1081560|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/bb397926.aspx|1|http://msdn.microsoft.com/en-us/library/dd383503.aspx|2|http://msdn.microsoft.com/en-us/library/9eekhta0.aspx|3|http://msdn.microsoft.com/en-us/library/bb535181.aspx|4|http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx^0|5|4|0|0|0|0|E|1|I|B|2|0|0|G|3|0|D|G|4|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@$A|15|B|16|1|17]]|C|$]]|$1|D|3|E|5|F|7|18|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|19|8|@]|9|@$A|1A|B|1B|1|1C]|$A|1D|B|1E|1|1F]]|C|$]]|$1|K|3|L|5|6|7|1G|8|@]|9|@$A|1H|B|1I|1|1J]]|C|$]]|$1|M|3|N|5|6|7|1K|8|@]|9|@$A|1L|B|1M|1|1N]]|C|$]]|$1|O|3|-4|5|6|7|1O|8|@]|9|@]|C|$]]]|P|$Q|$5|R|S|T|C|$U|V]]|W|$5|R|S|T|C|$U|X]]|Y|$5|R|S|T|C|$U|Z]]|10|$5|R|S|T|C|$U|11]]|12|$5|R|S|T|C|$U|13]]]]

You can use <a href="http://msdn.microsoft.com/en-us/library/bb397926.aspx">LINQ</a>:

<pre><code>int result = File.ReadLines(filePath).Count(line =&gt; line.StartsWith(word));
</code></pre>

<a href="http://msdn.microsoft.com/en-us/library/dd383503.aspx">File.ReadLines</a> returns an <a href="http://msdn.microsoft.com/en-us/library/9eekhta0.aspx">IEnumerable&lt;String&gt;</a> that lazily reads each line from the file without loading the whole file into memory.

<a href="http://msdn.microsoft.com/en-us/library/bb535181.aspx">Enumerable.Count</a> counts the lines that start with the word.

If you are calling this from an UI thread, use a <a href="http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx">BackgroundWorker</a>.

blocks|key|866819|text|StreamReader.ReadLine应该可以很好地工作。让框架选择缓冲，除非您通过分析知道可以做得更好。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|866820|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/system.io.streamreader.readline.aspx^0|0|L|0|L|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@$9|Q|A|R|1|S]]|E|$]]|$1|F|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]]]

<a href="http://msdn.microsoft.com/en-us/library/system.io.streamreader.readline.aspx" rel="noreferrer"><code>StreamReader.ReadLine</code></a> should work fine. Let the framework choose the buffering, unless you know by profiling you can do better.

blocks|key|1081482|text|TextReader.ReadLine()|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1081483|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/system.io.textreader.readline.aspx^0|0|L|0|L|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@$9|Q|A|R|1|S]]|E|$]]|$1|F|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]]]

<a href="http://msdn.microsoft.com/en-us/library/system.io.textreader.readline.aspx" rel="noreferrer"><code>TextReader.ReadLine()</code></a>

blocks|key|866932|text|我在Agenty的生产服务器上遇到了同样的问题，在那里我们看到了大文件(有时是10-25+gb+(\t)制表符分隔的txt文件)。经过大量的测试和研究，我找到了使用for/foreach循环读取小块大文件的最佳方法，并使用File.ReadLines()设置偏移量和限制逻辑。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|866933|int+TotalRows+=+File.ReadLines(Path).Count();+//+Count+the+number+of+rows+in+file+with+lazy+load
int+Limit+=+100000;+//+100000+rows+per+batch
for+(int+Offset+=+0;+Offset+<+TotalRows;+Offset+%2B=+Limit)
{
++var+table+=+Path.FileToTable(heading:+true,+delimiter:+'\t',+offset+:+Offset,+limit:+Limit);

+//+Do+all+your+processing+here+and+with+limit+and+offset+and+save+to+drive+in+append+mode
+//+The+append+mode+will+write+the+output+in+same+file+for+each+processed+batch.

++table.TableToFile(@"C:\output.txt");
}|code-block|syntax|javascript|866934|请参阅我的Github库中的完整代码：https://github.com/Agenty/FileReader/|866935|完全公开-我为拥有这个库和网站的代理公司工作|866936|entityMap|0|LINK|mutability|MUTABLE|url|https://www.agenty.com/|1|https://github.com/Agenty/FileReader/^0|2|6|0|0|0|J|11|1|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@$A|X|B|Y|1|Z]]|C|$]]|$1|D|3|E|5|F|7|10|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|11|8|@]|9|@$A|12|B|13|1|14]]|C|$]]|$1|K|3|L|5|6|7|15|8|@]|9|@]|C|$]]|$1|M|3|-4|5|6|7|16|8|@]|9|@]|C|$]]]|N|$O|$5|P|Q|R|C|$S|T]]|U|$5|P|Q|R|C|$S|V]]]]

I was facing same problem in our production server at <a href="https://www.agenty.com/" rel="nofollow noreferrer">Agenty</a> where we see large files (sometimes 10-25 gb (\t) tab delimited txt files). And after lots of testing and research I found the best way to read large files in small chunks with for/foreach loop and setting offset and limit logic with File.ReadLines().

<pre><code>int TotalRows = File.ReadLines(Path).Count(); // Count the number of rows in file with lazy load
int Limit = 100000; // 100000 rows per batch
for (int Offset = 0; Offset &lt; TotalRows; Offset += Limit)
{
 var table = Path.FileToTable(heading: true, delimiter: '\t', offset : Offset, limit: Limit);

 // Do all your processing here and with limit and offset and save to drive in append mode
 // The append mode will write the output in same file for each processed batch.

 table.TableToFile(@"C:\output.txt");
}
</code></pre>

See the complete code in my Github library : <a href="https://github.com/Agenty/FileReader/" rel="nofollow noreferrer">https://github.com/Agenty/FileReader/</a>

Full Disclosure - I work for Agenty, the company who owned this library and website

blocks|key|1081502|text|我一次读取该文件10,000字节。然后，我会分析这10,000个字节，将它们切成行，并将它们提供给FormatData函数。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1081503|在多个线程上拆分读取和行分析的加分。|1081504|我肯定会使用StringBuilder来收集所有字符串，并且可能会构建一个字符串缓冲区，以便始终在内存中保存大约100个字符串。|offset|length|1081505|entityMap|0|LINK|mutability|MUTABLE|url|https://msdn.microsoft.com/en-us/library/system.text.stringbuilder%2528v=vs.110%2529.aspx^0|0|0|6|D|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|R|8|@]|9|@$F|S|G|T|1|U]]|A|$]]|$1|H|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

I'd read the file 10,000 bytes at a time. Then I'd analyse those 10,000 bytes and chop them into lines and feed them to the FormatData function.

Bonus points for splitting the reading and line analysation on multiple threads.

I'd definitely use a <a href="https://msdn.microsoft.com/en-us/library/system.text.stringbuilder%28v=vs.110%29.aspx" rel="nofollow">StringBuilder</a> to collect all strings and might build a string buffer to keep about 100 strings in memory all the time.

I have a 1 GB text file which I need to read line by line. What is the best and fastest way to do this?

<pre><code>private void ReadTxtFile()
{ 
 string filePath = string.Empty;
 filePath = openFileDialog1.FileName;
 if (string.IsNullOrEmpty(filePath))
 {
 using (StreamReader sr = new StreamReader(filePath))
 {
 String line;
 while ((line = sr.ReadLine()) != null)
 {
 FormatData(line); 
 }
 }
 }
}
</code></pre>

In <code>FormatData()</code> I check the starting word of line which must be matched with a word and based on that increment an integer variable.

<pre><code>void FormatData(string line)
{
 if (line.StartWith(word))
 {
 globalIntVariable++;
 }
}
</code></pre>

How to read a large (1 GB) txt file in .NET?

服务器

我有一个1 GB的文本文件，我需要逐行阅读。执行此操作的最佳和最快方法是什么？private void ReadTxtFile(){                string filePath = string.Empty;    filePath = openFileDialog1.FileName;    if (string.IsNullOrEmpty(filePath))    {   

问如何在.NET中读取一个大(1 GB)的txt文件？
EN

回答 6

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在.NET中读取一个大(1 GB)的txt文件？EN

回答 6

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在.NET中读取一个大(1 GB)的txt文件？
EN