blocks|key|1784414|text|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1784415|您不应该尝试通过读取end然后进行处理来强制它进入内存。|1784416|entityMap^0|0|0^^$0|@$1|2|3|-4|4|5|6|E|7|@]|8|@]|9|$]]|$1|A|3|B|4|5|6|F|7|@]|8|@]|9|$]]|$1|C|3|-4|4|5|6|G|7|@]|8|@]|9|$]]]|D|$]]

<a href="http://msdn.microsoft.com/en-us/library/db5x7c0d(v=VS.90).aspx" rel="noreferrer">Probably to read it line by line.</a> 

You should rather not try to force it into memory by reading to end and then processing.

blocks|key|1998191|text|StreamReader.ReadLine应该可以很好地工作。让框架选择缓冲，除非您通过分析知道可以做得更好。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1998192|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/system.io.streamreader.readline.aspx^0|0|L|0|L|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@$9|Q|A|R|1|S]]|E|$]]|$1|F|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]]]

<a href="http://msdn.microsoft.com/en-us/library/system.io.streamreader.readline.aspx" rel="noreferrer"><code>StreamReader.ReadLine</code></a> should work fine. Let the framework choose the buffering, unless you know by profiling you can do better.

blocks|key|1784446|text|TextReader.ReadLine()|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1784447|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/system.io.textreader.readline.aspx^0|0|L|0|L|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@$9|Q|A|R|1|S]]|E|$]]|$1|F|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]]]

<a href="http://msdn.microsoft.com/en-us/library/system.io.textreader.readline.aspx" rel="noreferrer"><code>TextReader.ReadLine()</code></a>

blocks|key|1784460|text|如果您使用的是.NET+4.0，请尝试MemoryMappedFile，它是为此场景设计的类。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1784461|否则，您可以使用StreamReader.ReadLine。|style|CODE|1784462|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile.aspx^0|J|G|0|0|8|L|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]]|C|$]]|$1|D|3|E|5|6|7|T|8|@$A|U|B|V|F|G]]|9|@]|C|$]]|$1|H|3|-4|5|6|7|W|8|@]|9|@]|C|$]]]|I|$J|$5|K|L|M|C|$N|O]]]]

If you are using .NET 4.0, try <a href="http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile.aspx" rel="noreferrer">MemoryMappedFile</a> which is a designed class for this scenario.

You can use <code>StreamReader.ReadLine</code> otherwise.

blocks|key|1784518|text|使用StreamReader可能是解决这个问题的方法，因为您不想一次在内存中存储整个文件。MemoryMappedFile更适合随机访问，而不是顺序读取(顺序读取的速度是顺序读取的十倍，内存映射的速度是随机访问的十倍)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1784519|您也可以尝试从FileOptions设置为SequentialScan+(请参阅)的文件流中创建streamreader，但我怀疑这是否会有很大的不同。|1784520|然而，有一些方法可以使您的示例更有效，因为您在与阅读相同的循环中进行格式化。您正在浪费时钟周期，所以如果您想要更高的性能，使用多线程异步解决方案会更好，在多线程异步解决方案中，一个线程读取数据，另一个线程在数据可用时格式化数据。检出可能符合您需求的BlockingColletion：|1784521|1784522|如果您想要尽可能快的性能，根据我的经验，唯一的方法是按顺序读入同样大的二进制数据块，并并行地将其反序列化为文本，但在这一点上，代码开始变得复杂。|1784523|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|M|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|N|8|@]|9|@]|A|$]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|P|8|@]|9|@]|A|$]]]|J|$]]

Using StreamReader is probably the way to since you don't want the whole file in memory at once. MemoryMappedFile is more for random access than sequential reading (it's ten times as fast for sequential reading and memory mapping is ten times as fast for random access).

You might also try creating your streamreader from a filestream with FileOptions set to SequentialScan (see <a href="http://msdn.microsoft.com/en-us/library/system.io.fileoptions.aspx" rel="noreferrer">FileOptions Enumeration</a>), but I doubt it will make much of a difference.

There are however ways to make your example more effective, since you do your formatting in the same loop as reading. You're wasting clockcycles, so if you want even more performance, it would be better with a multithreaded asynchronous solution where one thread reads data and another formats it as it becomes available. Checkout BlockingColletion that might fit your needs:

<a href="http://blogs.msdn.com/b/csharpfaq/archive/2010/08/12/blocking-collection-and-the-producer-consumer-problem.aspx" rel="noreferrer">Blocking Collection and the Producer-Consumer Problem</a>

If you want the fastest possible performance, in my experience the only way is to read in as large a chunk of binary data sequentially and deserialize it into text in parallel, but the code starts to get complicated at that point.

blocks|key|1784608|text|我一次读取该文件10,000字节。然后，我会分析这10,000个字节，将它们切成行，并将它们提供给FormatData函数。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1784609|在多个线程上拆分读取和行分析的加分。|1784610|我肯定会使用StringBuilder来收集所有字符串，并且可能会构建一个字符串缓冲区，以便始终在内存中保存大约100个字符串。|offset|length|1784611|entityMap|0|LINK|mutability|MUTABLE|url|https://msdn.microsoft.com/en-us/library/system.text.stringbuilder%2528v=vs.110%2529.aspx^0|0|0|6|D|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|R|8|@]|9|@$F|S|G|T|1|U]]|A|$]]|$1|H|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

I'd read the file 10,000 bytes at a time. Then I'd analyse those 10,000 bytes and chop them into lines and feed them to the FormatData function.

Bonus points for splitting the reading and line analysation on multiple threads.

I'd definitely use a <a href="https://msdn.microsoft.com/en-us/library/system.text.stringbuilder%28v=vs.110%29.aspx" rel="nofollow">StringBuilder</a> to collect all strings and might build a string buffer to keep about 100 strings in memory all the time.

blocks|key|1998357|text|您可以使用LINQ|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1998358|int+result+=+File.ReadLines(filePath).Count(line+=>+line.StartsWith(word));|code-block|syntax|javascript|1998359|File.ReadLines返回一个IEnumerable，它懒惰地从文件中读取每一行，而不将整个文件加载到内存中。|1998360|Enumerable.Count计算以单词开头的行数。|1998361|如果从UI线程调用，请使用BackgroundWorker。|1998362|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/library/bb397926.aspx|1|http://msdn.microsoft.com/en-us/library/dd383503.aspx|2|http://msdn.microsoft.com/en-us/library/9eekhta0.aspx|3|http://msdn.microsoft.com/en-us/library/bb535181.aspx|4|http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx^0|5|4|0|0|0|0|E|1|I|B|2|0|0|G|3|0|D|G|4|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@$A|15|B|16|1|17]]|C|$]]|$1|D|3|E|5|F|7|18|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|19|8|@]|9|@$A|1A|B|1B|1|1C]|$A|1D|B|1E|1|1F]]|C|$]]|$1|K|3|L|5|6|7|1G|8|@]|9|@$A|1H|B|1I|1|1J]]|C|$]]|$1|M|3|N|5|6|7|1K|8|@]|9|@$A|1L|B|1M|1|1N]]|C|$]]|$1|O|3|-4|5|6|7|1O|8|@]|9|@]|C|$]]]|P|$Q|$5|R|S|T|C|$U|V]]|W|$5|R|S|T|C|$U|X]]|Y|$5|R|S|T|C|$U|Z]]|10|$5|R|S|T|C|$U|11]]|12|$5|R|S|T|C|$U|13]]]]

You can use <a href="http://msdn.microsoft.com/en-us/library/bb397926.aspx">LINQ</a>:

<pre><code>int result = File.ReadLines(filePath).Count(line =&gt; line.StartsWith(word));
</code></pre>

<a href="http://msdn.microsoft.com/en-us/library/dd383503.aspx">File.ReadLines</a> returns an <a href="http://msdn.microsoft.com/en-us/library/9eekhta0.aspx">IEnumerable&lt;String&gt;</a> that lazily reads each line from the file without loading the whole file into memory.

<a href="http://msdn.microsoft.com/en-us/library/bb535181.aspx">Enumerable.Count</a> counts the lines that start with the word.

If you are calling this from an UI thread, use a <a href="http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx">BackgroundWorker</a>.

blocks|key|1784618|text|我在Agenty的生产服务器上遇到了同样的问题，在那里我们看到了大文件(有时是10-25+gb+(\t)制表符分隔的txt文件)。经过大量的测试和研究，我找到了使用for/foreach循环读取小块大文件的最佳方法，并使用File.ReadLines()设置偏移量和限制逻辑。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1784619|int+TotalRows+=+File.ReadLines(Path).Count();+//+Count+the+number+of+rows+in+file+with+lazy+load
int+Limit+=+100000;+//+100000+rows+per+batch
for+(int+Offset+=+0;+Offset+<+TotalRows;+Offset+%2B=+Limit)
{
++var+table+=+Path.FileToTable(heading:+true,+delimiter:+'\t',+offset+:+Offset,+limit:+Limit);

+//+Do+all+your+processing+here+and+with+limit+and+offset+and+save+to+drive+in+append+mode
+//+The+append+mode+will+write+the+output+in+same+file+for+each+processed+batch.

++table.TableToFile(@"C:\output.txt");
}|code-block|syntax|javascript|1784620|请参阅我的Github库中的完整代码：https://github.com/Agenty/FileReader/|1784621|完全公开-我为拥有这个库和网站的代理公司工作|1784622|entityMap|0|LINK|mutability|MUTABLE|url|https://www.agenty.com/|1|https://github.com/Agenty/FileReader/^0|2|6|0|0|0|J|11|1|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@$A|X|B|Y|1|Z]]|C|$]]|$1|D|3|E|5|F|7|10|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|11|8|@]|9|@$A|12|B|13|1|14]]|C|$]]|$1|K|3|L|5|6|7|15|8|@]|9|@]|C|$]]|$1|M|3|-4|5|6|7|16|8|@]|9|@]|C|$]]]|N|$O|$5|P|Q|R|C|$S|T]]|U|$5|P|Q|R|C|$S|V]]]]

I was facing same problem in our production server at <a href="https://www.agenty.com/" rel="nofollow noreferrer">Agenty</a> where we see large files (sometimes 10-25 gb (\t) tab delimited txt files). And after lots of testing and research I found the best way to read large files in small chunks with for/foreach loop and setting offset and limit logic with File.ReadLines().

<pre><code>int TotalRows = File.ReadLines(Path).Count(); // Count the number of rows in file with lazy load
int Limit = 100000; // 100000 rows per batch
for (int Offset = 0; Offset &lt; TotalRows; Offset += Limit)
{
 var table = Path.FileToTable(heading: true, delimiter: '\t', offset : Offset, limit: Limit);

 // Do all your processing here and with limit and offset and save to drive in append mode
 // The append mode will write the output in same file for each processed batch.

 table.TableToFile(@"C:\output.txt");
}
</code></pre>

See the complete code in my Github library : <a href="https://github.com/Agenty/FileReader/" rel="nofollow noreferrer">https://github.com/Agenty/FileReader/</a>

Full Disclosure - I work for Agenty, the company who owned this library and website

I have a 1 GB text file which I need to read line by line. What is the best and fastest way to do this?

<pre><code>private void ReadTxtFile()
{ 
 string filePath = string.Empty;
 filePath = openFileDialog1.FileName;
 if (string.IsNullOrEmpty(filePath))
 {
 using (StreamReader sr = new StreamReader(filePath))
 {
 String line;
 while ((line = sr.ReadLine()) != null)
 {
 FormatData(line); 
 }
 }
 }
}
</code></pre>

In <code>FormatData()</code> I check the starting word of line which must be matched with a word and based on that increment an integer variable.

<pre><code>void FormatData(string line)
{
 if (line.StartWith(word))
 {
 globalIntVariable++;
 }
}
</code></pre>

How to read a large (1 GB) txt file in .NET?

内存映射

服务器

我有一个1 GB的文本文件，我需要逐行阅读。执行此操作的最佳和最快方法是什么？private void ReadTxtFile(){                string filePath = string.Empty;    filePath = openFileDialog1.FileName;    if ...

问如何在.NET中读取一个大(1 GB)的txt文件？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在.NET中读取一个大(1 GB)的txt文件？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在.NET中读取一个大(1 GB)的txt文件？
EN