entityMap|blocks|key|36jf6|text|Java虚拟机(JVM)使用固定的内存上限运行，您可以对其进行如下修改：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|59g68|java+-Xmx1024m+....|code-block|syntax|javascript|4qpli|例如，上面的选项(-Xmx...)将限制设置为1024+to。您可以根据需要进行修改(在您的机器、操作系统等的限制范围内)请注意，这与传统应用程序不同，传统应用程序会根据需要从操作系统分配越来越多的内存。|d7g1v|但是，更好的解决方案是修改您的应用程序，这样您就不需要加载|2url2|整体|79dbb|将文件一次性存入内存。这样就不需要调优JVM，也不会占用大量内存。^0|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|P|9|@]|A|@]|B|$]]|$2|C|4|D|6|E|8|Q|9|@]|A|@]|B|$F|G]]|$2|H|4|I|6|7|8|R|9|@]|A|@]|B|$]]|$2|J|4|K|6|7|8|S|9|@]|A|@]|B|$]]|$2|L|4|M|6|7|8|T|9|@]|A|@]|B|$]]|$2|N|4|O|6|7|8|U|9|@]|A|@]|B|$]]]]

The Java virtual machine (JVM) runs with a fixed upper memory limit, which you can modify thus:

<pre><code>java -Xmx1024m ....
</code></pre>

e.g. the above option (-Xmx...) sets the limit to 1024 megabytes. You can amend as necessary (within limits of your machine, OS etc.) Note that this is different from traditional applications which would allocate more and more memory from the OS upon demand. 

However a better solution is to rework your application such that you don't need to load the whole file into memory at one go. That way you don't have to tune your JVM, and you don't impose a huge memory footprint.

entityMap|blocks|key|4aisb|text|你不能在内存中读取10+in的文本文件。您必须先读取X+MB，对其执行某些操作，然后读取下一个X+MB。|type|unstyled|depth|inlineStyleRanges|entityRanges|data^0^^$0|$]|1|@$2|3|4|5|6|7|8|C|9|@]|A|@]|B|$]]]]

You can't read 10GB Textfile in memory. You have to read X MB first, do something with it and than read the next X MB.

entityMap|blocks|key|95dft|text|ByteArrayOutputStream写入内存中的缓冲区。如果这确实是您希望它工作的方式，那么您必须根据输入的最大可能大小来调整JVM堆的大小。此外，如果可能，您甚至可以在开始处理之前检查输入大小，以节省时间和资源。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|8nt83|另一种方法是流解决方案，其中在运行时使用的内存量是已知的(可能是可配置的，但在程序启动之前仍然是已知的)，但它是否可行完全取决于应用程序的域(因为您不能再使用内存中的缓冲区)，如果您不能/不想更改它，还可能取决于其余代码的体系结构。^0|0^^$0|$]|1|@$2|3|4|5|6|7|8|E|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|F|9|@]|A|@]|B|$]]]]

<code>ByteArrayOutputStream</code> writes to an in-memory buffer. If this is really how you want it to work, then you have to size the JVM heap after the maximum possible size of the input. Also, if possible, you may check the input size before even start processing to save time and resources.

The alternative approach is a streaming solution, where the amount of memory used at runtime is known (maybe configurable but still known before the program starts), but if it's feasible or not depends entirely on you application's domain (because you can't use an in-memory buffer anymore) and maybe the architecture of the rest of your code if you can't/don't want to change it.

entityMap|blocks|key|1uikl|text|尝试使用较大的缓冲区读取大小可能为10MB，然后检查。|type|unstyled|depth|inlineStyleRanges|entityRanges|data^0^^$0|$]|1|@$2|3|4|5|6|7|8|C|9|@]|A|@]|B|$]]]]

Try using a large buffer read size may be 10 mb and then check.

entityMap|blocks|key|ef08h|text|这个问题是你正在做的事情所固有的。无论何时何地，将整个文件读入内存都不是一个好主意。除非你有一些相当惊人的硬件，否则你真的无法用当前的技术将一个10+be的文件读入内存。找到一种逐行、逐记录、逐块处理它们的方法……|type|unstyled|depth|inlineStyleRanges|entityRanges|data^0^^$0|$]|1|@$2|3|4|5|6|7|8|C|9|@]|A|@]|B|$]]]]

The problem is inherent in what you're doing. Reading entire files into memory is always and everywhere a bad idea. You're really not going to be able to read a 10GB file into memory with current technology unless you have some pretty startling hardware. Find a way to process them line by line, record by record, chunk by chunk, ...

entityMap|blocks|key|36uk6|text|尝试使用|type|unstyled|depth|inlineStyleRanges|entityRanges|data|66ta0|java.nio.MappedByteBuffer|5veqn|..。|erd2e|http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html|118cf|您可以将文件的内容映射到内存，而无需手动复制。高级操作系统提供了内存映射，Java提供了API来利用这一特性。|barcj|如果我的理解是正确的，那么内存映射不会将文件的全部内容加载到内存中(意思是“根据需要加载和卸载部分内容”)，所以我猜一个10+up的文件不会占用您的内存。^0|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|M|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|N|9|@]|A|@]|B|$]]|$2|E|4|F|6|7|8|O|9|@]|A|@]|B|$]]|$2|G|4|H|6|7|8|P|9|@]|A|@]|B|$]]|$2|I|4|J|6|7|8|Q|9|@]|A|@]|B|$]]|$2|K|4|L|6|7|8|R|9|@]|A|@]|B|$]]]]

Try to use java.nio.MappedByteBuffer.

<a href="http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html">http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html</a>

You can map a file's content onto memory without copying it manually. High-level Operating Systems offer memory-mapping and Java has API to utilize the feature.

If my understanding is correct, memory-mapping does not load a file's entire content onto memory (meaning "loaded and unloaded partially as necessary"), so I guess a 10GB file won't eat up your memory.

entityMap|blocks|key|2lm58|text|是否必须获得完整的ByteArray()输出流？|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1l6uo|byte[]+data+=+bArrStream.toByteArray();|code-block|syntax|javascript|ca8v7|最好的方法是逐行读取&逐行写入。您可以使用|77esv|BufferedReader+或者Scanner读取大文件，如下所示。|offset|length|style|CODE|96hmh|import+java.io.*;
import+java.util.*;

public+class+FileReadExample+{
++public+static+void+main(String+args[])+throws+FileNotFoundException+{
++++File+fileObj+=+new+File(args[0]);

++++long+t1+=+System.currentTimeMillis();
++++try+{
++++++++//+BufferedReader+object+for+reading+the+file
++++++++BufferedReader+br+=+new+BufferedReader(new+FileReader(fileObj));+
++++++++//+Reading+each+line+of+file+using+BufferedReader+class
++++++++String+str;
++++++++while+(+(str+=+br.readLine())+!=+null)+{
++++++++++++System.out.println(str);
++++++++}
++++}catch(Exception+err){
++++++++err.printStackTrace();
++++}
++++long+t2+=+System.currentTimeMillis();
++++System.out.println("Time+taken+for+BufferedReader:"%2B(t2-t1));

++++t1+=+System.currentTimeMillis();
++++try+(
++++++++//+Scanner+object+for+reading+the+file
++++++++Scanner+scnr+=+new+Scanner(fileObj);)+{
++++++++//+Reading+each+line+of+file+using+Scanner+class
++++++++while+(scnr.hasNextLine())+{
++++++++++++String+strLine+=+scnr.nextLine();
++++++++++++//+print+data+on+console
++++++++++++System.out.println(strLine);
++++++++}
++++}
++++t2+=+System.currentTimeMillis();
++++System.out.println("Time+taken+for+scanner:"%2B(t2-t1));

++}
}|8rhf6|你可以替换System.out+使用您的ByteArrayOutputStream|ag1n|在上面的例子中。|g25f|请看下面的文章了解更多细节：|7n5bi|读取大文件|f2icg|看看相关的SE问题：|4pr5p|扫描仪与BufferedReader^0|0|0|0|0|E|H|7|0|0|5|A|K|L|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|13|9|@]|A|@]|B|$]]|$2|C|4|D|6|E|8|14|9|@]|A|@]|B|$F|G]]|$2|H|4|I|6|7|8|15|9|@]|A|@]|B|$]]|$2|J|4|K|6|7|8|16|9|@$L|17|M|18|N|O]|$L|19|M|1A|N|O]]|A|@]|B|$]]|$2|P|4|Q|6|E|8|1B|9|@]|A|@]|B|$F|G]]|$2|R|4|S|6|7|8|1C|9|@$L|1D|M|1E|N|O]|$L|1F|M|1G|N|O]]|A|@]|B|$]]|$2|T|4|U|6|7|8|1H|9|@]|A|@]|B|$]]|$2|V|4|W|6|7|8|1I|9|@]|A|@]|B|$]]|$2|X|4|Y|6|7|8|1J|9|@]|A|@]|B|$]]|$2|Z|4|10|6|7|8|1K|9|@]|A|@]|B|$]]|$2|11|4|12|6|7|8|1L|9|@]|A|@]|B|$]]]]

Is it mandatory to get entire <code>ByteArray()</code> of output stream?

<pre><code>byte[] data = bArrStream.toByteArray();
</code></pre>

Best approach is read line by line &amp; write it line by line. You can use <code>BufferedReader</code> or <code>Scanner</code> to read large files as below.

<pre><code>import java.io.*;
import java.util.*;

public class FileReadExample {
 public static void main(String args[]) throws FileNotFoundException {
 File fileObj = new File(args[0]);

 long t1 = System.currentTimeMillis();
 try {
 // BufferedReader object for reading the file
 BufferedReader br = new BufferedReader(new FileReader(fileObj)); 
 // Reading each line of file using BufferedReader class
 String str;
 while ( (str = br.readLine()) != null) {
 System.out.println(str);
 }
 }catch(Exception err){
 err.printStackTrace();
 }
 long t2 = System.currentTimeMillis();
 System.out.println("Time taken for BufferedReader:"+(t2-t1));

 t1 = System.currentTimeMillis();
 try (
 // Scanner object for reading the file
 Scanner scnr = new Scanner(fileObj);) {
 // Reading each line of file using Scanner class
 while (scnr.hasNextLine()) {
 String strLine = scnr.nextLine();
 // print data on console
 System.out.println(strLine);
 }
 }
 t2 = System.currentTimeMillis();
 System.out.println("Time taken for scanner:"+(t2-t1));

 }
}
</code></pre>

You can replace <code>System.out</code> with your <code>ByteArrayOutputStream</code> in above example.

Please have a look at below article for more details: <a href="http://www.devmanuals.com/tutorials/java/corejava/files/java-read-large-file-efficiently.html" rel="nofollow noreferrer">Read Large File</a>

Have a look at related SE question:

<a href="https://stackoverflow.com/questions/2231369/scanner-vs-bufferedreader">Scanner vs. BufferedReader</a>

entityMap|blocks|key|15ie8|text|即使您可以增加JVM内存限制，但这也是不必要的，而且分配10+it这样的巨大内存来处理文件听起来有些夸张，而且会占用大量资源。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|972n|当前您使用的是一个"ByteArrayOutputStream“，它保留一个内部存储器来保存数据。代码中的这一行将最后读取的2KB文件块附加到此缓冲区的末尾：|9anhs|bArrStream.write(localbuffer,+0,+i);|code-block|syntax|javascript|43r6v|bArrStream不断增长，最终会耗尽内存。|2oji3|相反，你应该重新组织你的算法，并以流的方式处理文件：|2ccip|InputStream+inFileReader+=+channelSFtp.get(path);+//+file+reading+from+ssh.
byte[]+localbuffer+=+new+byte[2048];

int+i+=+0;
while+(-1+!=+(i+=+inFileReader.read(buffer)))+{
++++//Deal+with+the+current+read+2KB+file+chunk+here
}

inFileReader.close();^0|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|P|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|Q|9|@]|A|@]|B|$]]|$2|E|4|F|6|G|8|R|9|@]|A|@]|B|$H|I]]|$2|J|4|K|6|7|8|S|9|@]|A|@]|B|$]]|$2|L|4|M|6|7|8|T|9|@]|A|@]|B|$]]|$2|N|4|O|6|G|8|U|9|@]|A|@]|B|$H|I]]]]

Even though you can increase the JVM memory limit, it is needless and allocating a huge memory like 10GB to process a file sounds overkill and resource intensive. 

Currently you are using a "ByteArrayOutputStream" which keeps an internal memory to keep the data. This line in your code keeps appending the last read 2KB file chunk to the end of this buffer:

<pre><code>bArrStream.write(localbuffer, 0, i);
</code></pre>

bArrStream keeps growing and eventually you run out of memory. 

Instead you should reorganize your algorithm and process the file in a streaming way:

<pre><code>InputStream inFileReader = channelSFtp.get(path); // file reading from ssh.
byte[] localbuffer = new byte[2048];

int i = 0;
while (-1 != (i = inFileReader.read(buffer))) {
 //Deal with the current read 2KB file chunk here
}

inFileReader.close();
</code></pre>

entityMap|blocks|key|bfg5f|text|您好，我假设您正在读取大的txt文件，并且数据是逐行设置的，请使用逐行读取的方法。据我所知，你可以阅读高达6+6GB可能更多。我强烈建议您尝试这种方法。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|aktoe|DATA1+DATA2+..。|8q18u|//+Open+the+file
+FileInputStream+fstream+=+new+FileInputStream("textfile.txt");
+BufferedReader+br+=+new+BufferedReader(new+InputStreamReader(fstream));

++String+strLine;

+//Read+File+Line+By+Line
+while+((strLine+=+br.readLine())+!=+null)+++{
++//+Print+the+content+on+the+console
++System.out.println+(strLine);
+}

+//Close+the+input+stream
+br.close();|code-block|syntax|javascript|d16ia|代码片段的引用^0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|L|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|M|9|@]|A|@]|B|$]]|$2|E|4|F|6|G|8|N|9|@]|A|@]|B|$H|I]]|$2|J|4|K|6|7|8|O|9|@]|A|@]|B|$]]]]

Hi I am assuming that you are reading large txt file and the data is set line by line , use line by line reading approach. As I know you can read up to 6GB may be more. I strongly advice you to try this approach.

DATA1
DATA2
...

<pre><code>// Open the file
 FileInputStream fstream = new FileInputStream("textfile.txt");
 BufferedReader br = new BufferedReader(new InputStreamReader(fstream));

 String strLine;

 //Read File Line By Line
 while ((strLine = br.readLine()) != null) {
 // Print the content on the console
 System.out.println (strLine);
 }

 //Close the input stream
 br.close();
</code></pre>

<a href="https://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java">Refrence for the code fragment</a>

entityMap|blocks|key|1bke3|text|以迭代方式逐行读取该文件。这将显著减少内存消耗。或者，您可以使用|type|unstyled|depth|inlineStyleRanges|entityRanges|data|16oo2|FileUtils.lineIterator(theFile，"UTF-8")；|c107t|c86vc|由Apache+Commons+IO提供。|8i77u|FileInputStream+inputStream+=+null;
Scanner+sc+=+null;
try+{
inputStream+=+new+FileInputStream(path);
sc+=+new+Scanner(inputStream,+"UTF-8");
while+(sc.hasNextLine())+{
++++String+line+=+sc.nextLine();
++++//+System.out.println(line);
}
//+note+that+Scanner+suppresses+exceptions
if+(sc.ioException()+!=+null)+{
++++throw+sc.ioException();
}
}+finally+{
if+(inputStream+!=+null)+{
++++inputStream.close();
}
if+(sc+!=+null)+{
++++sc.close();
}|code-block|syntax|javascript|2k1fh|}^0|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|O|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|P|9|@]|A|@]|B|$]]|$2|E|4|-4|6|7|8|Q|9|@]|A|@]|B|$]]|$2|F|4|G|6|7|8|R|9|@]|A|@]|B|$]]|$2|H|4|I|6|J|8|S|9|@]|A|@]|B|$K|L]]|$2|M|4|N|6|7|8|T|9|@]|A|@]|B|$]]]]

Read the file iteratively linewise. This would significantly reduce memory consumption. Alternately you may use 

<blockquote>
 FileUtils.lineIterator(theFile, "UTF-8");
</blockquote>

provided by Apache Commons IO.

<pre><code>FileInputStream inputStream = null;
Scanner sc = null;
try {
inputStream = new FileInputStream(path);
sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
 String line = sc.nextLine();
 // System.out.println(line);
}
// note that Scanner suppresses exceptions
if (sc.ioException() != null) {
 throw sc.ioException();
}
} finally {
if (inputStream != null) {
 inputStream.close();
}
if (sc != null) {
 sc.close();
}
</code></pre>

}

entityMap|blocks|key|coclv|text|使用命令行选项-Xmx运行Java，该选项设置堆的最大大小。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|90s4v|有关详细信息，请参阅此处。^0|0^^$0|$]|1|@$2|3|4|5|6|7|8|E|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|F|9|@]|A|@]|B|$]]]]

Run Java with the command-line option -Xmx, which sets the maximum size of the heap.

<a href="http://www.oracle.com/splash/rpls/embargoed.html#nonstandard" rel="noreferrer">See here for details..</a>

entityMap|blocks|key|a1umj|text|您应该增加堆大小，如以下答案所述：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1lke7|增加Java中的堆大小|amfg0|但请记住，Java运行时和您的代码也会占用一些空间，因此请在所需的最大值上添加一些缓冲区。^0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|G|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|H|9|@]|A|@]|B|$]]|$2|E|4|F|6|7|8|I|9|@]|A|@]|B|$]]]]

You should increase heap size as stated in the following answer:

<a href="https://stackoverflow.com/questions/1565388/increase-heap-size-in-java">Increase heap size in Java</a>

But remember that the Java runtime and you code take some space as well so add some buffer to the desired maximum.

entityMap|blocks|key|n2d8|text|简而言之，|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2gs4u|不采取任何措施，您可以将电流限制提高1.5倍。这意味着，如果您能够处理800MB，则可以处理1200MB。这也意味着如果通过一些技巧java+-Xm+....|c4cm6|您可以移动到当前代码可以处理7+7GB的点，您的问题就解决了，因为1.5倍将使您达到10.5+7GB，假设您的系统上有可用的空间，并且JVM可以获得该空间。|cuqkh|答案很长：|5rb2p|这个错误是不言而喻的。您达到了配置的实际内存限制。关于JVM的限制有很多猜测，我对此了解不多，因为我找不到任何官方信息。但是，您会受到诸如可用交换空间、内核地址空间使用、内存碎片等约束的限制。|1ddra|现在发生的情况是ByteArrayOutputStream|4uig6|如果不提供任何大小(这就是您的情况)，则使用大小为32的默认缓冲区创建对象。每当您调用write方法，则会启动一个内部机制。The+The+The|25kgc|openjdk实现版本7u40-b43|7r6b7|这似乎与您的错误的输出完全匹配，使用内部方法ensureCapacity|cujre|检查缓冲区是否有足够的空间来放置要写入的字节。如果没有足够的空间，则使用另一个内部方法grow以增大缓冲区的大小。方法grow|308ea|定义适当的大小并调用该方法copyOf|6idh9|来自班级Arrays|59vvn|来完成这项工作。缓冲区的适当大小是当前大小和保存所有内容(当前内容和要写入的新内容)所需的大小之间的最大值。方法copyOf|6gk71|2vo5f|（点击链接)为新缓冲区分配空间，将旧缓冲区的内容复制到新缓冲区并将其返回到|cu5vt|grow..。|1bd4n|您的问题发生在为新缓冲区分配空间时，write|78n9t|，则会出现可用内存耗尽的情况：|9kf0l|java.lang.OutOfMemoryError:+Java+heap+space..。|t943|如果我们看一下细节，你会读到2048年的大块。所以|75qer|第一次写入时，缓冲区的大小将从32增加到2048|9lc43|您的第二个调用将加倍到2|eahv6|*|skcn|2048|97qi|您的第三次调用将使其达到2%5E2|2q10t|cha6t|2048，你必须在需要分配之前再写两次。|4benu|然后是2%5E3|a1bbn|albcv|2048，在再次分配之前，您将有时间进行4个更多的写入。|1k56s|在某些情况下，您的缓冲区大小将为2%5E18|fvpaj|2pei8|2048，等于2%5E19|ebkeh|8ogec|1024或2%5E9|ff81g|9e6ib|2%5E20+(512+MB)|85aqh|然后是2%5E19|1b0q1|6t1v2|2048，即1024+MB或1+GB|8jdf8|在你的描述中不清楚的是，你可以以某种方式读到800MB，但不能超过800MB。你得跟我解释一下。|br76k|我希望你的限制恰好是2的幂(如果我们使用10个单位的幂，那就接近了)。在这一点上，我希望你马上就会遇到问题:+256MB，512MB，1+2GB，2+2GB，等等。|1npkr|当达到该限制时，并不意味着内存不足，而只是意味着不可能再分配两倍于已有缓冲区大小的缓冲区。这一观察为您的工作提供了改进的空间:找到您可以分配的最大缓冲区大小，并通过调用适当的构造函数预先预留它|e80pi|ByteArrayOutputStream+bArrStream+=+new+ByteArrayOutputStream(myMaxSize);|code-block|syntax|javascript|9vv0h|它的优点是减少了幕后内存分配的开销，让您满意。通过这样做，您将能够达到您现在拥有的1.5限制。这很简单，因为上一次增加缓冲区时，它从当前大小的一半增加到当前大小，并且在某些时候，您在内存中同时拥有当前缓冲区和旧缓冲区。但是你将不能超过你现在所拥有的3倍的限制。解释是完全相同的。|fl2l1|也就是说，我没有任何神奇的建议来解决这个问题，除了通过给定大小的块来处理数据之外，一次处理一个块。另一个好的方法是使用Takahiko+Kawasaki的建议，并使用MappedByteBuffer|fm85q|..。请记住，在任何情况下，您都需要至少10+GB的物理内存或交换内存才能加载10+GB的文件。|33mcp|请参见^0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|2U|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|2V|9|@]|A|@]|B|$]]|$2|E|4|F|6|7|8|2W|9|@]|A|@]|B|$]]|$2|G|4|H|6|7|8|2X|9|@]|A|@]|B|$]]|$2|I|4|J|6|7|8|2Y|9|@]|A|@]|B|$]]|$2|K|4|L|6|7|8|2Z|9|@]|A|@]|B|$]]|$2|M|4|N|6|7|8|30|9|@]|A|@]|B|$]]|$2|O|4|P|6|7|8|31|9|@]|A|@]|B|$]]|$2|Q|4|R|6|7|8|32|9|@]|A|@]|B|$]]|$2|S|4|T|6|7|8|33|9|@]|A|@]|B|$]]|$2|U|4|V|6|7|8|34|9|@]|A|@]|B|$]]|$2|W|4|X|6|7|8|35|9|@]|A|@]|B|$]]|$2|Y|4|Z|6|7|8|36|9|@]|A|@]|B|$]]|$2|10|4|X|6|7|8|37|9|@]|A|@]|B|$]]|$2|11|4|12|6|7|8|38|9|@]|A|@]|B|$]]|$2|13|4|14|6|7|8|39|9|@]|A|@]|B|$]]|$2|15|4|16|6|7|8|3A|9|@]|A|@]|B|$]]|$2|17|4|18|6|7|8|3B|9|@]|A|@]|B|$]]|$2|19|4|1A|6|7|8|3C|9|@]|A|@]|B|$]]|$2|1B|4|1C|6|7|8|3D|9|@]|A|@]|B|$]]|$2|1D|4|1E|6|7|8|3E|9|@]|A|@]|B|$]]|$2|1F|4|1G|6|7|8|3F|9|@]|A|@]|B|$]]|$2|1H|4|1I|6|7|8|3G|9|@]|A|@]|B|$]]|$2|1J|4|1K|6|7|8|3H|9|@]|A|@]|B|$]]|$2|1L|4|1M|6|7|8|3I|9|@]|A|@]|B|$]]|$2|1N|4|1I|6|7|8|3J|9|@]|A|@]|B|$]]|$2|1O|4|1P|6|7|8|3K|9|@]|A|@]|B|$]]|$2|1Q|4|1R|6|7|8|3L|9|@]|A|@]|B|$]]|$2|1S|4|1I|6|7|8|3M|9|@]|A|@]|B|$]]|$2|1T|4|1U|6|7|8|3N|9|@]|A|@]|B|$]]|$2|1V|4|1W|6|7|8|3O|9|@]|A|@]|B|$]]|$2|1X|4|1I|6|7|8|3P|9|@]|A|@]|B|$]]|$2|1Y|4|1Z|6|7|8|3Q|9|@]|A|@]|B|$]]|$2|20|4|1I|6|7|8|3R|9|@]|A|@]|B|$]]|$2|21|4|22|6|7|8|3S|9|@]|A|@]|B|$]]|$2|23|4|1I|6|7|8|3T|9|@]|A|@]|B|$]]|$2|24|4|25|6|7|8|3U|9|@]|A|@]|B|$]]|$2|26|4|27|6|7|8|3V|9|@]|A|@]|B|$]]|$2|28|4|1I|6|7|8|3W|9|@]|A|@]|B|$]]|$2|29|4|2A|6|7|8|3X|9|@]|A|@]|B|$]]|$2|2B|4|2C|6|7|8|3Y|9|@]|A|@]|B|$]]|$2|2D|4|2E|6|7|8|3Z|9|@]|A|@]|B|$]]|$2|2F|4|2G|6|7|8|40|9|@]|A|@]|B|$]]|$2|2H|4|2I|6|2J|8|41|9|@]|A|@]|B|$2K|2L]]|$2|2M|4|2N|6|7|8|42|9|@]|A|@]|B|$]]|$2|2O|4|2P|6|7|8|43|9|@]|A|@]|B|$]]|$2|2Q|4|2R|6|7|8|44|9|@]|A|@]|B|$]]|$2|2S|4|2T|6|7|8|45|9|@]|A|@]|B|$]]]]

Short answer,

without doing anything, you can push the current limit by a factor of 1.5. It means that, if you are able to process 800MB, you can process 1200 MB. It also means that if by some trick with <code>java -Xm ....</code> you can move to a point where your current code can process 7GB, your problem is solved, because the 1.5 factor will take you to 10.5GB, assuming you have that space available on your system and that JVM can get it.

Long answer:

The error is pretty self-descriptive. You hit the practical memory limit on your configuration. There is a lot of speculating about the limit that you can have with JVM, I do not know enough about that, since I can not find any official information. However, you will somehow be limited by constraints like the available swap, the kernel address space usage, the memory fragmentation, etc.

What is happening now is that <code>ByteArrayOutputStream</code> objects are created with a default buffer of size 32 if you do not supply any size (this is your case). Whenever you call the <code>write</code> method on the object, there is an internal machinery that is started. The <a href="http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/java/io/ByteArrayOutputStream.java?av=f" rel="nofollow">openjdk implementation release 7u40-b43</a> that seems to match perfectly with the output of your error, uses an internal method <code>ensureCapacity</code> to check that the buffer has enough room to put the bytes you want to write. If there is not enough room, another internal method <code>grow</code> is called to grow the size of the buffer. The method <code>grow</code> defines the appropriate size and calls the method <code>copyOf</code> from the class <code>Arrays</code> to do the job.
The appropriate size of the buffer is the maximum between the current size and the size riquired to hold all the content (the present content and the new content to be write).
The method <code>copyOf</code> from the class <code>Arrays</code> (<a href="http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/java/util/Arrays.java#Arrays.copyOf%28byte[]%2Cint%29" rel="nofollow">follow the link</a>) allocates the space for the new buffer, copy the content of the old buffer to the new one and return it to <code>grow</code>.

Your problem occurs at the allocation of the space for the new buffer, After some <code>write</code>, you got to a point where the available memory is exhausted: <code>java.lang.OutOfMemoryError: Java heap space</code>.

If we look into details, you are reading by chunks of 2048. So 

<ul>
<li>your first write to the grows the size of the buffer from 32 to 2048</li>
<li>your second call will double it to 2*2048</li>
<li>your third call will take it to 2^2*2048, you have to time to write two more times before the need of allocating.</li>
<li>then 2^3*2048, you will have the time for 4 mores writes before allocating again.</li>
<li>at some point, your buffer will be of size 2^18*2048 which is 2^19*1024 or 2^9*2^20 (512 MB)</li>
<li>then 2^19*2048 which is 1024 MB or 1 GB</li>
</ul>

Something that is unclear in your description is that you can somehow read up to 800MB, but can no go beyond. You have to explain that to me.

I expect that your limit be exactly a power of 2 (or close if we use power of 10 units somewere). In that regard, I expect you to start having trouble immediatly above one of these: 256MB, 512 MB, 1GB, 2GB, etc.

When you hit that limit, it does not mean that you are out of memory, it simply means that it is not possible to allocate another buffer of twice the size of the buffer you already have. This observation opens room for improvement in your work: find the maximum size of buffer that you can allocate and reserve it upfront by calling the appropriate constructor

<pre><code>ByteArrayOutputStream bArrStream = new ByteArrayOutputStream(myMaxSize);
</code></pre>

It has the advantage of reducing the overhead background memory allocation that happens under the hood to keep you happy. By doing this, you will be able to go to 1.5 the limit you have right now. This is simply because the last time the buffer was increased, it went from half the current size to the current size, and at some point you had both the current buffer and the old one together in memory. But you will not be able to go beyond 3 times the limit you are having now. The explanation is exactly the same. 

That been said, I do not have any magic suggestion to solve the problem apart from process your data by chunks of given size, one chunk at a time. Another good approach will be to use the suggestion of Takahiko Kawasaki and use <code>MappedByteBuffer</code>. Keep in mind that in any case you will need at least 10 GB of physical memory or swap memory to be able to load a file of 10GB.

see

entityMap|blocks|key|ds9om|text|经过思考，我决定给出第二个答案。我考虑了第二个答案的优点和缺点，而且优点是值得去尝试的。所以就是这样。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2calf|大多数建议的考虑事项都忽略了一个给定的事实:数组的大小有一个内置限制(包括|9pkm0|ByteArrayOutputStream)，您可以在Java中使用。而这一限制是由最大的int|bhvgk|值为2%5E31+-1(略小于2Giga)。这意味着您最多只能读取2+GB+(-1字节)，并将其放入一个ByteArrayOutputStream..。如果VM需要更多的控制，则数组大小的限制实际上可能会更小。|9o01f|我的建议是使用ArrayList的byte[]|35h42|而不是单个byte[]|bbal8|保存文件的全部内容。也去掉了不必要的输入步骤ByteArrayOutputStream|3pasa|在把它放进决赛之前data|e0e26|数组。以下是基于您的原始代码的示例：|5boba|InputStream+inFileReader+=+channelSFtp.get(path);+//+file+reading+from+ssh.

//+good+habits+are+good,+define+a+buffer+size
final+int+BUF_SIZE+=+(int)(Math.pow(2,30));+//1GB,+let's+not+go+close+to+the+limit

byte[]+localbuffer+=+new+byte[BUF_SIZE];

int+i+=+0;
while+(-1+!=+(i+=+inFileReader.read(localbuffer)))+{
++++if(i|code-block|syntax|javascript|fji9n|Simply+running+your+program+should+work+fine+on+64+bits+system+with+enough+physical+memory+or+swap.+Now+if+you+want+to+speed+it+up+to+help+the+VM+size+correctly+the+heap+at+the+beginning,+run+with+the+options+-Xms+and+-Xmx.+For+example+if+you+want+a+heap+of+12GB+to+be+able+to+handle+10GB+file,+use+java+-Xms12288m+-Xmx12288m+YourApp|3hi79^0|0|0|0|0|0|0|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|10|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|11|9|@]|A|@]|B|$]]|$2|E|4|F|6|7|8|12|9|@]|A|@]|B|$]]|$2|G|4|H|6|7|8|13|9|@]|A|@]|B|$]]|$2|I|4|J|6|7|8|14|9|@]|A|@]|B|$]]|$2|K|4|L|6|7|8|15|9|@]|A|@]|B|$]]|$2|M|4|N|6|7|8|16|9|@]|A|@]|B|$]]|$2|O|4|P|6|7|8|17|9|@]|A|@]|B|$]]|$2|Q|4|R|6|7|8|18|9|@]|A|@]|B|$]]|$2|S|4|T|6|U|8|19|9|@]|A|@]|B|$V|W]]|$2|X|4|Y|6|7|8|1A|9|@]|A|@]|B|$]]|$2|Z|4|-4|6|7|8|1B|9|@]|A|@]|B|$]]]]

After thinking about it, I decided to put a second answer. I considered the advantages and disadvantages of putting this second answer, and the advantages are worth going for it. So here it is.

Most of the suggested considerations are forgetting a given fact: There is a builtin limit in the size of arrays (including <code>ByteArrayOutputStream</code>) that you can have in Java. And that limit is dictated by the bigest <code>int</code> value which is 2^31 - 1(little bit less than 2Giga). This means that you can only read a maximum of 2 GB (-1 byte) and put it in a single <code>ByteArrayOutputStream</code>. The limit might actually be smaller for array size if the VM wants more control.

My suggestion is to use an <code>ArrayList</code> of <code>byte[]</code> instead of a single <code>byte[]</code> holding the full content of the file. And also remove the non necessary step of putting in <code>ByteArrayOutputStream</code> before putting it in a final <code>data</code> array. Here is an example based on your original code:

<pre><code>InputStream inFileReader = channelSFtp.get(path); // file reading from ssh.

// good habits are good, define a buffer size
final int BUF_SIZE = (int)(Math.pow(2,30)); //1GB, let's not go close to the limit

byte[] localbuffer = new byte[BUF_SIZE];

int i = 0;
while (-1 != (i = inFileReader.read(localbuffer))) {
 if(i&lt;BUF_SIZE){
 data.add( Arrays.copyOf(localbuffer, i) )
 // No need to reallocate the reading buffer, we copied the data
 }else{
 data.add(localbuffer)
 // reallocate the reading buffer
 localbuffer = new byte[BUF_SIZE]
 }
}

inFileReader.close();
// Process your data, keep in mind that you have a list of buffers.
// So you need to loop over the list
</code></pre>

Simply running your program should work fine on 64 bits system with enough physical memory or swap. Now if you want to speed it up to help the VM size correctly the heap at the beginning, run with the options <code>-Xms</code> and <code>-Xmx</code>. For example if you want a heap of 12GB to be able to handle 10GB file, use <code>java -Xms12288m -Xmx12288m YourApp</code>

I'm new to Java and working on reading very large files, need some help to understand the problem and solve it. We have got some legacy code which have to be optimized to make it run properly.The file size can vary from 10mb to 10gb only. only trouble start when file starting beyond 800mb size.

<pre><code>InputStream inFileReader = channelSFtp.get(path); // file reading from ssh.
byte[] localbuffer = new byte[2048];
ByteArrayOutputStream bArrStream = new ByteArrayOutputStream();

int i = 0;
while (-1 != (i = inFileReader.read(buffer))) {
bArrStream.write(localbuffer, 0, i);
}

byte[] data = bArrStream.toByteArray();
inFileReader.close();
bos.close();
</code></pre>

We are getting the error 

<pre><code>java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:2271)
 at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
 at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
</code></pre>

Any help would be appreciated?

Java OutOfMemoryError in reading a large text file

内存映射

虚拟机

Java

 我刚接触Java，正在读取非常大的文件，需要一些帮助来理解问题并解决它。我们有一些遗留代码，必须进行优化，使其运行properly.The文件大小可以从10mb到10 it。只有当文件大小超过800mb时才会出现问题。 InputStream inFileReader = channelSFtp.get(path);...

问Java OutOfMemoryError在读取大型文本文件中的应用
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Java OutOfMemoryError在读取大型文本文件中的应用EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Java OutOfMemoryError在读取大型文本文件中的应用
EN