blocks|key|1555218|text|我建议您定义一些“块”对象，每个对象在一个数组中保存(比方说)1+1Gb，然后创建一个这些对象的数组。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1555219|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

I suggest you define some "block" objects, each of which holds (say) 1Gb in an array, then make an array of those.

blocks|key|1555289|text|Java数组使用整数作为索引。因此，最大数组大小为Integer.MAX_VALUE。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1555290|(不幸的是，我自己找不到任何关于这一点的证据，但已经有discussions的plenty在他们的forums上对此进行了说明。)|offset|length|1555291|我认为目前你能做的最好的解决方案是做一个二维数组，即：|1555292|byte[][]+data;|code-block|syntax|javascript|1555293|entityMap|0|LINK|mutability|MUTABLE|url|http://forums.sun.com/thread.jspa?threadID=5384991&start=15&tstart=0|1|http://forums.sun.com/thread.jspa?threadID=584445|2|http://forums.sun.com/thread.jspa?threadID=5384991&tstart=1^0|0|R|B|0|13|6|1|1D|6|2|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Z|8|@]|9|@$D|10|E|11|1|12]|$D|13|E|14|1|15]|$D|16|E|17|1|18]]|A|$]]|$1|F|3|G|5|6|7|19|8|@]|9|@]|A|$]]|$1|H|3|I|5|J|7|1A|8|@]|9|@]|A|$K|L]]|$1|M|3|-4|5|6|7|1B|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|T]]|U|$5|P|Q|R|A|$S|V]]|W|$5|P|Q|R|A|$S|X]]]]

Java arrays use integers for their indices. As a result, the maximum array size is Integer.MAX_VALUE.

(Unfortunately, I can't find any proof from Sun themselves about this, but there are <a href="http://forums.sun.com/thread.jspa?threadID=584445" rel="nofollow noreferrer">plenty</a> of <a href="http://forums.sun.com/thread.jspa?threadID=5384991&amp;start=15&amp;tstart=0" rel="nofollow noreferrer">discussions</a> on their <a href="http://forums.sun.com/thread.jspa?threadID=5384991&amp;tstart=1" rel="nofollow noreferrer">forums</a> about it already.)

I think the best solution you could do in the meantime would be to make a 2D array, i.e.:

<pre><code>byte[][] data;
</code></pre>

blocks|key|1555367|text|否，数组按int%25s编制索引(使用short%25s的某些JavaCard版本除外)。您将需要将其分成较小的数组，可能包装在一个类型中，以提供get(long)、set(long,byte)等。对于如此大的数据段，您可能希望使用java.nio映射文件。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1555368|entityMap^0|5|3|H|5|1X|9|27|E|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]|$9|M|A|N|B|C]|$9|O|A|P|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|Q|8|@]|D|@]|E|$]]]|G|$]]

No, arrays are indexed by <code>int</code>s (except some versions of JavaCard that use <code>short</code>s). You will need to slice it up into smaller arrays, probably wrapping in a type that gives you <code>get(long)</code>, <code>set(long,byte)</code>, etc. With sections of data that large, you might want to map the file use java.nio.

blocks|key|1555458|text|您可以考虑使用FileChannel和MappedByteBuffer来对文件进行内存映射。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1555459|FileChannel+fCh+=+new+RandomAccessFile(file,"rw").getChannel();
long+size+=+fCh.size();
ByteBuffer+map+=+fCh.map(FileChannel.MapMode.READ_WRITE,+0,+fileSize);|code-block|syntax|javascript|1555460|编辑：|1555461|好吧，我是个笨蛋它看起来ByteBuffer也只接受32位的索引，这很奇怪，因为FileChannel.map的大小参数是一个长的……但是，如果您决定将文件拆分成多个2+2Gb的块进行加载，我仍然建议使用内存映射IO，因为这样可以带来相当大的性能优势。你基本上把所有的IO责任都转移到了操作系统内核上。|1555462|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

You might consider using FileChannel and MappedByteBuffer to memory map the file,

<pre><code>FileChannel fCh = new RandomAccessFile(file,"rw").getChannel();
long size = fCh.size();
ByteBuffer map = fCh.map(FileChannel.MapMode.READ_WRITE, 0, fileSize);
</code></pre>

Edit:

Ok, I'm an idiot it looks like ByteBuffer only takes a 32-bit index as well which is odd since the size parameter to FileChannel.map is a long... But if you decide to break up the file into multiple 2Gb chunks for loading I'd still recommend memory mapped IO as there can be pretty large performance benefits. You're basically moving all IO responsibility to the OS kernel.

blocks|key|1555470|text|package+com.deans.rtl.util;

import+java.io.FileInputStream;
import+java.io.IOException;

/**
+*+
+*+@author+william.deans@gmail.com
+*
+*+Written+to+work+with+byte+arrays+requiring+address+space+larger+than+32+bits.+
+*+
+*/

public+class+ByteArray64+{

++++private+final+long+CHUNK_SIZE+=+1024*1024*1024;+//1GiB

++++long+size;
++++byte+[][]+data;

++++public+ByteArray64(+long+size+)+{
++++++++this.size+=+size;
++++++++if(+size+==+0+)+{
++++++++++++data+=+null;
++++++++}+else+{
++++++++++++int+chunks+=+(int)(size/CHUNK_SIZE);
++++++++++++int+remainder+=+(int)(size+-+((long)chunks)*CHUNK_SIZE);
++++++++++++data+=+new+byte[chunks%2B(remainder==0?0:1)][];
++++++++++++for(+int+idx=chunks;+--idx>=0;+)+{
++++++++++++++++data[idx]+=+new+byte[(int)CHUNK_SIZE];
++++++++++++}
++++++++++++if(+remainder+!=+0+)+{
++++++++++++++++data[chunks]+=+new+byte[remainder];
++++++++++++}
++++++++}
++++}
++++public+byte+get(+long+index+)+{
++++++++if(+index<0+%7C%7C+index>=size+)+{
++++++++++++throw+new+IndexOutOfBoundsException("Error+attempting+to+access+data+element+"%2Bindex%2B".++Array+is+"%2Bsize%2B"+elements+long.");
++++++++}
++++++++int+chunk+=+(int)(index/CHUNK_SIZE);
++++++++int+offset+=+(int)(index+-+(((long)chunk)*CHUNK_SIZE));
++++++++return+data[chunk][offset];
++++}
++++public+void+set(+long+index,+byte+b+)+{
++++++++if(+index<0+%7C%7C+index>=size+)+{
++++++++++++throw+new+IndexOutOfBoundsException("Error+attempting+to+access+data+element+"%2Bindex%2B".++Array+is+"%2Bsize%2B"+elements+long.");
++++++++}
++++++++int+chunk+=+(int)(index/CHUNK_SIZE);
++++++++int+offset+=+(int)(index+-+(((long)chunk)*CHUNK_SIZE));
++++++++data[chunk][offset]+=+b;
++++}
++++/**
+++++*+Simulates+a+single+read+which+fills+the+entire+array+via+several+smaller+reads.
+++++*+
+++++*+@param+fileInputStream
+++++*+@throws+IOException
+++++*/
++++public+void+read(+FileInputStream+fileInputStream+)+throws+IOException+{
++++++++if(+size+==+0+)+{
++++++++++++return;
++++++++}
++++++++for(+int+idx=0;+idx<data.length;+idx%2B%2B+)+{
++++++++++++if(+fileInputStream.read(+data[idx]+)+!=+data[idx].length+)+{
++++++++++++++++throw+new+IOException("short+read");
++++++++++++}
++++++++}
++++}
++++public+long+size()+{
++++++++return+size;
++++}
}
}|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1555471|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>package com.deans.rtl.util;

import java.io.FileInputStream;
import java.io.IOException;

/**
 * 
 * @author william.deans@gmail.com
 *
 * Written to work with byte arrays requiring address space larger than 32 bits. 
 * 
 */

public class ByteArray64 {

 private final long CHUNK_SIZE = 1024*1024*1024; //1GiB

 long size;
 byte [][] data;

 public ByteArray64( long size ) {
 this.size = size;
 if( size == 0 ) {
 data = null;
 } else {
 int chunks = (int)(size/CHUNK_SIZE);
 int remainder = (int)(size - ((long)chunks)*CHUNK_SIZE);
 data = new byte[chunks+(remainder==0?0:1)][];
 for( int idx=chunks; --idx&gt;=0; ) {
 data[idx] = new byte[(int)CHUNK_SIZE];
 }
 if( remainder != 0 ) {
 data[chunks] = new byte[remainder];
 }
 }
 }
 public byte get( long index ) {
 if( index&lt;0 || index&gt;=size ) {
 throw new IndexOutOfBoundsException("Error attempting to access data element "+index+". Array is "+size+" elements long.");
 }
 int chunk = (int)(index/CHUNK_SIZE);
 int offset = (int)(index - (((long)chunk)*CHUNK_SIZE));
 return data[chunk][offset];
 }
 public void set( long index, byte b ) {
 if( index&lt;0 || index&gt;=size ) {
 throw new IndexOutOfBoundsException("Error attempting to access data element "+index+". Array is "+size+" elements long.");
 }
 int chunk = (int)(index/CHUNK_SIZE);
 int offset = (int)(index - (((long)chunk)*CHUNK_SIZE));
 data[chunk][offset] = b;
 }
 /**
 * Simulates a single read which fills the entire array via several smaller reads.
 * 
 * @param fileInputStream
 * @throws IOException
 */
 public void read( FileInputStream fileInputStream ) throws IOException {
 if( size == 0 ) {
 return;
 }
 for( int idx=0; idx&lt;data.length; idx++ ) {
 if( fileInputStream.read( data[idx] ) != data[idx].length ) {
 throw new IOException("short read");
 }
 }
 }
 public long size() {
 return size;
 }
}
}
</code></pre>

blocks|key|1339850|text|java目前不支持元素超过2%5E32直接数组，|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1339851|我希望将来能看到java的这个特性。|1339852|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

java doesn't support direct array with more than 2^32 elements presently,

hope to see this feature of java in future

blocks|key|1555541|text|我认为内存映射文件的想法(使用CPU的虚拟内存硬件)是正确的方法。除了MappedByteBuffer具有与本机阵列相同的2+2Gb限制。这个家伙声称已经用MappedByteBuffer的一个非常简单的替代方案解决了这个问题：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1555542|http://nyeggen.com/post/2014-05-18-memory-mapping-%253E2gb-of-data-in-java/|offset|length|1555543|https://gist.github.com/bnyeggen/c679a5ea6a68503ed19f#file-mmapper-java|1555544|不幸的是，当你读超过500Mb时，JVM就会崩溃。|1555545|entityMap|0|LINK|mutability|MUTABLE|url|http://nyeggen.com/post/2014-05-18-memory-mapping-%253E2gb-of-data-in-java/|1^0|0|0|21|0|0|0|1Z|1|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|T|8|@]|9|@$D|U|E|V|1|W]]|A|$]]|$1|F|3|G|5|6|7|X|8|@]|9|@$D|Y|E|Z|1|10]]|A|$]]|$1|H|3|I|5|6|7|11|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|12|8|@]|9|@]|A|$]]]|K|$L|$5|M|N|O|A|$P|Q]]|R|$5|M|N|O|A|$P|G]]]]

I think the idea of memory-mapping the file (using the CPU's virtual memory hardware) is the right approach. Except that MappedByteBuffer has the same limitation of 2Gb as native arrays. This guy claims to have solved the problem with a pretty simple alternative to MappedByteBuffer: 

<a href="http://nyeggen.com/post/2014-05-18-memory-mapping-%3E2gb-of-data-in-java/" rel="nofollow">http://nyeggen.com/post/2014-05-18-memory-mapping-%3E2gb-of-data-in-java/</a>

<a href="https://gist.github.com/bnyeggen/c679a5ea6a68503ed19f#file-mmapper-java" rel="nofollow">https://gist.github.com/bnyeggen/c679a5ea6a68503ed19f#file-mmapper-java</a>

Unfortunately the JVM crashes when you read beyond 500Mb.

I have a big file, it's expected to be around 12 GB. I want to load it all into memory on a beefy 64-bit machine with 16 GB RAM, but I think Java does not support byte arrays that big:

<pre><code>File f = new File(file);
long size = f.length();
byte data[] = new byte[size]; // &lt;- does not compile, not even on 64bit JVM
</code></pre>

Is it possible with Java?

The compile error from the Eclipse compiler is:

<pre><code>Type mismatch: cannot convert from long to int
</code></pre>

javac gives:

<pre><code>possible loss of precision
found : long
required: int
 byte data[] = new byte[size];
</code></pre>

Java array with more than 4gb elements

内存映射

编译器

Eclipse

Java

我有一个很大的文件，预计大约有12 GB。我想在一台拥有16 GB RAM的健壮的64位机器上将其全部加载到内存中，但我认为Java不支持那么大的字节数组：File f = new File(file);long size = f.length();byte data[] = new byte[size]; // <-...

问元素超过4 4gb的Java数组
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问元素超过4 4gb的Java数组EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问元素超过4 4gb的Java数组
EN