我正在使用普通Java实现mapreduce框架,而不使用第三方库,
我正在寻找Split阶段实现的帮助,我有一个文本文件需要被平均分成段,并将每个段传递给不同的进程。
如何在java中实现多个进程,用户设置进程数,然后将每个文本段传递给每个进程?
发布于 2019-06-05 14:35:59
我试图完成类似的事情(在数组上进行多线程操作),也许这个测试用例可以帮助你获得通用算法:
final int cpuCores=4; //or Runtime.getRuntime().availableProcessors();
@Test
public void threadedOperation()
{
int[] ints=new int[new Random().nextInt(2000)+4000];
System.out.println("Array size="+ints.length);
Arrays.fill(ints,2);
int chunksize;
final int arraysize=ints.length;
final int remainder=arraysize % cpuCores;
//checking if the work can be split among threads evenly
if(remainder==0)
{
chunksize=arraysize/cpuCores;
}
else{
//can't split evenly, so take the remainder into acoount
int in=arraysize-remainder;
chunksize=in/cpuCores;
System.out.println("Remainder="+remainder);
}
System.out.println("Chunk size="+chunksize);
System.out.println("Threads="+cpuCores);
assert chunksize*cpuCores+remainder==arraysize;
for (int core = 0; core < cpuCores; core++) {
int start=core * chunksize;
int end = start + chunksize;
if(core==cpuCores-1)
{
//here I "add" the remainder to the last thread
if (remainder != 0) {
end=start+chunksize+remainder;
}
}
System.out.println("Start="+start+" end="+end);
int finalEnd = end;
Thread thread=new Thread(() -> {
for (int i = start; i < finalEnd; i++) {
//do your work here, I'm just asserting that the array is filled
assert ints[i]==2;
}
});
thread.start();
}
}
https://stackoverflow.com/questions/-100006928
复制相似问题