blocks|key|2349185|text|如果系统有足够的存储空间，你可以通过DB来完成。例如：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2349186|在表A和B中创建一个H2或sqlite+DB+(数据存储在磁盘上，尽可能多地分配缓存)加载文本文件(在'url‘列上创建索引)|2349187|select+url+from+A+where+URL+not+in+(select+distinct+url+from+B)
select+url+from+B+where+URL+not+in+(select+distinct+url+from+A)|code-block|syntax|javascript|2349188|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

If system has enough storage, you can do this via DB. For example :

Create an H2 or sqlite DB (data stored on disk, allocate as much
 cache as system can afford) 
Load text file in tables A and B (create index on 'url' column)

<pre><code>select url from A where URL not in (select distinct url from B)
select url from B where URL not in (select distinct url from A)
</code></pre>

blocks|key|2349209|text|下面是我提出的解决方案的要点：https://gist.github.com/nddipiazza/16cb2a0d23ee60a07121893c26065de4|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2349210|import+com.google.common.collect.Sets;
import+org.apache.commons.io.FileUtils;
import+org.apache.commons.io.LineIterator;

import+java.io.File;
import+java.io.IOException;
import+java.util.HashSet;
import+java.util.Set;

public+class+DiffTextFilesUtil+{
++static+public+int+CHUNK_SIZE+=+100000;

++static+public+class+DiffResult+{
++++public+Set<String>+addedVals+=+new+HashSet<>();
++++public+Set<String>+removedVals+=+new+HashSet<>();
++}

++/**
+++*+Gets+diff+result+of+two+sorted+files+with+each+other.
+++*+@param+lhs+left+hand+file+-+sort+this+using+com.google.code.externalsortinginjava:externalsortinginjava:0.2.5
+++*+@param+rhs+right+hand+file+-+sort+this+using+com.google.code.externalsortinginjava:externalsortinginjava:0.2.5
+++*+@return+DiffResult.addedVals+were+added+from+lhs+to+rhs.+DiffResult.removedVals+were+removed+from+lhs+to+rhs.
+++*+@throws+IOException
+++*/
++public+static+DiffResult+diff(File+lhs,+File+rhs)+throws+IOException+{

++++DiffResult+diffResult+=+new+DiffResult();

++++LineIterator+lhsIter+=+FileUtils.lineIterator(lhs);
++++LineIterator+rhsIter+=+FileUtils.lineIterator(rhs);

++++String+lhsTop+=+null;
++++String+rhsTop+=+null;
++++while+(lhsIter.hasNext())+{
++++++int+ct+=+CHUNK_SIZE;

++++++Set<String>+setLhs+=+Sets.newHashSet();
++++++Set<String>+setRhs+=+Sets.newHashSet();
++++++while+(lhsIter.hasNext()+&&+--ct+>+0)+{
++++++++lhsTop+=+lhsIter.nextLine();
++++++++setLhs.add(lhsTop);
++++++}
++++++while+(rhsIter.hasNext())+{
++++++++if+(rhsTop+!=+null+&&+rhsTop.compareTo(lhsTop)+>+0)+{
++++++++++break;
++++++++}+else+if+(rhsTop+!=+null+&&+rhsTop.compareTo(lhsTop)+==+0)+{
++++++++++setRhs.add(rhsTop);
++++++++++rhsTop+=+null;
++++++++++break;
++++++++}+else+if+(rhsTop+!=+null)+{
++++++++++setRhs.add(rhsTop);
++++++++}
++++++++rhsTop+=+rhsIter.next();
++++++}
++++++if+(rhsTop+!=+null)+{
++++++++setRhs.add(rhsTop);
++++++}
++++++Sets.difference(setLhs,+setRhs).copyInto(diffResult.removedVals);
++++++Sets.difference(setRhs,+setLhs).copyInto(diffResult.addedVals);
++++}
++++return+diffResult;
++}
}|code-block|syntax|javascript|2349211|entityMap|0|LINK|mutability|MUTABLE|url|https://gist.github.com/nddipiazza/16cb2a0d23ee60a07121893c26065de4^0|F|1V|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

Here is a gist of the solution I came up with: <a href="https://gist.github.com/nddipiazza/16cb2a0d23ee60a07121893c26065de4" rel="nofollow noreferrer">https://gist.github.com/nddipiazza/16cb2a0d23ee60a07121893c26065de4</a>

<pre><code>import com.google.common.collect.Sets;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.LineIterator;

import java.io.File;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;

public class DiffTextFilesUtil {
 static public int CHUNK_SIZE = 100000;

 static public class DiffResult {
 public Set&lt;String&gt; addedVals = new HashSet&lt;&gt;();
 public Set&lt;String&gt; removedVals = new HashSet&lt;&gt;();
 }

 /**
 * Gets diff result of two sorted files with each other.
 * @param lhs left hand file - sort this using com.google.code.externalsortinginjava:externalsortinginjava:0.2.5
 * @param rhs right hand file - sort this using com.google.code.externalsortinginjava:externalsortinginjava:0.2.5
 * @return DiffResult.addedVals were added from lhs to rhs. DiffResult.removedVals were removed from lhs to rhs.
 * @throws IOException
 */
 public static DiffResult diff(File lhs, File rhs) throws IOException {

 DiffResult diffResult = new DiffResult();

 LineIterator lhsIter = FileUtils.lineIterator(lhs);
 LineIterator rhsIter = FileUtils.lineIterator(rhs);

 String lhsTop = null;
 String rhsTop = null;
 while (lhsIter.hasNext()) {
 int ct = CHUNK_SIZE;

 Set&lt;String&gt; setLhs = Sets.newHashSet();
 Set&lt;String&gt; setRhs = Sets.newHashSet();
 while (lhsIter.hasNext() &amp;&amp; --ct &gt; 0) {
 lhsTop = lhsIter.nextLine();
 setLhs.add(lhsTop);
 }
 while (rhsIter.hasNext()) {
 if (rhsTop != null &amp;&amp; rhsTop.compareTo(lhsTop) &gt; 0) {
 break;
 } else if (rhsTop != null &amp;&amp; rhsTop.compareTo(lhsTop) == 0) {
 setRhs.add(rhsTop);
 rhsTop = null;
 break;
 } else if (rhsTop != null) {
 setRhs.add(rhsTop);
 }
 rhsTop = rhsIter.next();
 }
 if (rhsTop != null) {
 setRhs.add(rhsTop);
 }
 Sets.difference(setLhs, setRhs).copyInto(diffResult.removedVals);
 Sets.difference(setRhs, setLhs).copyInto(diffResult.addedVals);
 }
 return diffResult;
 }
}
</code></pre>

I have the following scenario:

<ul>
<li>url text file A</li>
<li>url text file B</li>
</ul>

Each file's size is around 4Gb.

I need to calculate:

<ul>
<li>all urls in A that are not in B</li>
<li>all urls in B that are not in A</li>
</ul>

All of the Java-diff examples I'm finding online load the entire list in memory (either with a Map or using an MMap solution). My system doesn't have swap and lacks the memory to be able to do this without External-Memory.

Does anyone know of a solution for this? 

This project can do huge file sorts without eating up tons of memory <a href="https://github.com/lemire/externalsortinginjava" rel="nofollow noreferrer">https://github.com/lemire/externalsortinginjava</a>

I am looking for something similar, but for generating diffs. I'm going to start by trying to implement this using that project as a baseline.

Comparing two text large files with URLs in Java with external memory only?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我有以下场景：url文本文件Aurl文本文件B每个文件的大小约为4 4Gb。我需要计算一下：A中不在B中的所有urls B中不在A中的所有urls我在网上找到的所有Java-diff示例都将整个列表加载到内存中(使用Map或使用MMap解决方案)。我的系统没有交换空间，也没有足够的内存在没有外部内存的情况下做到这一点。...

问比较两个文本大文件与只有外部存储器的Java中的URL？
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问比较两个文本大文件与只有外部存储器的Java中的URL？EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问比较两个文本大文件与只有外部存储器的Java中的URL？
EN