blocks|key|1741156|text|pagerank模块有一个很好的networkx实现。它使用scipy/numpy作为矩阵实现。下面两个关于堆栈溢出的问题应该足以让您开始工作。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1741157|加权边如何影响网络中的PageRank？|unordered-list-item|1741158|枕骨？|1741159|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/9136539/how-to-weighted-edges-affect-pagerank-in-networkx|1|https://stackoverflow.com/questions/13040548/networkx-differences-between-pagerank-pagerank-numpy-and-pagerank-scipy^0|G|8|0|0|K|0|0|0|3|1|0^^$0|@$1|2|3|4|5|6|7|U|8|@$9|V|A|W|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|X|8|@]|D|@$9|Y|A|Z|1|10]]|E|$]]|$1|I|3|J|5|H|7|11|8|@]|D|@$9|12|A|13|1|14]]|E|$]]|$1|K|3|-4|5|6|7|15|8|@]|D|@]|E|$]]]|L|$M|$5|N|O|P|E|$Q|R]]|S|$5|N|O|P|E|$Q|T]]]]

Python <code>networkx</code> module has a nice implementation of pagerank. It uses scipy/numpy for the matrix implementation. The below two questions on stackoverflow should be enough to get you started. 

<ul>
<li><a href="https://stackoverflow.com/questions/9136539/how-to-weighted-edges-affect-pagerank-in-networkx">How do weighted edges affect PageRank in networkx?</a></li>
<li><a href="https://stackoverflow.com/questions/13040548/networkx-differences-between-pagerank-pagerank-numpy-and-pagerank-scipy">Networkx: Differences between pagerank, pagerank_numpy, and pagerank_scipy?</a></li>
</ul>

blocks|key|2219722|text|正如Dan+建议的那样，试图增加堆大小。如果从命令行运行Java应用程序，只需添加具有所需堆大小的开关-Xmx即可。假设您将Java代码编译到一个名为pagerank.jar的可运行JAR文件中，并且希望将堆大小设置为512+MB，您将发出以下命令：|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|CODE|entityRanges|data|2219723|java+-jar+-Xmx512m+pagerank.jar|code-block|syntax|javascript|2219724|编辑：，但只有在没有那么多“页面”的情况下才能起作用.一个100万x+1百万数组太大，无法容纳您的RAM+(1万亿倍*+64位双值=+7.27595761+to+)。您应该更改算法，从磁盘加载大块数据，对其进行操作，并将其存储回磁盘。|2219725|为此，您可以使用像Neo4j这样的图形数据库。|2219726|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Neo4j^0|2|4|C|7|1F|4|23|C|0|0|0|3|0|9|5|0|0^^$0|@$1|2|3|4|5|6|7|X|8|@$9|Y|A|Z|B|C]|$9|10|A|11|B|C]|$9|12|A|13|B|D]|$9|14|A|15|B|D]]|E|@]|F|$]]|$1|G|3|H|5|I|7|16|8|@]|E|@]|F|$J|K]]|$1|L|3|M|5|6|7|17|8|@$9|18|A|19|B|C]]|E|@]|F|$]]|$1|N|3|O|5|6|7|1A|8|@]|E|@$9|1B|A|1C|1|1D]]|F|$]]|$1|P|3|-4|5|6|7|1E|8|@]|E|@]|F|$]]]|Q|$R|$5|S|T|U|F|$V|W]]]]

As Dan W suggested, try to increase the heap size. If you run your Java application from the command line, just add the switch <code>-Xmx</code> with the desired heap size. Let's assume you compiled your Java code into a runnable JAR file called <code>pagerank.jar</code>, and you want to set your heap size to 512 MB, you would issue the following command:

<pre><code>java -jar -Xmx512m pagerank.jar
</code></pre>

EDIT:
But that only works if you don't have that many "pages" ... A 1 Million x 1 Million array is too big to fit into your RAM (1 trillion times * 64 bit double value = 7.27595761 terabytes). You should change your algorithm to load chunks of data from the disk, manipulate it, and store it back to disk.

You could use a graph database like <a href="http://en.wikipedia.org/wiki/Neo4j" rel="nofollow">Neo4j</a> for that purpose.

blocks|key|1103160|text|PageRank是由谷歌使用'Pregel‘BSP+(实际上只是关键字)框架执行的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1103161|我记得阿帕奇图+(另一个Pregel)，它在它的基准包中包含了一个版本的PageRank。|offset|length|1103162|这是一段关于Gi相图的视频：这是一个介绍，它专门讨论如何处理PageRank。|1103163|如果不起作用的话：|1103164|在Java中，有一个名为GoldenOrb的Pregel实现。|1103165|PageRank算法的伪代码是这里+(Pregel的不同实现)。|1103166|您必须阅读BSP和PageRank来处理您拥有的数据的大小。|1103167|entityMap|0|LINK|mutability|MUTABLE|url|http://giraph.apache.org/|1|http://www.youtube.com/watch?v=3ZrqPEIPRe4|2|http://goldenorbos.org/index.html|3|https://github.com/igrigorik/pregel^0|0|3|4|0|0|0|D|1|0|0|C|9|2|0|F|2|3|0|0^^$0|@$1|2|3|4|5|6|7|13|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|14|8|@]|9|@$D|15|E|16|1|17]]|A|$]]|$1|F|3|G|5|6|7|18|8|@]|9|@$D|19|E|1A|1|1B]]|A|$]]|$1|H|3|I|5|6|7|1C|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|1D|8|@]|9|@$D|1E|E|1F|1|1G]]|A|$]]|$1|L|3|M|5|6|7|1H|8|@]|9|@$D|1I|E|1J|1|1K]]|A|$]]|$1|N|3|O|5|6|7|1L|8|@]|9|@]|A|$]]|$1|P|3|-4|5|6|7|1M|8|@]|9|@]|A|$]]]|Q|$R|$5|S|T|U|A|$V|W]]|X|$5|S|T|U|A|$V|Y]]|Z|$5|S|T|U|A|$V|10]]|11|$5|S|T|U|A|$V|12]]]]

PageRank is performed by Google using the 'Pregel' BSP (really just keywords) framework.

I remembered <a href="http://giraph.apache.org/" rel="nofollow">Apache Giraph</a> (another Pregel), which includes a version of PageRank in its benchmark package.

<a href="http://www.youtube.com/watch?v=3ZrqPEIPRe4" rel="nofollow">Here's a video about Giraph</a>: it's an introduction, and it specifically talks about handling PageRank.

If that doesn't work:

In Java there is an implementation of Pregel called <a href="http://goldenorbos.org/index.html" rel="nofollow">GoldenOrb</a>.

Pseudo code for the PageRank algorithm is <a href="https://github.com/igrigorik/pregel" rel="nofollow">here</a> (on a different implementation of Pregel).

You'll have to read around BSP, and PageRank to handle the size of data you have.

blocks|key|1103175|text|您不必存储整个1000000x1000000矩阵，因为大多数矩阵条目将为零。相反，您可以(例如)存储每一行的非零项列表，并编写矩阵函数直接使用它，而不必将其展开为完整的矩阵。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1103176|这种压缩表示被称为稀疏矩阵格式，并且大多数矩阵库都有一个选项来构建和处理稀疏矩阵。|offset|length|1103177|稀疏矩阵的一个缺点是，将其中两个矩阵相乘会导致矩阵的稀疏性大大降低。然而，PageRank算法的设计使您不需要这样做:超链接矩阵是常数，并且只有分数向量被更新。|1103178|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Sparse_matrix^0|0|9|4|0|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@$D|R|E|S|1|T]]|A|$]]|$1|F|3|G|5|6|7|U|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

You don't have to store the whole 1000000x1000000 matrix, because most matrix entries will be zero. Instead, you can (for example) store a list of nonzero entries for each row, and write your matrix functions to use it directly, without expanding it into a full matrix.

This kind of compressed representation is called a <a href="http://en.wikipedia.org/wiki/Sparse_matrix" rel="nofollow">sparse matrix</a> format, and most matrix libraries have an option to build and work with sparse matrices.

One disadvantage with sparse matrices is that multiplying two of them will result in a matrix which is much less sparse. However, the PageRank algorithm is designed so that you don't need to do that: the hyperlink matrix is constant, and only the score vector is updated.

blocks|key|1741311|text|几点建议：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1741312|使用python，而不是Java:+python是一种优秀的原型语言，它具有可用的稀疏矩阵(在has中)以及许多其他优点。正如其他人所指出的，它还具有pagerank实现。|unordered-list-item|1741313|将数据全部存储在内存中:任何类型的轻量级数据库都可以，例如sqlite、hibernate、.|1741314|数据块的工作:如果有一个大矩阵NxN，将其分解为小块MxM，其中M是N的一个分数，适合内存。结合稀疏矩阵，您可以使用真正的大N(数亿到数十亿，取决于数据的稀疏程度)。|1741315|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$]]|$1|E|3|F|5|D|7|M|8|@]|9|@]|A|$]]|$1|G|3|H|5|D|7|N|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|O|8|@]|9|@]|A|$]]]|J|$]]

A few suggestions:

<ul>
<li>Use python, not Java: python is an excellent prototyping language, and has available sparse matrices (in scipy) as well as many other goodies. As others have noted, it also has a pagerank implementation.</li>
<li>Store your data not all in memory: any type of lightweight database would be fine, for instance sqlite, hibernate, ...</li>
<li>Work on tiles of the data: if there is a big matrix NxN, break it up into small tiles MxM where M is a fraction of N, that fit in memory. Combined with sparse matrices this allows you to work with really big N (hundreds of millions to billions, depending on how sparse the data is).</li>
</ul>

blocks|key|2219888|text|由于矩阵是稀疏的，所以可以实现svd、pca、mds或包括svd的Lsi之类的降维。有一个实现这种过程的库，称为Jama。你可以找到它，这里|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2219889|entityMap|0|LINK|mutability|MUTABLE|url|http://math.nist.gov/javanumerics/jama/^0|1W|2|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

Because the matrix is sparse you can implement dimensionality reduction like svd,pca,mds or Lsi that includes svd. There is a library to implement this kind of processes which is called Jama. You can find it <a href="http://math.nist.gov/javanumerics/jama/" rel="nofollow">here</a>

blocks|key|1741466|text|这是一篇关于pagerank的很棒的文章。我从它这里实现了一个Perl版本，以便与特克斯一起使用。但是，如果您只想了解pagerank以及本文中讨论的各个方面如何影响结果(阻尼因子、直接或无向图等)，我建议在R或八度中进行实验。如果您想学习如何有效地实现它，那么从零开始编程，就像您正在做的那样，是最好的方法。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1741467|大多数web图(或网络)都是非常稀疏的，这意味着图的矩阵表示中的大部分条目为零。用于表示稀疏矩阵的一种常见数据结构是散列映射，其中没有存储零值。例如，如果矩阵是|1741468|1,+0,+0
0,+0,+2,
0,+3,+0|code-block|syntax|javascript|1741469|二维散列映射将只存储hm(0,0)=1，hm(1,2)=2和hm(2,1)=3的值。因此，在一个web图的1,000,000×1,000,000矩阵中，我预计只有几百万个值是非零的。如果每行平均只有5个非零值，哈希映射将使用大约5*(8%2B8%2B8)_10%5E6字节~115+If来存储它(左int索引为8，右int索引为8，双值为8)。平方矩阵将使用8_10%5E6*10%5E6+~7兆字节。|1741470|在Java中实现高效的稀疏矩阵-向量乘并不容易，如果您不想把时间花在算法的那个方面，那么已经有一些实施了。稀疏矩阵乘是pagerank算法最难实现的方面，因此它变得更容易(也更有趣)。|1741471|entityMap|0|LINK|mutability|MUTABLE|url|http://search.cpan.org/%257Ekubina/Graph-Centrality-Pagerank/|1|http://search.cpan.org/%257Ekubina/Text-Categorize-Textrank/|2|https://www.r-project.org/|3|https://www.gnu.org/software/octave/|4|https://en.wikipedia.org/wiki/Sparse_matrix|5|https://docs.oracle.com/javase/6/docs/api/java/util/HashMap.html|6|https://doi.org/10.1007/3-540-48686-0_1|7|https://web.archive.org/web/20120112124729/http://commons.apache.org:80/math/apidocs/org/apache/commons/math/linear/SparseRealMatrix.html^0|O|2|0|15|3|1|2W|1|2|2Y|2|3|0|G|2|4|1M|4|5|0|0|1C|4|6|0|1D|2|7|0^^$0|@$1|2|3|4|5|6|7|1A|8|@]|9|@$A|1B|B|1C|1|1D]|$A|1E|B|1F|1|1G]|$A|1H|B|1I|1|1J]|$A|1K|B|1L|1|1M]]|C|$]]|$1|D|3|E|5|6|7|1N|8|@]|9|@$A|1O|B|1P|1|1Q]|$A|1R|B|1S|1|1T]]|C|$]]|$1|F|3|G|5|H|7|1U|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|1V|8|@]|9|@$A|1W|B|1X|1|1Y]]|C|$]]|$1|M|3|N|5|6|7|1Z|8|@]|9|@$A|20|B|21|1|22]]|C|$]]|$1|O|3|-4|5|6|7|23|8|@]|9|@]|C|$]]]|P|$Q|$5|R|S|T|C|$U|V]]|W|$5|R|S|T|C|$U|X]]|Y|$5|R|S|T|C|$U|Z]]|10|$5|R|S|T|C|$U|11]]|12|$5|R|S|T|C|$U|13]]|14|$5|R|S|T|C|$U|15]]|16|$5|R|S|T|C|$U|17]]|18|$5|R|S|T|C|$U|19]]]]

That is a great article to learn about pagerank. I implemented a Perl version from it <a href="http://search.cpan.org/%7Ekubina/Graph-Centrality-Pagerank/" rel="nofollow noreferrer">here</a> to use with <a href="http://search.cpan.org/%7Ekubina/Text-Categorize-Textrank/" rel="nofollow noreferrer">Textrank</a>. However, if you want to just learn about pagerank and how the various aspects discussed in the article affect the results (dampening factor, direct or undirected graph, etc.), I would recommend running experiments in <a href="https://www.r-project.org/" rel="nofollow noreferrer">R</a> or <a href="https://www.gnu.org/software/octave/" rel="nofollow noreferrer">Octave</a>. If you want to learn how to implement it efficiently, then programming it up from scratch, as you are doing, is best.
Most web graphs (or networks) are very <a href="https://en.wikipedia.org/wiki/Sparse_matrix" rel="nofollow noreferrer">sparse</a>, which means most of the entries in the matrix representation of the graph are zero. A common data structure used to represent a sparse matrix is a <a href="https://docs.oracle.com/javase/6/docs/api/java/util/HashMap.html" rel="nofollow noreferrer">hash-map</a>, where the zero values are not stored. For example, if the matrix was
<pre><code>1, 0, 0
0, 0, 2,
0, 3, 0
</code></pre>
a two dimension hash-map would store only the values for hm(0,0)=1, hm(1,2)=2, and hm(2,1)=3. So in a 1,000,000 by 1,000,000 matrix of a <a href="https://doi.org/10.1007/3-540-48686-0_1" rel="nofollow noreferrer">web graph</a>, I would expect only a few million values to be non-zero. If each row averages only 5 non-zero values, a hash-map will use about 5*(8+8+8)10^6 bytes ~ 115mb to store it (8 for the left int index, 8 for the right int index, and 8 for the double value). The square matrix will use 810^6*10^6 ~ 7 terabytes.
Implementing an efficient sparse matrix-vector multiply in Java is not trivial, and there are some already <a href="https://web.archive.org/web/20120112124729/http://commons.apache.org:80/math/apidocs/org/apache/commons/math/linear/SparseRealMatrix.html" rel="nofollow noreferrer">implemented</a> if you don't want to devote time to that aspect of the algorithm. The sparse-matrix multiply is the most difficult aspect to implement of the pagerank algorithm, so after that it gets easier (and more interesting).

After reading theory of PageRank algorithm from <a href="http://www.ams.org/samplings/feature-column/fcarc-pagerank" rel="nofollow">this site</a> I would like to play with it. 
I am trying to implement this in Java. I mean I would like to play with PageRank in detail (like giving different weights and so on). For this I need to build hyperlink matrix. If I have 1 million nodes then my hyperlink matrix will be 1 million x 1 million size, which causes this exception:

<pre><code>Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
 at WebGraph.main(WebGraph.java:6)
</code></pre>

How can I implement PageRank in Java, is there any way of storing hyperlink matrix?

PageRank implementation for Research

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

在阅读了PageRank算法的理论后，我想利用算法。我正在尝试用Java实现这一点。我的意思是，我想详细地玩PageRank (比如给出不同的权重等等)。为此，我需要建立超链接矩阵。如果我有100万个节点，那么我的超级链接矩阵将是100万×100万大小，这将导致这个异常：Exception in thread "mai...

问用于研究的PageRank实现
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问用于研究的PageRank实现EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问用于研究的PageRank实现
EN