RAPIDS cuGraph库是一组图形分析,用于处理GPU数据帧中的数据 - 请参阅cuDF。 cuGraph旨在提供类似NetworkX的API,这对数据科学家来说很熟悉,因此他们现在可以更轻松地构建GPU加速的工作流程
官方文档:
支持的模型:
关联文章:
nvidia-rapids︱cuDF与pandas一样的DataFrame库
NVIDIA的python-GPU算法生态 ︱ RAPIDS 0.10
nvidia-rapids︱cuGraph(NetworkX-like)关系图模型
Conda安装,https://github.com/rapidsai/cugraph:
# CUDA 10.0
conda install -c nvidia -c rapidsai -c numba -c conda-forge -c defaults cugraph cudatoolkit=10.0
# CUDA 10.1
conda install -c nvidia -c rapidsai -c numba -c conda-forge -c defaults cugraph cudatoolkit=10.1
# CUDA 10.2
conda install -c nvidia -c rapidsai -c numba -c conda-forge -c defaults cugraph cudatoolkit=10.2
docker版本,可参考:https://rapids.ai/start.html#prerequisites
docker pull rapidsai/rapidsai:cuda10.1-runtime-ubuntu16.04-py3.7
docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \
rapidsai/rapidsai:cuda10.1-runtime-ubuntu16.04-py3.7
cuGraph已在将领先的图形框架集成到一个简单易用的接口方面迈出了新的一步。几个月前,RAPIDS收到了来自佐治亚理工学院的Hornet副本,并将其重构和重命名为cuHornet。这一名称更改表明,源代码已偏离Georgia Tech基准并体现了代码API和数据结构与RAPIDS cuGraph的匹配。cuHornet的加入提供了基于边界的编程模型、动态数据结构以及现有分析的列表。除了核心数函数之外,可用的前两个cuHornet算法是Katz centrality 和K-Cores。
cuGraph是RAPIDS的图形分析库,针对cuGraph我们推出了一个由两个新原语支持的多GPU PageRank算法:这是一个COO到CSR的多GPU数据转换器,和一个计算顶点度的函数。这些原语会被用于将源和目标边缘列从Dask Dataframe转换为图形格式,并使PageRank能够跨越多个GPU进行缩放。
下图显示了新的多GPU PageRank算法的性能。与之前的PageRank基准运行时刻不同,这些运行时刻只是测量PageRank解算器的性能。这组运行时刻包括Dask DataFrame到CSR的转换、PageRank执行以及从CSR返回到DataFrame的结果转换。平均结果显示,新的多GPU PageRank分析比100节点Spark集群快10倍以上。
图1:cuGraph PageRank在不同数量的边缘和NVIDIA Tesla V 100上计算所用的时间
下图仅查看Bigdata数据集、5000万个顶点和19.8亿条边,并运行HiBench端到端测试。HiBench基准运行时刻包括数据读取、运行PageRank,然后得到所有顶点的得分。此前,HiBench分别在10、20、50和100个节点的Google GCP上进行了测试。
图2:5千万边缘端到端PageRank运行时刻,cuGraph PageRank vs Spark Graph(越低越好)
参考:https://github.com/rapidsai/cugraph
import cugraph
# assuming that data has been loaded into a cuDF (using read_csv) Dataframe
gdf = cudf.read_csv("graph_data.csv", names=["src", "dst"], dtype=["int32", "int32"] )
# create a Graph using the source (src) and destination (dst) vertex pairs the GDF
G = cugraph.Graph()
G.add_edge_list(gdf, source='src', destination='dst')
# Call cugraph.pagerank to get the pagerank scores
gdf_page = cugraph.pagerank(G)
for i in range(len(gdf_page)):
print("vertex " + str(gdf_page['vertex'][i]) +
" PageRank is " + str(gdf_page['pagerank'][i]))
cugraph.pagerank(G,alpha=0.85, max_iter=100, tol=1.0e-5)
Returns:
安装:
# The notebook compares cuGraph to NetworkX,
# therefore there some additional non-RAPIDS python libraries need to be installed.
# Please run this cell if you need the additional libraries
!pip install networkx
!pip install scipy
代码模块:
# Import needed libraries
import cugraph
import cudf
from collections import OrderedDict
# NetworkX libraries
import networkx as nx
from scipy.io import mmread
# 相关参数
# define the parameters
max_iter = 100 # The maximum number of iterations
tol = 0.00001 # tolerance
alpha = 0.85 # alpha
# Define the path to the test data
datafile='../data/karate-data.csv'
# NetworkX
# Read the data, this also created a NetworkX Graph
file = open(datafile, 'rb')
Gnx = nx.read_edgelist(file)
pr_nx = nx.pagerank(Gnx, alpha=alpha, max_iter=max_iter, tol=tol)
cuGraph模型:
# cuGraph
# Read the data
gdf = cudf.read_csv(datafile, names=["src", "dst"], delimiter='\t', dtype=["int32", "int32"] )
# create a Graph using the source (src) and destination (dst) vertex pairs from the Dataframe
G = cugraph.Graph()
G.from_cudf_edgelist(gdf, source='src', destination='dst')
# Call cugraph.pagerank to get the pagerank scores
gdf_page = cugraph.pagerank(G)
# Find the most important vertex using the scores
# This methods should only be used for small graph
bestScore = gdf_page['pagerank'][0]
bestVert = gdf_page['vertex'][0]
for i in range(len(gdf_page)):
if gdf_page['pagerank'][i] > bestScore:
bestScore = gdf_page['pagerank'][i]
bestVert = gdf_page['vertex'][i]
print("Best vertex is " + str(bestVert) + " with score of " + str(bestScore))
# A better way to do that would be to find the max and then use that values in a query
pr_max = gdf_page['pagerank'].max()
def print_pagerank_threshold(_df, t=0) :
filtered = _df.query('pagerank >= @t')
for i in range(len(filtered)):
print("Best vertex is " + str(filtered['vertex'][i]) +
" with score of " + str(filtered['pagerank'][i]))
print_pagerank_threshold(gdf_page, pr_max)
sort_pr = gdf_page.sort_values('pagerank', ascending=False)
d = G.degrees()
d.sort_values('out_degree', ascending=False).head(4)
关联结果: