blocks|key|136365|text|从斯帕利版本0.4开始，不支持从RStudio桌面连接到远程Spark集群。相反，正如您提到的，推荐的方法是在Spark集群中安装RStudio服务器。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|136366|尽管如此，sparklyr中的常春藤枝正在探索与Livy的集成，这将使RStudio桌面能够通过Livy连接到远程Spark集群。|136367|entityMap|0|LINK|mutability|MUTABLE|url|https://cran.r-project.org/web/packages/sparklyr/index.html|1|https://github.com/rstudio/sparklyr/tree/feature/livy^0|6|3|1|3|0|0|F|4|1|0^^$0|@$1|2|3|4|5|6|7|R|8|@$9|S|A|T|B|C]]|D|@$9|U|A|V|1|W]]|E|$]]|$1|F|3|G|5|6|7|X|8|@]|D|@$9|Y|A|Z|1|10]]|E|$]]|$1|H|3|-4|5|6|7|11|8|@]|D|@]|E|$]]]|I|$J|$5|K|L|M|E|$N|O]]|P|$5|K|L|M|E|$N|Q]]]]

As of <a href="https://cran.r-project.org/web/packages/sparklyr/index.html" rel="noreferrer">sparklyr</a> version <code>0.4</code>, it is unsupported to connect from the RStudio desktop to a remote Spark cluster. Instead, as you mention, the recommended approach is to install RStudio Server within the Spark cluster.

That said, the <a href="https://github.com/rstudio/sparklyr/tree/feature/livy" rel="noreferrer">livy branch</a> in sparklyr is exploring integration with Livy that would enable the RStudio desktop to connect to a remote Spark cluster through Livy.

blocks|key|2056648|text|使用最新版本的sparklyr+(例如，版本0.9.2+)，就可以连接到远程星火集群。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2056649|下面是一个连接到星火独立集群版本2.3.1的示例。有关其他主URL方案，请参见主URL。|BOLD|2056650|#install.packages("sparklyr")
library(sparklyr)

#+You+have+to+install+locally+(on+the+driver+where+RStudio+is+running)+the+same+Spark+version
spark_v+<-+"2.3.1"
cat("Installing+Spark+in+the+directory:",+spark_install_dir())
spark_install(version+=+spark_v)

sc+<-+spark_connect(spark_home+=+spark_install_find(version=spark_v)$sparkVersionDir,+
++++++++++++++++++++master+=+"spark://ip-[MY_PRIVATE_IP]:7077")

sc$master
#+"spark://ip-[MY_PRIVATE_IP]:7077"|code-block|syntax|javascript|2056651|我写了一个关于这个主题的https://www.back2code.me/2018/11/configure-sparklyr-to-connect-to-a-standalone-spark-cluster/。|2056652|entityMap|0|LINK|mutability|MUTABLE|url|https://spark.apache.org/docs/latest/submitting-applications.html#master-urls|1|https://www.back2code.me/2018/11/configure-sparklyr-to-connect-to-a-standalone-spark-cluster/^0|M|5|0|A|2|G|5|13|4|0|0|0|C|2L|1|0^^$0|@$1|2|3|4|5|6|7|Z|8|@$9|10|A|11|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|12|8|@$9|13|A|14|B|H]|$9|15|A|16|B|C]]|D|@$9|17|A|18|1|19]]|E|$]]|$1|I|3|J|5|K|7|1A|8|@]|D|@]|E|$L|M]]|$1|N|3|O|5|6|7|1B|8|@]|D|@$9|1C|A|1D|1|1E]]|E|$]]|$1|P|3|-4|5|6|7|1F|8|@]|D|@]|E|$]]]|Q|$R|$5|S|T|U|E|$V|W]]|X|$5|S|T|U|E|$V|Y]]]]

Using more recent version of sparklyr (version <code>0.9.2</code> for example) it's possible to connect to a remote Spark cluster.

Here is an example to connect to a Spark standalone cluster version <code>2.3.1</code>.
See <a href="https://spark.apache.org/docs/latest/submitting-applications.html#master-urls" rel="noreferrer">Master URLs</a> for other master URL schemes.

<pre><code>#install.packages("sparklyr")
library(sparklyr)

# You have to install locally (on the driver where RStudio is running) the same Spark version
spark_v &lt;- "2.3.1"
cat("Installing Spark in the directory:", spark_install_dir())
spark_install(version = spark_v)

sc &lt;- spark_connect(spark_home = spark_install_find(version=spark_v)$sparkVersionDir, 
 master = "spark://ip-[MY_PRIVATE_IP]:7077")

sc$master
# "spark://ip-[MY_PRIVATE_IP]:7077"
</code></pre>

I've written a <a href="https://www.back2code.me/2018/11/configure-sparklyr-to-connect-to-a-standalone-spark-cluster/" rel="noreferrer">post</a> on this topic.

blocks|key|136432|text|最后，我使用李薇成功地将本地R连接到星火集群的云实例(在我的例子中是HD洞察力)|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|136433|在sparklyr的spark_connect中，有一个连接到livy的选项。(方法=+"livy")|style|BOLD|136434|sc+<-+spark_connect(master+=+"https://<clustername>.azurehdinsight.net/livy/",
+++++++++++++++++++++method+=+"livy",+config+=+livy_config(
+++++++++++++++++++++++username+=+"<admin>",
+++++++++++++++++++++++password+=+rstudioapi::askForPassword("Livy+password:")))|code-block|syntax|javascript|136435|entityMap|0|LINK|mutability|MUTABLE|url|https://livy.incubator.apache.org/^0|6|2|0|0|19|4|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@$A|V|B|W|1|X]]|C|$]]|$1|D|3|E|5|6|7|Y|8|@$A|Z|B|10|F|G]]|9|@]|C|$]]|$1|H|3|I|5|J|7|11|8|@]|9|@]|C|$K|L]]|$1|M|3|-4|5|6|7|12|8|@]|9|@]|C|$]]]|N|$O|$5|P|Q|R|C|$S|T]]]]

I finally managed to connect my local R to a cloud instance of Spark cluster (HD insights in my case) using <a href="https://livy.incubator.apache.org/" rel="nofollow noreferrer">Livy</a>

within sparklyr's spark_connect there is an option to connect to livy. (Method = "livy")

<pre><code>sc &lt;- spark_connect(master = "https://&lt;clustername&gt;.azurehdinsight.net/livy/",
 method = "livy", config = livy_config(
 username = "&lt;admin&gt;",
 password = rstudioapi::askForPassword("Livy password:")))
</code></pre>

I would like to connect my local desktop RStudio session to a remote spark session via sparklyr. When you go to add a new connection in the sparklyr ui tab in RStudio and choose cluster is says that you have to be running on the cluster, or have a high bandwidth connection to the cluster. 

Can anyone shed light on how to create that kind of connection? I am not sure how to create reproducible example of this, but in general what I would like to do is: 

<pre><code>library(sparklyr)
sc &lt;- spark_connect(master = "spark://ip-[MY_PRIVATE_IP]:7077", spark_home = "/home/ubuntu/spark-2.0.0", version="2.0.0")
</code></pre>

from a remote server. I understand that there will be latency, especially if trying to pass data between the remotes. I also understand that it would be better to have the rstudio-server on the actual cluster- but that is not always possible, and I am looking for a sparklyr option for interacting between my server and my desktop RStudio session. Thanks.

Connect sparklyr to remote spark connection

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我想将本地桌面RStudio会话通过sparklyr连接到远程spark会话。当您在RStudio中的sparklyr选项卡中添加一个新连接并选择群集时，表示您必须在集群上运行，或者具有到集群的高带宽连接。有人能说明如何建立这种联系吗？我不知道如何创造这方面的可复制的例子，但总的来说，我想做的是：library(spa...

问将火花连接到远程火花连接。
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问将火花连接到远程火花连接。EN