腾讯云开发者社区-腾讯云

开发者社区

文档建议反馈控制台

最新优惠活动

文章/答案/技术大牛

发布

Jerry的SAP技术分享

Jerry Wang，2007 年从电子科技大学计算机专业硕士毕业后加入 SAP 成都研究院工作至今。

专栏作者

10840

文章

7377884

阅读量

159

订阅数

Spark 大数据实战：基于 RDD 的大数据处理分析

之前笔者参加了公司内部举办的一个 Big Data Workshop，接触了一些 Spark 的皮毛，后来在工作中陆陆续续又学习了一些 Spark 的实战知识。

2024-01-13

1590

how to run JavaWordCount in Spark

java spark maven

The general steps could be found in this link: http://stackoverflow.com/questions/22252534/how-to-run-a-spark-java-program-from-command-line

2021-02-20

2850

how to attach source code of Spark

Created by Wang, Jerry, last modified on Aug 27, 2015

2020-03-06

3810

Spark练习 - 提交作业到集群 - submit job via cluster

then ps -aux 7334 5.6 0.6 1146992 221652 pts/0 Sl 12:34 0:05 /usr/jdk1.7.0_79/bin/java -cp /root/devExpert/spark-1.4.1/sbin/…/conf/:/root/devExpert/spar monitor master node via url: http://10.128.184.131:8080 启动两个worker:

2020-03-06

8180

Spark平台上提交作业到集群生成的日志文件

spark 编程算法 java

./spark-submit --class “org.apache.spark.examples.JavaWordCount” --master spark://NKGV50849583FV1:7077 /root/devExpert/spark-1.4.1/example-java-build/JavaWordCount/target/JavaWordCount-1.jar /root/devExpert/spark-1.4.1/bin/test.txt added by Jerry: loading load-spark-env.sh !!! !!!1 added by Jerry:… /root/devExpert/spark-1.4.1/conf added by Jerry, number of Jars: 1 added by Jerry, launch_classpath: /root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar added by Jerry,RUNNER:/usr/jdk1.7.0_79/bin/java added by Jerry, printf argument list: org.apache.spark.deploy.SparkSubmit --class org.apache.spark.examples.JavaWordCount --master spark://NKGV50849583FV1:7077 /root/devExpert/spark-1.4.1/example-java-build/JavaWordCount/target/JavaWordCount-1.jar /root/devExpert/spark-1.4.1/bin/test.txt added by Jerry, I am in if-else branch: /usr/jdk1.7.0_79/bin/java -cp /root/devExpert/spark-1.4.1/conf/:/root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar:/root/devExpert/spark-1.4.1/lib_managed/jars /datanucleus-rdbms-3.2.9.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanu cleus-core-3.2.10.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-a pi-jdo-3.2.6.jar -Xms512m -Xmx512m -XX:MaxPermSize=256m org.apache.spark.deploy. SparkSubmit --master spark://NKGV50849583FV1:7077 --class org.apache.spark.examp les.JavaWordCount /root/devExpert/spark-1.4.1/example-java-build/JavaWordCount/t arget/JavaWordCount-1.jar /root/devExpert/spark-1.4.1/bin/test.txt

2020-03-06

8240

Spark平台上提交作业到集群生成的日志文件

java spark 编程算法

./spark-class org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077 NKGV50849583FV1:~/devExpert/spark-1.4.1/bin # ./spark-class org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077 added by Jerry: loading load-spark-env.sh !!!1 added by Jerry:… /root/devExpert/spark-1.4.1/conf added by Jerry, number of Jars: 1 added by Jerry, launch_classpath: /root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar added by Jerry,RUNNER:/usr/jdk1.7.0_79/bin/java added by Jerry, printf argument list: org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077 added by Jerry, I am in if-else branch: /usr/jdk1.7.0_79/bin/java -cp /root/devExpert/spark-1.4.1/conf/:/root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-core-3.2.10.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar -Xms512m -Xmx512m -XX:MaxPermSize=256m org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077 Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties 15/08/16 12:55:28 INFO Worker: Registered signal handlers for [TERM, HUP, INT]

2020-03-06

5240

Spark集群的启动日志

spark 编程算法 java

added by Jerry:… /root/devExpert/spark-1.4.1/sbin/…/conf – ( Jerry: I haven’t copied out the template from my own confiugration file yet ) starting org.apache.spark.deploy.master.Master, logging to /root/devExpert/spark-1.4.1/sbin/…/logs/spark-root-org.apache.spark.deploy.master.Master-1-NKGV50849583FV1.out Jerry- location of log file:

2020-03-06

7360

Jerry's spark demo application

java scala spark

在Cloud for Customer的Service Control Center里能看到C4C升级时间：

2020-03-03

3390

Jerry's spark demo application

java scala spark

在Cloud for Customer的Service Control Center里能看到C4C升级时间：

2020-03-03

3090

运行在Spark大数据上的应用体系架构

spark 大数据

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

2020-03-02

3010

spark-submit的执行原理

shell spark bash bash 指令编程算法

$ cd dirname $0 和PWD%/* shell变量的一些特殊用法在命令行状态下单纯执行 $ cd dirname $0 是毫无意义的。因为他返回当前路径的"."。这个命令写在脚本文件里才有作用，他返回这个脚本文件放置的目录，并可以根据这个目录来定位所要运行程序的相对位置（绝对位置除外）。在/home/admin/test/下新建test.sh内容如下： cd dirname $0 echo pwd 然后返回到/home/admin/执行 sh test/test.sh 运行结果: /home/admin/test 这样就可以知道一些和脚本一起部署的文件的位置了，只要知道相对位置就可以根据这个目录来定位，而可以不用关心绝对位置。这样脚本的可移植性就提高了，扔到任何一台服务器，（如果是部署脚本）都可以执行。 $0当前Shell程序的文件名 dirname $0，获取当前Shell程序的路径 cd dirname $0，进入当前Shell程序的目录实际上是直接执行pwd命令：

2019-12-18

9440

没有更多了

社区活动

腾讯技术创作狂欢月

“码”上创作 21 天，分 10000 元奖品池！

Python精品学习库

代码在线跑，知识轻松学

博客搬家 | 分享价值百万资源包

自行/邀约他人一键搬运博客，速成社区影响力并领取好礼

技术创作特训营·精选知识专栏

往期视频·千货材料·成员作品最新动态