版权声明:欢迎转载,请注明出处,谢谢。 https://blog.csdn.net/boling_cavalry/article/details/87510822
Scala语言在函数式编程方面的优势适合Spark应用开发,IDEA是我们常用的IDE工具,今天就来实战IDEA开发Scala版的Spark应用;
以下是开发环境:
以下是运行环境:
如果您想在几分钟之内搭建好Spark集群环境,请参考《docker下,极速搭建spark集群(含hdfs集群)》;
package com.bolingcavalry.sparkscalademo.app
import org.apache.spark.{SparkConf, SparkContext}
/**
* @Description: 第一个scala语言的spark应用
* @author: willzhao E-mail: zq2599@gmail.com
* @date: 2019/2/16 20:23
*/
object FirstDemo {
def main(args: Array[String]): Unit={
val conf = new SparkConf()
.setAppName("first spark app(scala)")
.setMaster("local[1]");
new SparkContext(conf)
.parallelize(List(1,2,3,4,5,6))
.map(x=>x*x)
.filter(_>10)
.collect()
.foreach(println);
}
}
以上代码的功能很简单:创建用一个数组,将每个元素做平方运算,再丢弃小于10的元素,然后逐个打印出来;
2019-02-17 09:04:21 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2019-02-17 09:04:21 INFO DAGScheduler:54 - ResultStage 0 (collect at FirstDemo.scala:20) finished in 0.276 s
2019-02-17 09:04:21 INFO DAGScheduler:54 - Job 0 finished: collect at FirstDemo.scala:20, took 0.328611 s
16
25
36
2019-02-17 09:04:21 INFO SparkContext:54 - Invoking stop() from shutdown hook
2019-02-17 09:04:21 INFO AbstractConnector:318 - Stopped Spark@452ba1db{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-02-17 09:04:21 INFO SparkUI:54 - Stopped Spark web UI at http://DESKTOP-82CCEBN:4040
2019-02-17 09:04:21 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2019-02-17 09:04:21 INFO MemoryStore:54 - MemoryStore cleared
2019-02-17 09:04:21 INFO BlockManager:54 - BlockManager stopped
2019-02-17 09:04:21 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2019-02-17 09:04:21 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2019-02-17 09:04:21 INFO SparkContext:54 - Successfully stopped SparkContext
2019-02-17 09:04:21 INFO ShutdownHookManager:54 - Shutdown hook called
2019-02-17 09:04:21 INFO ShutdownHookManager:54 - Deleting directory C:\Users\12167\AppData\Local\Temp\spark-4bbb584a-c7c2-4dc8-9c7e-473de7f8c326
Process finished with exit code 0
spark-submit --class com.bolingcavalry.sparkscalademo.app.FirstDemo /root/jars/sparkscalademo.jar
至此,idea开发spark应用实战就完成了,希望在您配置开发环境的时候本文能够提供一些参考;