Spark wordcount开发并提交到单机(伪分布式)运行

使用的ide是idea

packagecom.fsl

importorg.apache.spark.{SparkConf,SparkContext}

/**

* Created by Juvenile on 2017/2/24.

* 统计字符出现次数

*/

objectWordCountScala {

defmain(args: Array[String]) {

if(args.length <1) {

System.err.println("Usage: ")

System.exit(1)

}

valconf =newSparkConf()

valsc =newSparkContext(conf)

valline = sc.textFile(args(0))

line.flatMap(_.split("\t")).map((_,1)).reduceByKey(_ + _).collect.foreach(println)

sc.stop

}

}

导出成jar包。

上次到集群服务器: mySparkTest-1.0.jar

进入spark文件夹下执行 spark-submit --name WordCountScala --class com.fsl.WordCountScala --master spark://master:7077 --executor-memory 1G --total-executor-cores 1 /root/mySparkTest-1.0.jar hdfs://master:9000/user/data.txt 即可