前往小程序，Get更优阅读体验！
立即前往
文档建议反馈控制台
首页
学习
活动
专区
工具
TVP
最新优惠活动
文章/答案/技术大牛
发布
首页
学习
活动
专区
工具
TVP最新优惠活动
返回腾讯云官网
社区首页 >专栏 >Spark2.x学习笔记：16、Spark Streaming入门实例NetworkWordCount

Spark2.x学习笔记：16、Spark Streaming入门实例NetworkWordCount

作者头像
程裕强
发布于 2018-01-02 16:50:44
1.1K0
发布于 2018-01-02 16:50:44
举报
文章被收录于专栏：大数据学习笔记大数据学习笔记

16、 Spark Streaming入门实例NetworkWordCount

16.1 源码解析

在Spark源码的spark-2.1.0-bin-hadoop2.7\examples\src\main\scala\org\apache\spark\examples\streaming目录下即可找到NetworkWordCount.scala源文件。

代码语言：javascript
复制
// scalastyle:off println
package org.apache.spark.examples.streaming

import org.apache.spark.SparkConf
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Seconds, StreamingContext}

object NetworkWordCount {
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: NetworkWordCount <hostname> <port>")
      System.exit(1)
    }

    StreamingExamples.setStreamingLogLevels()

    // Create the context with a 1 second batch size
    //创建SparkConf实例
    val sparkConf = new SparkConf().setAppName("NetworkWordCount")
    //每隔1秒钟处理一批数据
    val ssc = new StreamingContext(sparkConf, Seconds(1))

    // Create a socket stream on target ip:port and count the
    // words in input stream of \n delimited text (eg. generated by 'nc')
    // Note that no duplication in storage level only for running locally.
    // Replication necessary in distributed scenario for fault tolerance.
    //创建一个socket流，监听args(0)的args(1)端口输入的数据
    val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
    //flatMap是把将每一行使用空格做分解，words是个集合，存放单词
    val words = lines.flatMap(_.split(" "))
    //单词个数统计
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    //启动计算作业
    ssc.start()
    //等待结束，什么时候结束作业，即触发什么条件会让作业执行结束
    ssc.awaitTermination()
  }
}
// scalastyle:on println

16.2 测试运行

（1）修改CPU核数 因为我的Spark程序是在虚拟机中运行，如果虚拟机是单核，会导致NetworkWordCount卡住。所以需要修改虚拟机的核心数，这里修改node1节点为2个核即可。

这里写图片描述
这里写图片描述

（2）nc nc是netcat的命令。如果没有安装nc软件，可以通过下面命令安装

代码语言：javascript
复制
[root@node1 ~]# yum install nc
代码语言：javascript
复制
[root@node1 ~]# nc --help
...
  -l, --listen               Bind and listen for incoming connections
  -k, --keep-open            Accept multiple connections in listen mode
...

永久监听TCP端口9999

代码语言：javascript
复制
[root@node1 ~]# nc -lk 9999

（3）启动NetworkWordCount

代码语言：javascript
复制
[root@node1 ~]# run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999
17/11/01 09:26:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-------------------------------------------
Time: 1509542803000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542804000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542805000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542806000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542807000 ms
-------------------------------------------

（4）流处理

在9999端口不停输入数据

代码语言：javascript
复制
[root@node1 ~]# nc -lk 9999
hi
hello
how do you do!
i am hadron
java
hadoop
spark

可以看到NetworkWordCount 实时处理结果

代码语言：javascript
复制
[root@node1 ~]# run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999
17/11/01 09:26:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-------------------------------------------
Time: 1509542803000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542804000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542805000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542806000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542807000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542808000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542809000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542810000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542811000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542812000 ms
-------------------------------------------
(hi,1)

-------------------------------------------
Time: 1509542813000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542814000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542815000 ms
-------------------------------------------
(hello,1)

-------------------------------------------
Time: 1509542816000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542817000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542818000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542819000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542820000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542821000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542822000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542823000 ms
-------------------------------------------
(do!,1)
(how,1)
(you,1)
(do,1)

-------------------------------------------
Time: 1509542824000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542825000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542826000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542827000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542828000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542829000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542830000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542831000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542832000 ms
-------------------------------------------
(hadron,1)
(am,1)
(i,1)

-------------------------------------------
Time: 1509542833000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542834000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542835000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542836000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542837000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542838000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542839000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542840000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542841000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542842000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542843000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542844000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542845000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542846000 ms
-------------------------------------------
(java,1)

-------------------------------------------
Time: 1509542847000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542848000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542849000 ms
-------------------------------------------
(hadoop,1)

-------------------------------------------
Time: 1509542850000 ms
-------------------------------------------

-------------------------------------------
Time: 1509542851000 ms
-------------------------------------------
(spark,1)

-------------------------------------------
Time: 1509542852000 ms
本文参与 腾讯云自媒体分享计划，分享自作者个人站点/博客。
原始发表：2017-11-01 ，如有侵权请联系 cloudcommunity@tencent.com 删除
机器学习
spark
大数据

本文分享自 作者个人站点/博客 前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ，欢迎热爱写作的你一起参与！

机器学习
spark
大数据
评论
登录后参与评论
0 条评论
热度
最新
登录 后参与评论
推荐阅读
LV.
文章
0
获赞
0
目录
  • 16、 Spark Streaming入门实例NetworkWordCount
    • 16.1 源码解析
      • 16.2 测试运行
      相关产品与服务
      大数据
      全栈大数据产品，面向海量数据场景，帮助您 “智理无数，心中有数”！
      产品介绍
      精选特惠 用云无忧
      领券

      腾讯云开发者

      扫码关注腾讯云开发者

      扫码关注腾讯云开发者

      领取腾讯云代金券

      热门产品

      热门推荐

      更多推荐

      Copyright © 2013 - 2024 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

      深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

      腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

      问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档

      Copyright © 2013 - 2024 Tencent Cloud.

      All Rights Reserved. 腾讯云 版权所有

      登录 后参与评论