温馨提示:要看高清无码套图,请使用手机打开并单击图片放大查看。
1.文档编写目的
使用Hue可以方便的通过界面制定Oozie的工作流,支持Hive、Pig、Spark、Java、Sqoop、MapReduce、Shell等等。Spark?那能不能支持Spark2的呢,接下来本文章就主要讲述如何使用Hue创建Spark1和Spark2的Oozie工作流。
1.添加Spark2到Oozie的Share-lib
2.创建Spark2的Oozie工作流
3.创建Spark1的Oozie工作流
4.总结
1.CM和CDH版本为5.11.2
2.操作系统:RedHat7.2
3.采用sudo权限的ec2-user用户操作
2.Oozie共享库添加Spark2
1.查看当前Oozie的share-lib共享库HDFS目录
ec2-user@ip-172-31-22-86jars$ oozie admin -ooziehttp://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie-sharelibupdate
ShareLib update status
host =http://ec2-54-169-0-36.ap-southeast-1.compute.amazonaws.com:11000/oozie
status = Successful
sharelibDirOld =hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424
sharelibDirNew =hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424
ec2-user@ip-172-31-22-86jars$
2.在Oozie的/user/oozie/share/lib/lib_20170921070424创建spark2目录
[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs -mkdir /user/oozie/share/lib/lib_20170921070424/spark2
3.向spark2目录添加spark2的jars和oozie-sharelib-spark*.jar
[ec2-user@ip-172-31-22-86 jars]$ pwd
/opt/cloudera/parcels/SPARK2/lib/spark2/jars
[ec2-user@ip-172-31-22-86 jars]$ sudo -u hdfs hadoop fs -put *.jar /user/oozie/share/lib/lib_20170921070424/spark2
[ec2-user@ip-172-31-22-86 spark]$ pwd
/opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark
[ec2-user@ip-172-31-22-86 spark]$ sudo -u hdfs hadoop fs -put oozie-sharelib-spark*.jar /user/oozie/share/lib/lib_20170921070424/spark2
4.修改目录属主及权限
[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs –chown -R oozie:oozie /user/oozie/share/lib/lib_20170921070424/spark2
[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs –chmod -R 775 /user/oozie/share/lib/lib_20170921070424/spark2
5.更新Oozie的share-lib
[ec2-user@ip-172-31-22-86 spark]$ oozie admin -oozie http://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie -sharelibupdate
[ShareLib update status]
host = http://ec2-54-169-0-36.ap-southeast-1.compute.amazonaws.com:11000/oozie
status = Successful
sharelibDirOld = hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424
sharelibDirNew = hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424
[ec2-user@ip-172-31-22-86 spark]$
6.确认spark2已经添加到共享库
[ec2-user@ip-172-31-22-86 spark]$ oozie admin -oozie http://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie -shareliblist
[Available ShareLib]
spark2
oozie
hive
distcp
hcatalog
sqoop
mapreduce-streaming
spark
hive2
pig
[ec2-user@ip-172-31-22-86 spark]$
3.创建Spark2的Oozie工作流
1.登录Hue,创建Oozie工作流
2.进入WorkSpace
点击lib
在命令行将Spark2自带的example例子上传到/user/hue/oozie/workspaces/hue-oozie-1507832616.28/lib目录
[ec2-user@ip-172-31-22-86 jars]$ pwd
/opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars
[ec2-user@ip-172-31-22-86 jars]$ sudo -u hdfs hadoop fs -put spark-examples_2.11-2.1.0.cloudera1.jar /user/hue/oozie/workspaces/hue-oozie-1507832616.28/lib
[ec2-user@ip-172-31-22-86 jars]$
3.添加Spark2任务
设置使用Spark2,否则默认使用的Spark1
完成配置,点击保存
4.保存完成后,点击运行测试是否正常
运行成功
4.创建Spark1的Oozie工作流
1.创建Oozie工作流
进入WorkSpace
2.将Spark1的example的jar包上传至该WorkSpace的lib目录
[ec2-user@ip-172-31-22-86 lib]$ pwd
/opt/cloudera/parcels/CDH/lib/spark/examples/lib
[ec2-user@ip-172-31-22-86 lib]$ sudo -u hdfs hadoop fs -put spark-examples-1.6.0-cdh5.11.2-hadoop2.6.0-cdh5.11.2.jar /user/hue/oozie/workspaces/hue-oozie-1507860705.24/lib
[ec2-user@ip-172-31-22-86 lib]$
3.添加Spark1的任务到Oozie
4.保存Oozie,然后点击提交
运行成功
5.常见问题
1.在使用Hue创建Spark2的Oozie工作流时运行异常
2017-10-16 23:20:07,086 WARN org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ec2-54-179-152-169.ap-southeast-1.compute.amazonaws.com] USER[admin] GROUP[-] TOKEN[] APP[MyFirstSpark2] JOB[0000000-171016230402705-oozie-oozi-W] ACTION[0000000-171016230402705-oozie-oozi-W@spark-1411] Launcher exception: Exception when registering SparkListener
org.apache.spark.SparkException: Exception when registering SparkListener
at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2193)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:562)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:178)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:90)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: com.cloudera.spark.lineage.ClouderaNavigatorListener
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2159)
at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2156)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2156)
... 34 more
解决方法:
保存配置然后重启相应服务。
6.总结
使用Oozie创建Spark的工作流,如果需要运行Spark2的作业则需要向Oozie的Share-lib库中添加Spark2的支持,并在创建Spark2作业的时候需要指定Share-lib为spark2(Oozie默认的支持Spark1),否则会运行失败报Spark2的类找不到异常。
参考文档:
醉酒鞭名马,少年多浮夸! 岭南浣溪沙,呕吐酒肆下!挚友不肯放,数据玩的花! 温馨提示:要看高清无码套图,请使用手机打开并单击图片放大查看。
推荐关注Hadoop实操,第一时间,分享更多Hadoop干货,欢迎转发和分享。
原创文章,欢迎转载,转载请注明:转载自微信公众号Hadoop实操