首页
学习
活动
专区
工具
TVP
发布
社区首页 >问答首页 >使用sbt for Spark打包JAR文件

使用sbt for Spark打包JAR文件
EN

Stack Overflow用户
提问于 2018-10-08 01:18:10
回答 1查看 834关注 0票数 0

我正在关注他们的入门页面:https://spark.apache.org/docs/latest/quick-start.html,在倒数第二个代码片段中,我必须使用sbt将我的文件打包到.jar中

我的build.sbt:

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" % "spark-sql" % "2.3.2"

我的SimpleApp.scala:

/* SimpleApp.scala */
import org.apache.spark.sql.SparkSession

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/usr/local/spark/README.md" // Should be some file on your system
    val spark = SparkSession.builder.appName("Simple Application").getOrCreate()
    val logData = spark.read.textFile(logFile).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println(s"Lines with a: $numAs, Lines with b: $numBs")
    spark.stop()
  }
}

我将文件放在正确的目录中,比如requested,并从/usr/local/spark/examples/ where build运行sbt package

然后,我继续得到这个非常长的错误:

test@test-ThinkPad-X230:/usr/local/spark/examples$ sbt package
[info] Loading project definition from /usr/local/spark/examples/project
[info] Loading settings for project examples from build.sbt ...
[info] Set current project to Simple Project (in build file:/usr/local/spark/examples/)
[info] Updating ...
[info] downloading https://repo1.maven.org/maven2/org/apache/avro/avro/1.7.7/avro-1.7.7.jar ...
[info]  [SUCCESSFUL ] org.apache.avro#avro;1.7.7!avro.jar (322ms)
[info] Done updating.
[info] Compiling 188 Scala sources and 125 Java sources to /usr/local/spark/examples/target/scala-2.11/classes ...
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalFileLR.scala:23:8: not found: object breeze
[error] import breeze.linalg.{DenseVector, Vector}
[error]        ^
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalFileLR.scala:39:19: not found: type DenseVector
[error]     DataPoint(new DenseVector(nums.slice(1, D + 1)), nums(0))
[error]                   ^
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalFileLR.scala:60:13: not found: value DenseVector
[error]     val w = DenseVector.fill(D) {2 * rand.nextDouble - 1}
[error]             ^
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalFileLR.scala:65:22: not found: value DenseVector
[error]       val gradient = DenseVector.zeros[Double](D)
[error]                      ^
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalKMeans.scala:26:8: not found: object breeze
[error] import breeze.linalg.{squaredDistance, DenseVector, Vector}
[error]        ^
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalKMeans.scala:42:27: not found: type DenseVector
[error]   def generateData: Array[DenseVector[Double]] = {
[error]                           ^
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalKMeans.scala:43:32: not found: type DenseVector
[error]     def generatePoint(i: Int): DenseVector[Double] = {
[error]                                ^
[error] /usr/local/spark/examples/src/main/scala/org/apache/spark/examples/LocalKMeans.scala:44:7: not found: value DenseVector
[error]       DenseVector.fill(D) {rand.nextDouble * R}
[error]       ^

而且还在继续。我说不出我做错了什么。

EN

回答 1

Stack Overflow用户

发布于 2018-10-08 01:41:55

发现问题: build.sbt应该在/spark/中,而不是在/spark/examples

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/52690986

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档