无法解析重载方法“”groupByKey“”
无法解析重载方法“”groupByKey“”
提问于 2019-04-29 04:01:20
我正在尝试编译这段代码：
// Imports
import org.apache.spark.sql.{Row, SQLContext, SparkSession}
import org.apache.spark.sql.types._
import org.apache.spark.{SparkConf, SparkContext}
...
// Initialization
val conf = new SparkConf().setAppName("spark-test").setMaster("local")
val sc = new SparkContext(conf)
val sparkSession = SparkSession.builder.config(sc.getConf).getOrCreate()
import sparkSession.implicits._
...
val sqlContext = sparkSession
val dfPlayersT = sqlContext.createDataFrame(nPlayer,schemaN)
dfPlayersT.createOrReplaceTempView("tPlayers")
val dfPlayers = sqlContext.sql("select age-min_age as exp,tPlayers.* from
tPlayers join (select name,min(age)as min_age from tPlayers group by name)
as t1 on tPlayers.name=t1.name order by tPlayers.name, exp ")
val pStats = dfPlayers.sort(dfPlayers("name"),dfPlayers("exp").asc)
.map(x=>(x.getString(1),(x.getDouble(50),x.getDouble(40),x.getInt(2),
x.getInt(3),Array(x.getDouble(31),x.getDouble(32),x.getDouble(33),
x.getDouble(34),x.getDouble(35),x.getDouble(36),x.getDouble(37),
x.getDouble(38),x.getDouble(39)),x.getInt(0))))
.groupByKey() // Error但是得到了一个错误：
Error:(217, 57) overloaded method value groupByKey with alternatives:
[K](func: org.apache.spark.api.java.function.MapFunction[(String, (Double, Double, Int, Int, Array[Double], Int)),K], encoder: org.apache.spark.sql.Encoder[K])org.apache.spark.sql.KeyValueGroupedDataset[K,(String, (Double, Double, Int, Int, Array[Double], Int))] <and>
[K](func: ((String, (Double, Double, Int, Int, Array[Double], Int))) => K)(implicit evidence$4: org.apache.spark.sql.Encoder[K])org.apache.spark.sql.KeyValueGroupedDataset[K,(String, (Double, Double, Int, Int, Array[Double], Int))]
cannot be applied to ()
x.getDouble(38),x.getDouble(39)),x.getInt(0)))).groupByKey()这是我的build.sbt文件：
name := "ScalaHello"
version := "0.1"
scalaVersion := "2.12.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.2"
libraryDependencies += "org.apache.spark" %% "spark-catalyst" % "2.4.2"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.2"我想问题出在sparkSession的初始化上，但是我不知道哪里出了问题。
回答 1
Stack Overflow用户
发布于 2019-04-29 05:04:51
应该是吗？
.groupByKey(_._1)或
.groupByKey(_._2._1) 或
.groupByKey(_._2._2)..。
或
.groupByKey(_._2._6) 页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/55894126
复制相关文章
相似问题