所以我有这段代码
dbs.foreach({
var map = scala.collection.mutable.Map[String, mutable.MutableList[String]]()
db =>
val resultList = getTables(hive, db)
map+=(db -> resultList)
})
这样做是循环遍历数据库列表,为每个数据库执行show tables in db
调用,然后将数据库->表添加到映射中。既然有大约5秒的等待时间来等待配置单元查询返回,那么如何同时完成这项工作呢?
更新代码--
def getAllTablesConcurrent(hive: JdbcHive, dbs: mutable.MutableList[String]): Map[String, mutable.MutableList[String]] = {
implicit val context:ExecutionContext = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(10))
val futures = dbs.map {
db =>
Future(db, getTables(hive, db))
}
val map = Await.result( Future.sequence(futures), Duration(10, TimeUnit.SECONDS) ).toMap
map
}
https://stackoverflow.com/questions/51564138
复制相似问题