使用gradle打包spring boot jar以供spark提交
使用gradle打包spring boot jar以供spark提交
提问于 2020-11-17 11:08:09
当我在spark上运行由gradle构建的spring boot打包的JAR时，在主类上面对一个ClassNotFoundException：
spark2-submit --class com.test.DriverMain test.jar ...我正在使用org.springframework.boot:spring-boot-gradle-plugin:2.2.0.RELEASE
我还尝试删除主类名称的-- ClassCastException引用，在类上传递，但后来在运行过程中遇到了很多类：
java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.functions$$anonfun$14.f$10 of type org.apache.spark.sql.api.java.UDF1 in instance of org.apache.spark.sql.functions$$anonfun$14回答 1
Stack Overflow用户
回答已采纳
发布于 2020-11-17 11:08:09
在spark文档(https://spark.apache.org/docs/latest/submitting-applications.html)上，它有sbt和maven插件的链接，介绍了如何打包一个兼容的jar以便在spark上提交(但是没有gradle)。
spring-boot-gradle-plugin的jar结构输出如下：
test.jar
- BOOT-INF
- lib
- ... jars dependencies
- classes
- com \ test \ ...
- org
- springframework \ boot \loader \ ...
- META-INF
- MANIFEST.MF预期的结构类似于：
test.jar
- com \ test \ ...
- jar dependencies in package and classes format like org \ springframework \ data \ jpa \ ...下面是gradle的工作代码片段：
buildscript {
repositories repos
dependencies {
classpath com.github.jengelman.gradle.plugins:shadow:5.2.0
}
}
...
apply plugin: 'com.github.johnrengelman.shadow'
// this way I won't be needing any spring boot gradle related plugin for packaging and dependency mgt
dependencies {
implementation platform("org.springframework.boot:spring-boot-dependencies:2.2.0.RELEASE")
...
}
// set to false, else it will be packaged twice, one for the shaded jar, one with the normal jar
jar {
enabled = false
}
// creates the spring boot shaded jar
import com.github.jengelman.gradle.plugins.shadow.transformers.PropertiesFileTransformer
shadowJar {
zip64 true
mergeServiceFiles()
append 'META-INF/spring.handlers'
append 'META-INF/spring.schemas'
append 'META-INF/spring.tooling'
transform(PropertiesFileTransformer) {
paths = ['META-INF/spring.factories' ]
mergeStrategy = "append"
}
archiveFileName = "test-${version}.jar"
}
// shaded jar will be built whenever jar is being invoked
jar.dependsOn(shadowJar)参考链接：https://github.com/spring-projects/spring-boot/issues/1828#issuecomment-231104288
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/64868767
复制相关文章
相似问题
腾讯云开发者
