前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >flink读取kafka报shaded ByteArrayDeserializer异常

flink读取kafka报shaded ByteArrayDeserializer异常

原创
作者头像
平常心
修改2021-07-07 10:16:52
3.7K1
修改2021-07-07 10:16:52
举报
文章被收录于专栏:个人总结系列个人总结系列

异常描述

代码语言:javascript
复制
	value.deserializer = class org.apache.flink.kafka.shaded.org.apache.kafka.common.serialization.ByteArrayDeserializer

2021-07-06 17:55:20,519 INFO  org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl [] - Source: Custom Source -> Filter -> Sink: Unnamed (1/1)#223 discarding 0 drained requests
2021-07-06 17:55:20,519 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: Custom Source -> Filter -> Sink: Unnamed (1/1)#223 (0326263def4826d9563fef3519fed530) switched from RUNNING to FAILED.
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:820) ~[flink-app-jar.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666) ~[flink-app-jar.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646) ~[flink-app-jar.jar:?]
	at org.apache.flink.streaming.connectors.kafka.internals.KafkaPartitionDiscoverer.initializeConnections(KafkaPartitionDiscoverer.java:55) ~[flink-app-jar.jar:?]
	at org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.open(AbstractPartitionDiscoverer.java:94) ~[flink-app-jar.jar:?]
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:551) ~[flink-app-jar.jar:?]
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) ~[flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) ~[flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:428) ~[flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:543) ~[flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93) ~[flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:533) ~[flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573) ~[flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) [flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) [flink-dist_2.11-1.12-vvr-3.0.2-SNAPSHOT.jar:1.12-vvr-3.0.2-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.apache.kafka.common.KafkaException: class org.apache.flink.kafka.shaded.org.apache.kafka.common.serialization.ByteArrayDeserializer is not an instance of org.apache.kafka.common.serialization.Deserializer
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:374) ~[flink-app-jar.jar:?]
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:392) ~[flink-app-jar.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:713) ~[flink-app-jar.jar:?]
	... 15 more

编写的代码使用的pom是使用

代码语言:javascript
复制
flink-connector-kafka_2.11-1.12.0.jar

对应改jar依赖的是原生的kafka内容,不是shaded内容

但是在flink环境下面,已经提供了

代码语言:javascript
复制
flink-sql-connector-kafka_2.11-1.12.0.jar

可以看到提供的内容,对应进行maven pom文件去掉flink-connector-kafka就可以了,引用flink-sql-connector就可以解决这个问题。

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
相关产品与服务
大数据
全栈大数据产品,面向海量数据场景,帮助您 “智理无数,心中有数”!
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档