如何在提交步骤时启用flink访问emr上的/var/lib/flink
如何在提交步骤时启用flink访问emr上的/var/lib/flink
提问于 2022-09-22 02:58:18
我在发布标签上:EMR-6.7.0 Hadoop发行版:Amazon3.2.1应用程序:Livy 0.7.1，ZooKeeper 3.5.7，Flink 1.14.2
我也在每个emr-6.X版本中尝试过这一点，并得到了相同的错误。
我试图提交一个简单的flink步骤，使用添加一个步骤，在电子病历上使用args，flink-session -d。
如果出现以下错误，则会失败。
2022-09-22 02:06:08,883 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting YarnSessionClusterEntrypoint down with application status FAILED. Diagnostics org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:254)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:193)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:617)
at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:90)
Caused by: java.nio.file.AccessDeniedException: /var/lib/flink
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.apache.flink.runtime.rest.RestServerEndpoint.checkAndCreateUploadDir(RestServerEndpoint.java:583)
at org.apache.flink.runtime.rest.RestServerEndpoint.createUploadDir(RestServerEndpoint.java:566)
at org.apache.flink.runtime.rest.RestServerEndpoint.<init>(RestServerEndpoint.java:126)
at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.<init>(WebMonitorEndpoint.java:211)
at org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.<init>(DispatcherRestEndpoint.java:68)
at org.apache.flink.runtime.rest.SessionRestEndpointFactory.createRestEndpoint(SessionRestEndpointFactory.java:62)
at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:161)
... 9 more
.
2022-09-22 02:06:08,886 INFO org.apache.flink.runtime.blob.BlobServer
[] - Stopped BLOB server at 0.0.0.0:35111
2022-09-22 02:06:08,888 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service.
2022-09-22 02:06:08,891 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service.
2022-09-22 02:06:08,931 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon.
2022-09-22 02:06:08,932 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports.
2022-09-22 02:06:08,943 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon.
2022-09-22 02:06:08,943 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports.
2022-09-22 02:06:08,956 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.
2022-09-22 02:06:08,963 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.
2022-09-22 02:06:08,979 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopped Akka RPC service.
2022-09-22 02:06:08,987 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopped Akka RPC service.
2022-09-22 02:06:08,988 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Could not start cluster entrypoint YarnSessionClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnSessionClusterEntrypoint.
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:216) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:617) [flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:90) [flink-dist_2.12-1.14.2.jar:1.14.2]
Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:254) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:193) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_342]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_342]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.2.1-amzn-7.jar:?]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
... 2 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/flink
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_342]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_342]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_342]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_342]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_342]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_342]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_342]
at org.apache.flink.runtime.rest.RestServerEndpoint.checkAndCreateUploadDir(RestServerEndpoint.java:583) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.RestServerEndpoint.createUploadDir(RestServerEndpoint.java:566) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.RestServerEndpoint.<init>(RestServerEndpoint.java:126) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.<init>(WebMonitorEndpoint.java:211) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint.<init>(DispatcherRestEndpoint.java:68) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.rest.SessionRestEndpointFactory.createRestEndpoint(SessionRestEndpointFactory.java:62) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:161) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:254) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:193) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_342]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_342]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.2.1-amzn-7.jar:?]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190) ~[flink-dist_2.12-1.14.2.jar:1.14.2]
... 2 more它说的主要原因是
由: java.nio.file.AccessDeniedException: /var/lib/flink引起
我该怎么解决这个问题？我的想法快用完了。
回答 1
Stack Overflow用户
发布于 2022-11-15 02:43:25
这可能是因为您将任务节点用于Flink集群。任务节点没有创建/var/lib/flink文件夹，因此当Flink作业被提交到任务节点时，它将失败。
遗憾的是，在文档中没有明确提到这一点。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/73808870
复制相关文章
相似问题
腾讯云开发者
