[BrokerLoad问题排查] -Server asks us to fall back to SIMPLE auth
[BrokerLoad问题排查] -Server asks us to fall back to SIMPLE auth
报错: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections
PS: 已经修改Doris代码并提交到github, 如果不想看分析过程, 直接拉到底部, 查看PR.
问题描述
doris版本: 0.12.21
hadoop jar版本: 2.7.3
用户使用kerberos授权访问hdfs后, 再使用simple方式broker load就会失败. 相同的issue(可惜没有人处理): https://github.com/apache/incubator-doris/issues/2440, 我补充了重现步骤. 下面将一步步分享怎么发现、debug确认问题、处理问题
发现问题
混用kerberos和simple两种授权方式时, broker会出现导入失败. 并且有如下关键报错:
cause by: Failed
on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow
secure connections- 首先查看 Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections 这个报错是哪儿报的. 通过查看broker源代码, 发现报错文本在hadoop common包里面, 路径是: org.apache.hadoop.ipc.Client:
- 分析: 通过查看UserGroupInformation.isSecurityEnabled()源码和debug broker发现, 使用先使用Kerberos授权后, 再使用SIMPLE时, UserGroupInformation.isSecurityEnabled()中涉及到的一个全局变量authenticationMethod是错误的.
- 怀疑: 使用SIMPLE时, authenticationMethod还是Kerberos. 通过调试, 发现确实如此. 解决办法: 使用SIMPLE时, 主动初始化authenticationMethod, 覆盖之前设置的值
- 再次分析: fallbackAllowed变量是false时, 会爆当前的这个异常. 所以查找是否有办法控制fallbackAllowed值, 沿着fallbackAllowed被赋值的情况看, 找到如下代码:
通过google后, 发现CommonConfigurationKeys.IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY这个配置可以控制keerberos授权失败后, 降级退回到SIMPLE模式验证.
确认Broker代码逻辑
通过查看doris broker的源代码, 发现初始化hdfs client的代码路径是https://github.com/apache/incubator-doris/blob/master/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java#L173, 核心代码如下:
public BrokerFileSystem getDistributedFileSystem(String path, Map<String, String> properties) {
...
cachedFileSystem.putIfAbsent(fileSystemIdentity, new BrokerFileSystem(fileSystemIdentity));
fileSystem = cachedFileSystem.get(fileSystemIdentity);
fileSystem.getLock().lock();
try {
...
if (fileSystem.getDFSFileSystem() == null) {//fileSystem初始化过程
logger.info("could not find file system for path " + path + " create a new one");
Configuration conf = new HdfsConfiguration();
String tmpFilePath = null;
if (authentication.equals(AUTHENTICATION_KERBEROS)){
// Kerberos 授权初始化方式
conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
AUTHENTICATION_KERBEROS);
...
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab(principal, keytab);
...
}
...
FileSystem dfsFileSystem = null;
if (authentication.equals(AUTHENTICATION_SIMPLE) &&
properties.containsKey(USER_NAME_KEY) && !Strings.isNullOrEmpty(username)) {
// SIMPLE 授权初始化方式
UserGroupInformation ugi = UserGroupInformation.createRemoteUser(username);
...
} else {
dfsFileSystem = FileSystem.get(pathUri.getUri(), conf);
}
fileSystem.setFileSystem(dfsFileSystem);
}
return fileSystem;
} catch (Exception e) {
logger.error("errors while connect to " + path, e);
throw new BrokerException(TBrokerOperationStatusCode.NOT_AUTHORIZED, e);
} finally {
fileSystem.getLock().unlock();
}
}通过查看dfsFileSystem初始化过程来看, 主要需要注意添加注释的地方, 需要做如下改动
- 需要在simple模式初始化时, 主动添加配置,说明当前时SIMPLE模式验证
- kerberos模式初始化时, 添加CommonConfigurationKeys.IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH_ALLOWED_KEY配置, 使得kerberos授权失败后, 可以退回SIMPLE验证.
修改源码, 提交PR
https://github.com/apache/incubator-doris/pull/5412