我正在构建一个小型Hadoop集群,它由两个节点组成:一个主节点+一个工作节点。我使用的是Hadoop (3.2)的最新版本,所有内容都由root用户执行。在安装过程中,我已经能够hdfs namenode -format
了。下一步是使用start-dfs.sh
启动HDFS守护进程。
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master]
bash v3.2+ is required. Sorry.
下面是日志中生成的日志:
$ journalctl --since "1 min ago"
-- Logs begin at Thu 2019-08-29 11:12:27 CEST, end at Thu 2019-08-29 11:46:40 CEST. --
Aug 29 11:46:40 master su[3329]: (to root) root on pts/0
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3334]: (to root) root on pts/0
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3389]: (to root) root on pts/0
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session closed for user root
当我使用Zsh时,我登录到bash控制台进行尝试。遗憾的是,我也得到了同样的结果。事实上,这个错误发生在所有sbin/start-*.sh
脚本中。然而,hadoop
和yarn
命令的工作原理就像一种魅力。
由于我在互联网上没有发现多少关于这个错误的信息,所以我在这里很高兴有任何建议!
其他技术细节
操作系统信息:
$ lsb_release -d
Description: Debian GNU/Linux 10 (buster)
$ uname -srm
Linux 4.19.0-5-amd64 x86_64
可用的Java版本(两者都尝试过):
$ update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
* 1 /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin/java 1081 manual mode
2 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
您可能对一些ENV变量感兴趣:
$ env
USER=root
LOGNAME=root
HOME=/root
PATH=/root/bin:/usr/local/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/usr/bin/zsh
TERM=rxvt-unicode
JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64
HADOOP_HOME=/usr/local/hadoop
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
ZSH=/root/.oh-my-zsh
Hadoop可执行文件的输出:
$ hadoop version
Hadoop 3.2.0
Source code repository https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf
Compiled by sunilg on 2019-01-08T06:08Z
Compiled with protoc 2.5.0
From source with checksum d3f0795ed0d9dc378e2c785d3668f39
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.0.jar
我的Zsh和Bash安装:
$ zsh --version
zsh 5.7.1 (x86_64-debian-linux-gnu)
$ bash --version
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnu)
# only available in a console using *bash*
$ echo ${BASH_VERSINFO[@]}
5 0 3 1 release x86_64-pc-linux-gnu
发布于 2019-08-29 14:23:30
TL;DR:使用不同的用户(例如hadoop)而不是根用户。
我找到了解决办法,但没有对正在发生的事情有深刻的理解。尽管我有多难过,但我找到的解决办法是:
与root用户一起运行:
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master_bis]
bash v3.2+ is required. Sorry
然后,我创建了一个hadoop用户,并在Hadoop安装(R/W access)上授予了这个用户特权。在与这个新用户登录之后,我有以下命令的输出,这给我带来了一些麻烦:
$ start-dfs.sh
Starting namenodes on [master]
Starting datanodes
Starting secondary namenodes [master_bis]
此外,我注意到在使用Java 11时,start-yarn.sh
创建的进程没有列在jps
的输出中。切换到Java 8解决了我的问题(不要忘记更新/etc/environment
和hadoop-env.sh
中的所有$JAVA_HOME
变量)。
成功\o/但是,我很高兴理解根用户为什么不能这样做。我知道使用根是一种坏习惯,但在实验环境中,有一个干净的“接近”的生产环境是我们不感兴趣的。请提供有关这方面的任何资料:)。
发布于 2021-05-22 09:00:31
试一试
chsh -s /bin/bash
将默认的shell更改为bash
https://stackoverflow.com/questions/57696217
复制相似问题