1/运行mr程序出错
connecting to resoucemanager
retrying ....
retrying .....
原因是没有启动yarn或者启动失败,正常启动yarn后,又报下面的错误:
System times on machines may be out of sync,check system time and time zones
原因是各个节点的时间没有统一,使用NTP
统一时间,或是使用secureCRT
自带的
send chat to all sessions
菜单统一给节点执行sudo date -s 2018-03-20 10:53
2/初始化工作目录结构
hdfs namenode -format
只是初始化了namenode
的工作目录
而datanode
的工作目录是在datanode
启动(执行startdfs.sh
命令)后自己初始化的
使用tree
命令查看目录结构(如果没有这个命令,要先安装sudo yum install -y tree
)
hdpdata 树结构.png
3/datanode不被namenode识别的问题 namenode在format初始化的时候会形成两个标识: blockPoolId: clusterId:
新的datanode加入时,会获取这两个标识作为自己工作目录中的标识。
一旦namenode重新format后,namenode的身份标识已变,而datanode如果依然 持有原来的id,就不会被namenode识别。
namenode.png
datanode.png
4/datanode下线后多久看到效果 datanode不是一下线就会被namenode认定为下线的,有一个超时时间
5/关于副本数量的问题
副本数由客户端的参数dfs.replication
决定(优先级: API conf.set
> 自定义配置文件 > jar包中的hdfs-default.xml)
6/往集群提交数据时,hdfs dfs -put employee.txt /person
,关键字是Bad connect ack with firstBadLink as 192.168.92.112:50010
18/04/17 23:06:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/17 23:06:52 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 192.168.92.112:50010
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1401)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1302)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:536)
18/04/17 23:06:52 INFO hdfs.DFSClient: Abandoning BP-2054255948-192.168.92.111-1523928758660:blk_1073741825_1001
18/04/17 23:06:52 INFO hdfs.DFSClient: Excluding datanode 192.168.92.112:50010
18/04/17 23:06:52 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 192.168.92.113:50010
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1401)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1302)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:536)
18/04/17 23:06:52 INFO hdfs.DFSClient: Abandoning BP-2054255948-192.168.92.111-1523928758660:blk_1073741826_1002
18/04/17 23:06:52 INFO hdfs.DFSClient: Excluding datanode 192.168.92.113:50010
原因是集群中其他节点的防火墙没有关掉:
service iptables status
service iptables status
chkconfig iptables off