前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >Hadoop 在 Centos7 下的单机布署(一). Standalone Operation

Hadoop 在 Centos7 下的单机布署(一). Standalone Operation

作者头像
franket
发布2021-08-12 09:40:03
5190
发布2021-08-12 09:40:03
举报
文章被收录于专栏:技术杂记

前言

Apache Hadoop 是一个专注于可靠,弹性,分布式计算框架的开源软件项目

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

主要包含以下几个模块:

  • Hadoop Common : 支持其它 hadoop 模块的通用工具
  • Hadoop Distributed File System : 简称 HDFS , 给应用数据提供高吞吐性能的分布式文件系统
  • Hadoop YARN : 工作调度与集群资源管理的框架
  • Hadoop MapReduce : 大数据集的并行处理系统

Hadoop 生态圈中的其它项目可以参考 Hadoop-related projects

Tip: 当前的最新稳定版为 Hadoop Release 2.8.1 发布于 08 June, 2017

这里根据官方的文档给出最新版 hadoop 在 Centos7 下的单机布署方案,详细可以参考 Setting up a Single Node Cluster

此文章借鉴了 Hadoop Wiki 中的部分内容和 Running Hadoop on Ubuntu Linux (Single-Node Cluster) 中部分操作


概要


系统环境

代码语言:javascript
复制
[root@much ~]# hostnamectl 
   Static hostname: much
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 33dc28f7e76c4903ad9b603b77e29a7c
           Boot ID: 8cfd2a8aec4f4235b4776f7cd2fdfdb1
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-514.21.1.el7.x86_64
      Architecture: x86-64
[root@much ~]# ip a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:e3:df:87 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 84190sec preferred_lft 84190sec
    inet6 fe80::2bb7:5b3:9584:d8eb/64 scope link 
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:d3:ec:e7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.207/24 brd 192.168.56.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fed3:ece7/64 scope link 
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:16:5e:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:16:5e:11 brd ff:ff:ff:ff:ff:ff
[root@much ~]# uname -a 
Linux much 3.10.0-514.21.1.el7.x86_64 #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@much ~]# 

目标

  • 构建一个 hadoop 的单点伪集群
  • hdfs 的基本操作

基础依赖

  • Java
  • ssh

由于 Hadoop 是使用 Java 语言开发出来的项目,所以 Java 运行环境是必要的, Hadoop 支持的 Java 版本可以参考 Hadoop Java Versions

之所以需要 sshd , 是因为 Hadoop 是通过 ssh 执行脚本来管理远程节点


安装 Java

代码语言:javascript
复制
[root@much ~]# rpm  -qa | grep epel 
[root@much ~]# yum  list all | grep -i epel 
epel-release.noarch                        7-9                         extras   
[root@much ~]# yum  install epel-release.noarch 
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.yun-idc.com
 * c7-media: 
 * extras: mirrors.163.com
 * updates: centos.ustc.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-9 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================
 Package                  Arch               Version           Repository          Size
========================================================================================
Installing:
 epel-release             noarch             7-9               extras              14 k

Transaction Summary
========================================================================================
Install  1 Package

Total download size: 14 k
Installed size: 24 k
Is this ok [y/d/N]: y
Downloading packages:
epel-release-7-9.noarch.rpm                                      |  14 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : epel-release-7-9.noarch                                              1/1 
  Verifying  : epel-release-7-9.noarch                                              1/1 

Installed:
  epel-release.noarch 0:7-9                                                             

Complete!
[root@much ~]# 
[root@much ~]# rpm  -qa | grep epel 
epel-release-7-9.noarch
[root@much ~]# 
[root@much ~]# yum list all | grep -i java | grep -i  jdk
java-1.7.0-openjdk.x86_64               1:1.7.0.141-2.6.10.1.el7_3     @updates 
java-1.7.0-openjdk-headless.x86_64      1:1.7.0.141-2.6.10.1.el7_3     @updates 
java-1.8.0-openjdk.x86_64               1:1.8.0.131-3.b12.el7_3        @updates 
java-1.8.0-openjdk-headless.x86_64      1:1.8.0.131-3.b12.el7_3        @updates 
java-1.6.0-openjdk.x86_64               1:1.6.0.41-1.13.13.1.el7_3     updates  
java-1.6.0-openjdk-demo.x86_64          1:1.6.0.41-1.13.13.1.el7_3     updates  
java-1.6.0-openjdk-devel.x86_64         1:1.6.0.41-1.13.13.1.el7_3     updates  
java-1.6.0-openjdk-javadoc.x86_64       1:1.6.0.41-1.13.13.1.el7_3     updates  
java-1.6.0-openjdk-src.x86_64           1:1.6.0.41-1.13.13.1.el7_3     updates  
java-1.7.0-openjdk-accessibility.x86_64 1:1.7.0.141-2.6.10.1.el7_3     updates  
java-1.7.0-openjdk-demo.x86_64          1:1.7.0.141-2.6.10.1.el7_3     updates  
java-1.7.0-openjdk-devel.x86_64         1:1.7.0.141-2.6.10.1.el7_3     updates  
java-1.7.0-openjdk-javadoc.noarch       1:1.7.0.141-2.6.10.1.el7_3     updates  
java-1.7.0-openjdk-src.x86_64           1:1.7.0.141-2.6.10.1.el7_3     updates  
java-1.8.0-openjdk.i686                 1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk.x86_64               1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-accessibility.x86_64 1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-accessibility-debug.x86_64
java-1.8.0-openjdk-debug.i686           1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-debug.x86_64         1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-debuginfo.i686       1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-debuginfo.x86_64     1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-demo.x86_64          1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-demo-debug.x86_64    1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-devel.i686           1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-devel.x86_64         1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-devel-debug.i686     1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-devel-debug.x86_64   1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-headless.i686        1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-headless.x86_64      1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-headless-debug.i686  1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-headless-debug.x86_64
java-1.8.0-openjdk-javadoc.noarch       1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-javadoc-debug.noarch 1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-javadoc-zip.noarch   1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-javadoc-zip-debug.noarch
java-1.8.0-openjdk-src.x86_64           1:1.8.0.141-1.b16.el7_3        updates  
java-1.8.0-openjdk-src-debug.x86_64     1:1.8.0.141-1.b16.el7_3        updates  
ldapjdk-javadoc.noarch                  4.18-16.el7_3                  updates  
[root@much ~]# yum install java-1.8.0-openjdk.x86_64  java-1.8.0-openjdk-devel.x86_64  java-1.8.0-openjdk-headless.x86_64  
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.yun-idc.com
 * c7-media: 
 * epel: mirrors.ustc.edu.cn
 * extras: mirrors.163.com
 * updates: centos.ustc.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.131-3.b12.el7_3 will be updated
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.141-1.b16.el7_3 will be an update
---> Package java-1.8.0-openjdk-devel.x86_64 1:1.8.0.141-1.b16.el7_3 will be installed
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.131-3.b12.el7_3 will be updated
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.141-1.b16.el7_3 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================
 Package                        Arch      Version                      Repository  Size
========================================================================================
Installing:
 java-1.8.0-openjdk-devel       x86_64    1:1.8.0.141-1.b16.el7_3      updates    9.7 M
Updating:
 java-1.8.0-openjdk             x86_64    1:1.8.0.141-1.b16.el7_3      updates    234 k
 java-1.8.0-openjdk-headless    x86_64    1:1.8.0.141-1.b16.el7_3      updates     31 M

Transaction Summary
========================================================================================
Install  1 Package
Upgrade  2 Packages

Total download size: 41 M
Is this ok [y/d/N]: y
Downloading packages:
updates/7/x86_64/prestodelta                                     | 954 kB  00:00:00     
Delta RPMs reduced 234 k of updates to 58 k (75% saved)
(1/3): java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3_1.8.0.141-1.b16. |  58 kB  00:00:00     
(2/3): java-1.8.0-openjdk-devel-1.8.0.141-1.b16.el7_3.x86_64.rpm | 9.7 MB  00:00:03     
(3/3): java-1.8.0-openjdk-headless-1.8.0.141-1.b16.el7_3.x86_64. |  31 MB  00:00:03     
----------------------------------------------------------------------------------------
Total                                                       10 MB/s |  41 MB  00:04     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 1:java-1.8.0-openjdk-headless-1.8.0.141-1.b16.el7_3.x86_64           1/5 
warning: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/lib/security/java.security created as /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/lib/security/java.security.rpmnew
  Updating   : 1:java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64                    2/5 
  Installing : 1:java-1.8.0-openjdk-devel-1.8.0.141-1.b16.el7_3.x86_64              3/5 
  Cleanup    : 1:java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64                    4/5 
  Cleanup    : 1:java-1.8.0-openjdk-headless-1.8.0.131-3.b12.el7_3.x86_64           5/5 
  Verifying  : 1:java-1.8.0-openjdk-headless-1.8.0.141-1.b16.el7_3.x86_64           1/5 
  Verifying  : 1:java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64                    2/5 
  Verifying  : 1:java-1.8.0-openjdk-devel-1.8.0.141-1.b16.el7_3.x86_64              3/5 
  Verifying  : 1:java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64                    4/5 
  Verifying  : 1:java-1.8.0-openjdk-headless-1.8.0.131-3.b12.el7_3.x86_64           5/5 

Installed:
  java-1.8.0-openjdk-devel.x86_64 1:1.8.0.141-1.b16.el7_3                               

Updated:
  java-1.8.0-openjdk.x86_64 1:1.8.0.141-1.b16.el7_3                                     
  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.141-1.b16.el7_3                            

Complete!
[root@much ~]# 

检查 JAVA_HOME

代码语言:javascript
复制
[root@much ~]# java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
[root@much ~]# which java 
/usr/bin/java
[root@much ~]# ll /usr/bin/java
lrwxrwxrwx. 1 root root 22 8月   6 23:55 /usr/bin/java -> /etc/alternatives/java
[root@much ~]# ll /etc/alternatives/java
lrwxrwxrwx. 1 root root 73 8月   6 23:55 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java
[root@much ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java
-rwxr-xr-x. 1 root root 7336 7月  21 06:37 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java
[root@much ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/
total 4
drwxr-xr-x. 2 root root 4096 8月   6 23:55 bin
drwxr-xr-x. 3 root root  132 8月   6 23:55 include
drwxr-xr-x. 4 root root   28 8月   6 23:55 jre
drwxr-xr-x. 3 root root  144 8月   6 23:55 lib
drwxr-xr-x. 2 root root  204 8月   6 23:55 tapset
[root@much ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/bin/
total 420
-rwxr-xr-x. 1 root root   7400 7月  21 06:37 appletviewer
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 extcheck
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 idlj
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 jar
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 jarsigner
-rwxr-xr-x. 1 root root   7336 7月  21 06:37 java
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 javac
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 javadoc
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 javah
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 javap
-rwxr-xr-x. 1 root root   2806 7月  21 06:01 java-rmi.cgi
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 jcmd
-rwxr-xr-x. 1 root root   7416 7月  21 06:37 jconsole
-rwxr-xr-x. 1 root root   7400 7月  21 06:37 jdb
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 jdeps
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 jhat
-rwxr-xr-x. 1 root root   7448 7月  21 06:37 jinfo
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 jjs
-rwxr-xr-x. 1 root root   7448 7月  21 06:37 jmap
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 jps
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 jrunscript
-rwxr-xr-x. 1 root root   7408 7月  21 06:37 jsadebugd
-rwxr-xr-x. 1 root root   7456 7月  21 06:37 jstack
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 jstat
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 jstatd
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 keytool
-rwxr-xr-x. 1 root root   7400 7月  21 06:37 native2ascii
-rwxr-xr-x. 1 root root   7456 7月  21 06:37 orbd
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 pack200
-rwxr-xr-x. 1 root root   7400 7月  21 06:37 policytool
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 rmic
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 rmid
-rwxr-xr-x. 1 root root   7400 7月  21 06:37 rmiregistry
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 schemagen
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 serialver
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 servertool
-rwxr-xr-x. 1 root root   7464 7月  21 06:37 tnameserv
-rwxr-xr-x. 1 root root 103424 7月  21 06:37 unpack200
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 wsgen
-rwxr-xr-x. 1 root root   7392 7月  21 06:37 wsimport
-rwxr-xr-x. 1 root root   7384 7月  21 06:37 xjc
[root@much ~]#

确保 ssh rsync 已经安装

代码语言:javascript
复制
[root@much ~]# yum install openssh rsync 
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.yun-idc.com
 * c7-media: 
 * epel: mirrors.ustc.edu.cn
 * extras: mirrors.163.com
 * updates: centos.ustc.edu.cn
Package openssh-6.6.1p1-35.el7_3.x86_64 already installed and latest version
Package rsync-3.0.9-17.el7.x86_64 already installed and latest version
Nothing to do
[root@much ~]#

关闭 selinux

代码语言:javascript
复制
[root@much ~]# getenforce 
Enforcing
[root@much ~]# setenforce  0 
[root@much ~]# getenforce 
Permissive
[root@much ~]#

创建 hadoop 用户 

代码语言:javascript
复制
[root@much ~]# useradd hadoop 
[root@much ~]# passwd hadoop
Changing password for user hadoop.
New password: 
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: 
passwd: all authentication tokens updated successfully.
[root@much ~]# 

配置用户免密码登录

代码语言:javascript
复制
[root@much ~]# su - hadoop 
[hadoop@much ~]$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
cb:02:3a:db:ce:eb:e3:d0:72:be:54:84:8f:cb:d7:c1 hadoop@much
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|     .           |
|    . .          |
|     + .         |
|    o o E        |
|   + + o o       |
|  = * o +        |
|   @.. .         |
|  .+X+           |
+-----------------+
[hadoop@much ~]$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
[hadoop@much ~]$ chmod 600 $HOME/.ssh/authorized_keys
[hadoop@much ~]$ ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is 0c:52:20:0a:00:e3:1a:5d:c6:fc:79:b3:e8:6e:d6:f1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Last login: Mon Aug  7 00:04:13 2017
[hadoop@much ~]$ exit
logout
Connection to localhost closed.
[hadoop@much ~]$ ssh localhost
Last login: Mon Aug  7 00:05:48 2017 from localhost
[hadoop@much ~]$ exit
logout
Connection to localhost closed.
[hadoop@much ~]$ 

Tip: 配置证书登录的过程中,要注意 authorized_keys 文件的权限,如果没有达到预期效果可以通过 ssh -vvv localhost 来进行 debug

代码语言:javascript
复制
[hadoop@much ~]$ ssh -vvv localhost
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [::1] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/hadoop/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/hadoop/.ssh/id_rsa type 1
debug1: identity file /home/hadoop/.ssh/id_rsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_dsa type -1
debug1: identity file /home/hadoop/.ssh/id_dsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_ed25519 type -1
debug1: identity file /home/hadoop/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "localhost" from file "/home/hadoop/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/hadoop/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 0c:52:20:0a:00:e3:1a:5d:c6:fc:79:b3:e8:6e:d6:f1
debug3: load_hostkeys: loading entries for host "localhost" from file "/home/hadoop/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/hadoop/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /home/hadoop/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/hadoop/.ssh/id_rsa (0x7f284841f2c0),
debug2: key: /home/hadoop/.ssh/id_dsa ((nil)),
debug2: key: /home/hadoop/.ssh/id_ecdsa ((nil)),
debug2: key: /home/hadoop/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1001)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1001)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/hadoop/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp cb:02:3a:db:ce:eb:e3:d0:72:be:54:84:8f:cb:d7:c1
debug3: sign_and_send_pubkey: RSA cb:02:3a:db:ce:eb:e3:d0:72:be:54:84:8f:cb:d7:c1
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to localhost ([::1]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IPV6_TCLASS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env HOSTNAME
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env HISTSIZE
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env HISTCONTROL
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env LESSOPEN
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Aug  7 00:05:55 2017 from localhost
[hadoop@much ~]$ 

关闭 IPv6

代码语言:javascript
复制
[root@much ~]# sysctl -a | grep -i net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
[root@much ~]# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
[root@much ~]#
[root@much ~]# vim  /etc/sysctl.conf
[root@much ~]# sysctl -p 
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
[root@much ~]# sysctl -a | grep -i net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
[root@much ~]# 
[root@much ~]# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

[root@much ~]# 
[root@much ~]#  cat /proc/sys/net/ipv6/conf/all/disable_ipv6
1
[root@much ~]# 

Note: 如果不关闭 IPv6 会在后面的 MapReduce 任务调度过程中出错,因为 :8032 端口会绑定到 IPv6 的地址上面,从而使本地的 IPv4 的访问方式访问不到服务,而不断重试


下载 Hadoop 包

Hadoop 包的下载地址可以参考 Apache Hadoop Releases Download

这里我们下载最新的稳定版 hadoop-2.8.1.tar.gz

本文系转载,前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文系转载前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • 前言
  • 概要
    • 系统环境
      • 目标
        • 基础依赖
          • 安装 Java
            • 确保 ssh rsync 已经安装
              • 关闭 selinux
                • 创建 hadoop 用户 
                  • 配置用户免密码登录
                    • 关闭 IPv6
                      • 下载 Hadoop 包
                      相关产品与服务
                      云 HDFS
                      云 HDFS(Cloud HDFS,CHDFS)为您提供标准 HDFS 访问协议,您无需更改现有代码,即可使用高可用、高可靠、多维度安全、分层命名空间的分布式文件系统。 只需几分钟,您就可以在云端创建和挂载 CHDFS,来实现您大数据存储需求。随着业务需求的变化,您可以实时扩展或缩减存储资源,CHDFS 存储空间无上限,满足您海量大数据存储与分析业务需求。此外,通过 CHDFS,您可以实现计算与存储分离,极大发挥计算资源灵活性,同时实现存储数据永久保存,降低您大数据分析资源成本。
                      领券
                      问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档