Hadoop 在 Centos7 下的单机布署(二).HDFS.Pseudo-Distributed Operation
Hadoop 在 Centos7 下的单机布署(二).HDFS.Pseudo-Distributed Operation
franket
发布于 2021-08-12 09:39:40
发布于 2021-08-12 09:39:40
文章被收录于专栏：技术杂记
前言
随着人工智能在各领域的大面积普及，数据分析在各个层面的广泛应用
其底层的技术在圈子里越来越受到重视和热捧
Apache Hadoop 技术栈或者说技术生态圈绝对是一个不容忽视的中坚力量
Apache Hadoop 是一个专注于可靠，弹性，分布式计算框架的开源软件项目
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
主要包含以下几个模块：
- Hadoop Common : 支持其它 hadoop 模块的通用工具
- Hadoop Distributed File System : 简称 HDFS , 给应用数据提供高吞吐性能的分布式文件系统
- Hadoop YARN : 工作调度与集群资源管理的框架
- Hadoop MapReduce : 大数据集的并行处理系统
Hadoop 生态圈中的其它项目可以参考 Hadoop-related projects
Tip: 当前的最新稳定版为 Hadoop Release 2.8.1 发布于 08 June, 2017
前面根据官方的文档给出最新版 Hadoop 在 Centos7 下的单机布署方案，详细可以参考 Hadoop 在 Centos7 下的单机布署(一). Standalone Operation
这里基于前面的操作，给出最新版 Hadoop 在 Centos7 下 HDFS 的伪分布模式(单机)的布署方案
此文章借鉴了 Hadoop Wiki 中的部分内容和 Running Hadoop on Ubuntu Linux (Single-Node Cluster) 中的部分操作
概要
系统环境
[root@much ~]# hostnamectl
Static hostname: much
Icon name: computer-vm
Chassis: vm
Machine ID: 33dc28f7e76c4903ad9b603b77e29a7c
Boot ID: 8cfd2a8aec4f4235b4776f7cd2fdfdb1
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-514.21.1.el7.x86_64
Architecture: x86-64
[root@much ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:e3:df:87 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 84190sec preferred_lft 84190sec
inet6 fe80::2bb7:5b3:9584:d8eb/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d3:ec:e7 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.207/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fed3:ece7/64 scope link
valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:16:5e:11 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:16:5e:11 brd ff:ff:ff:ff:ff:ff
[root@much ~]# uname -a
Linux much 3.10.0-514.21.1.el7.x86_64 #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@much ~]# 目标
- 构建一个 hadoop 的单点伪集群
- hdfs 的基本操作
单机测试
参考 Hadoop 在 Centos7 下的单机布署(一). Standalone Operation 中的操作构建出脱机版的 Hadoop
我们在本地尝试执行一个 wordcount 的脱机任务
[hadoop@much hadoop-2.8.1]$ rm -rf output/
[hadoop@much hadoop-2.8.1]$ ls
21centry etc input libexec NOTICE.txt sbin
bin include lib LICENSE.txt README.txt share
[hadoop@much hadoop-2.8.1]$ cat 21centry
In the middle of a vacation, you’re often having so much fun you want a memento to remember it. But that cute market trinket can quickly turn into a dust magnet when you get home. Here are some ideas that put the “fun” in functional.
Grouping similar items
Consider collections- This has the potential to get out of control, but if you travel a limited amount, think about one item that you could buy in every location. It might be a small basket, a serving bowl, or a textile. Over time, you can build up your collection slowly and it will show off your personality as you build it.
Confine trinkets to one space- If you stick to a theme when it comes to choosing your souvenirs, try to be thoughtful about how you place them. For example, if you decide you love masks, put them all on one wall as a group of artwork. If you have one mask in each room, it will just look like someone is always watching you! On the other hand, if you collect blankets or throws, you may want to strategically place a different one in each room where you might want to curl up.
Know when to stop- The goal here is to not turn your space into one of those freakish sideshows you see on the internet where every inch of space is covered. Every few trips, reassess your needs and whether one more of the same item could be overkill. When a collection is complete, appreciate it and move on to something else you’ve been eyeing on your travels.
A selective approach
Select wisely- T-shirts, shot glasses, magnets, and snow globes can be a default purchase, but are rarely appreciated once they’re out of a suitcase. Look for more unique items that may not scream tourist, but instead remind you of a city. For example, you may not want a keychain with the Eiffel Tower on it in Paris, but a unique watering can in the window of a gardening store brings a smile to your face. Buy it! Every time you use it you’ll be reminded of your trip, but the item won’t be cluttering your home.
Keep your colour scheme in mind- Sometimes when you go on vacation, especially to a tropical destination, you’ll fall in love a vibrant blue or orange. But, when you get home, you wonder what was wrong with your eyes as the hue is way too much for your décor. Keep that in mind as you shop and perhaps select a more subdued tone unless you are completely confident that bold is the look you’re after.
Some of the most interesting homes have carefully chosen items that incorporate a story into the overall décor. Your vacations make up the story of your life and the items you bring home can help create a unique look, as interesting as your life.
[hadoop@much hadoop-2.8.1]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar wordcount ./21centry ./test_output
17/08/07 12:05:22 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/08/07 12:05:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/08/07 12:05:28 INFO input.FileInputFormat: Total input files to process : 1
17/08/07 12:05:28 INFO mapreduce.JobSubmitter: number of splits:1
17/08/07 12:05:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local279813974_0001
17/08/07 12:05:28 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/08/07 12:05:28 INFO mapreduce.Job: Running job: job_local279813974_0001
17/08/07 12:05:28 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/08/07 12:05:28 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/07 12:05:28 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/08/07 12:05:28 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/08/07 12:05:28 INFO mapred.LocalJobRunner: Waiting for map tasks
17/08/07 12:05:28 INFO mapred.LocalJobRunner: Starting task: attempt_local279813974_0001_m_000000_0
17/08/07 12:05:28 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/07 12:05:28 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/08/07 12:05:28 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/07 12:05:28 INFO mapred.MapTask: Processing split: file:/home/hadoop/hadoop-2.8.1/21centry:0+2637
17/08/07 12:05:28 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/07 12:05:28 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/07 12:05:28 INFO mapred.MapTask: soft limit at 83886080
17/08/07 12:05:28 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/07 12:05:28 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/07 12:05:28 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/07 12:05:28 INFO mapred.LocalJobRunner:
17/08/07 12:05:28 INFO mapred.MapTask: Starting flush of map output
17/08/07 12:05:28 INFO mapred.MapTask: Spilling map output
17/08/07 12:05:28 INFO mapred.MapTask: bufstart = 0; bufend = 4577; bufvoid = 104857600
17/08/07 12:05:28 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26212452(104849808); length = 1945/6553600
17/08/07 12:05:28 INFO mapred.MapTask: Finished spill 0
17/08/07 12:05:28 INFO mapred.Task: Task:attempt_local279813974_0001_m_000000_0 is done. And is in the process of committing
17/08/07 12:05:28 INFO mapred.LocalJobRunner: map
17/08/07 12:05:28 INFO mapred.Task: Task 'attempt_local279813974_0001_m_000000_0' done.
17/08/07 12:05:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local279813974_0001_m_000000_0
17/08/07 12:05:28 INFO mapred.LocalJobRunner: map task executor complete.
17/08/07 12:05:28 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/08/07 12:05:28 INFO mapred.LocalJobRunner: Starting task: attempt_local279813974_0001_r_000000_0
17/08/07 12:05:28 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/07 12:05:28 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17/08/07 12:05:28 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/07 12:05:28 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@76f76d2b
17/08/07 12:05:28 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/08/07 12:05:28 INFO reduce.EventFetcher: attempt_local279813974_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
17/08/07 12:05:28 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local279813974_0001_m_000000_0 decomp: 3374 len: 3378 to MEMORY
17/08/07 12:05:28 INFO reduce.InMemoryMapOutput: Read 3374 bytes from map-output for attempt_local279813974_0001_m_000000_0
17/08/07 12:05:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 3374, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->3374
17/08/07 12:05:28 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
17/08/07 12:05:28 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/08/07 12:05:28 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
17/08/07 12:05:28 INFO mapred.Merger: Merging 1 sorted segments
17/08/07 12:05:28 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 3370 bytes
17/08/07 12:05:28 INFO reduce.MergeManagerImpl: Merged 1 segments, 3374 bytes to disk to satisfy reduce memory limit
17/08/07 12:05:28 INFO reduce.MergeManagerImpl: Merging 1 files, 3378 bytes from disk
17/08/07 12:05:28 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
17/08/07 12:05:28 INFO mapred.Merger: Merging 1 sorted segments
17/08/07 12:05:28 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 3370 bytes
17/08/07 12:05:28 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/08/07 12:05:28 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
17/08/07 12:05:28 INFO mapred.Task: Task:attempt_local279813974_0001_r_000000_0 is done. And is in the process of committing
17/08/07 12:05:28 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/08/07 12:05:28 INFO mapred.Task: Task attempt_local279813974_0001_r_000000_0 is allowed to commit now
17/08/07 12:05:28 INFO output.FileOutputCommitter: Saved output of task 'attempt_local279813974_0001_r_000000_0' to file:/home/hadoop/hadoop-2.8.1/test_output/_temporary/0/task_local279813974_0001_r_000000
17/08/07 12:05:28 INFO mapred.LocalJobRunner: reduce > reduce
17/08/07 12:05:28 INFO mapred.Task: Task 'attempt_local279813974_0001_r_000000_0' done.
17/08/07 12:05:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local279813974_0001_r_000000_0
17/08/07 12:05:28 INFO mapred.LocalJobRunner: reduce task executor complete.
17/08/07 12:05:29 INFO mapreduce.Job: Job job_local279813974_0001 running in uber mode : false
17/08/07 12:05:29 INFO mapreduce.Job: map 100% reduce 100%
17/08/07 12:05:29 INFO mapreduce.Job: Job job_local279813974_0001 completed successfully
17/08/07 12:05:29 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=616258
FILE: Number of bytes written=1259654
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=17
Map output records=487
Map output bytes=4577
Map output materialized bytes=3378
Input split bytes=104
Combine input records=487
Combine output records=270
Reduce input groups=270
Reduce shuffle bytes=3378
Reduce input records=270
Reduce output records=270
Spilled Records=540
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=440401920
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2637
File Output Format Counters
Bytes Written=2326
[hadoop@much hadoop-2.8.1]$ echo $?
0
[hadoop@much hadoop-2.8.1]$ ll test_output/
total 4
-rw-r--r--. 1 hadoop hadoop 2298 Aug 7 12:05 part-r-00000
-rw-r--r--. 1 hadoop hadoop 0 Aug 7 12:05 _SUCCESS
[hadoop@much hadoop-2.8.1]$ head -n 20 test_output/part-r-00000
A 1
But 1
But, 1
Buy 1
Confine 1
Consider 1
Eiffel 1
Every 2
For 2
Grouping 1
Here 1
If 2
In 1
It 1
Keep 2
Know 1
Look 1
On 1
Over 1
Paris, 1
[hadoop@much hadoop-2.8.1]$ Note: test_output 目录是在执行过程中被 Hadoop 自动创建的，不用事先自己创建，如果事先自己创建了，会出错，解决办法是在一个有写权限的位置指定一个未被创建的子目录
创建数据存放目录
[hadoop@much hadoop-2.8.1]$ ls
21centry bin include lib LICENSE.txt README.txt share
4300-0.txt etc input libexec NOTICE.txt sbin
[hadoop@much hadoop-2.8.1]$ mkdir hdfs
[hadoop@much hadoop-2.8.1]$ ll hdfs/
total 0
[hadoop@much hadoop-2.8.1]$ ll hdfs/ -d
drwxrwxr-x. 2 hadoop hadoop 6 Aug 7 12:19 hdfs/
[hadoop@much hadoop-2.8.1]$ cd hdfs/
[hadoop@much hdfs]$ pwd
/home/hadoop/hadoop-2.8.1/hdfs
[hadoop@much hdfs]$ 测试证书登录
[hadoop@much hadoop-2.8.1]$ ssh localhost
Last login: Mon Aug 7 11:03:56 2017
[hadoop@much ~]$ exit
logout
Connection to localhost closed.
[hadoop@much hadoop-2.8.1]$ ssh localhost
Last login: Mon Aug 7 13:35:02 2017 from localhost
[hadoop@much ~]$ exit
logout
Connection to localhost closed.
[hadoop@much hadoop-2.8.1]$HDFS 的架构
HDFS 的架构可以参考 Overview of the HDFS Architecture
这是 Application 与 NameNode 还有 DataNode 之间的交互架构
hdfs_runtime.jpg
这是 DataNodes 之间的复制架构
hdfs_blockreplication.gif
HDFS 的配置
修改 etc/hadoop/core-site.xml
[hadoop@much hadoop-2.8.1]$ vim etc/hadoop/core-site.xml
[hadoop@much hadoop-2.8.1]$ cat etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hadoop/hadoop-2.8.1/hdfs</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
[hadoop@much hadoop-2.8.1]$修改 etc/hadoop/hdfs-site.xml
[hadoop@much hadoop-2.8.1]$ vim etc/hadoop/hdfs-site.xml
[hadoop@much hadoop-2.8.1]$ cat etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/hadoop-2.8.1/hdfs/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/hadoop-2.8.1/hdfs/dfs/data</value>
</property>
</configuration>
[hadoop@much hadoop-2.8.1]$ 格式化 hdfs
[hadoop@much hadoop-2.8.1]$ ll hdfs/
total 0
[hadoop@much hadoop-2.8.1]$ bin/hdfs namenode -format
17/08/07 14:04:21 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: user = hadoop
STARTUP_MSG: host = much/10.0.2.15
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.8.1
/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释
* STARTUP_MSG: classpath = /home/hadoop/hadoop-2.8.1/etc/hadoop:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/hadoop-auth-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jcip-annotations-1.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/json-smart-1.1.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jsch-0.1.51.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/hadoop-annotations-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/httpclient-4.5.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/httpcore-4.4.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/hadoop-common-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/hadoop-common-2.8.1-tests.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/common/hadoop-nfs-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/okio-1.4.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-2.8.1-tests.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1-tests.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1-tests.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/fst-2.24.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/objenesis-2.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-math-2.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-api-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-common-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-common-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-client-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-registry-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar:/home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1-tests.jar:/contrib/capacity-scheduler/*.jar
*/
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe; compiled by 'vinodkv' on 2017-06-02T06:14Z
STARTUP_MSG: java = 1.8.0_141
************************************************************/
17/08/07 14:04:21 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/08/07 14:04:21 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-dabe1865-130e-430c-99e4-d54608b2dad3
17/08/07 14:04:22 INFO namenode.FSEditLog: Edit logging is async:false
17/08/07 14:04:22 INFO namenode.FSNamesystem: KeyProvider: null
17/08/07 14:04:22 INFO namenode.FSNamesystem: fsLock is fair: true
17/08/07 14:04:22 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
17/08/07 14:04:22 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/08/07 14:04:22 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/08/07 14:04:22 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/08/07 14:04:22 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Aug 07 14:04:22
17/08/07 14:04:22 INFO util.GSet: Computing capacity for map BlocksMap
17/08/07 14:04:22 INFO util.GSet: VM type = 64-bit
17/08/07 14:04:22 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
17/08/07 14:04:22 INFO util.GSet: capacity = 2^21 = 2097152 entries
17/08/07 14:04:22 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/08/07 14:04:22 INFO blockmanagement.BlockManager: defaultReplication = 1
17/08/07 14:04:22 INFO blockmanagement.BlockManager: maxReplication = 512
17/08/07 14:04:22 INFO blockmanagement.BlockManager: minReplication = 1
17/08/07 14:04:22 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
17/08/07 14:04:22 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/08/07 14:04:22 INFO blockmanagement.BlockManager: encryptDataTransfer = false
17/08/07 14:04:22 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
17/08/07 14:04:22 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
17/08/07 14:04:22 INFO namenode.FSNamesystem: supergroup = supergroup
17/08/07 14:04:22 INFO namenode.FSNamesystem: isPermissionEnabled = true
17/08/07 14:04:22 INFO namenode.FSNamesystem: HA Enabled: false
17/08/07 14:04:22 INFO namenode.FSNamesystem: Append Enabled: true
17/08/07 14:04:22 INFO util.GSet: Computing capacity for map INodeMap
17/08/07 14:04:22 INFO util.GSet: VM type = 64-bit
17/08/07 14:04:22 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
17/08/07 14:04:22 INFO util.GSet: capacity = 2^20 = 1048576 entries
17/08/07 14:04:22 INFO namenode.FSDirectory: ACLs enabled? false
17/08/07 14:04:22 INFO namenode.FSDirectory: XAttrs enabled? true
17/08/07 14:04:22 INFO namenode.NameNode: Caching file names occurring more than 10 times
17/08/07 14:04:22 INFO util.GSet: Computing capacity for map cachedBlocks
17/08/07 14:04:22 INFO util.GSet: VM type = 64-bit
17/08/07 14:04:22 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
17/08/07 14:04:22 INFO util.GSet: capacity = 2^18 = 262144 entries
17/08/07 14:04:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/08/07 14:04:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/08/07 14:04:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
17/08/07 14:04:22 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/08/07 14:04:22 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/08/07 14:04:22 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/08/07 14:04:22 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/08/07 14:04:22 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/08/07 14:04:22 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/08/07 14:04:22 INFO util.GSet: VM type = 64-bit
17/08/07 14:04:22 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
17/08/07 14:04:22 INFO util.GSet: capacity = 2^15 = 32768 entries
17/08/07 14:04:22 INFO namenode.FSImage: Allocated new BlockPoolId: BP-788595051-10.0.2.15-1502085862480
17/08/07 14:04:22 INFO common.Storage: Storage directory /home/hadoop/hadoop-2.8.1/hdfs/dfs/name has been successfully formatted.
17/08/07 14:04:22 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoop-2.8.1/hdfs/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/08/07 14:04:22 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoop-2.8.1/hdfs/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.
17/08/07 14:04:22 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/08/07 14:04:22 INFO util.ExitUtil: Exiting with status 0
17/08/07 14:04:22 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at much/10.0.2.15
************************************************************/
[hadoop@much hadoop-2.8.1]$ echo $?
0
[hadoop@much hadoop-2.8.1]$ 格式化完成后可以看到新创建的文件夹 hdfs 里多出来了东西
[hadoop@much hadoop-2.8.1]$ ll hdfs/
total 0
drwxrwxr-x. 3 hadoop hadoop 18 Aug 7 14:04 dfs
[hadoop@much hadoop-2.8.1]$ tree hdfs/
hdfs/
└── dfs
└── name
└── current
├── fsimage_0000000000000000000
├── fsimage_0000000000000000000.md5
├── seen_txid
└── VERSION
3 directories, 4 files
[hadoop@much hadoop-2.8.1]$ 启动 NameNode 与 DataNode 守护进程
本文系转载，前往查看
如有侵权，请联系 cloudcommunity@tencent.com 删除。
本文系转载，前往查看
如有侵权，请联系 cloudcommunity@tencent.com 删除。
评论
登录后参与评论
推荐阅读
腾讯云开发者
