查看Hadoop HDFS 中的一个文件对应block信息

本文地址:http://blog.csdn.net/chengyuqiang/article/details/78163091

如果需要查看Hadoop HDFS 中的一个文件对应block信息,比如block数、block大小、block所在位置等,可以使用hdfs fsck命令。

1. HDFS示例文件

hdfs dfs -ls /user/root/input

[root@node1 data]# hdfs dfs -ls /user/root/input
Found 7 items
-rw-r--r--   3 root supergroup     281498 2017-09-20 10:11 /user/root/input/Hamlet.txt
-rw-r--r--   3 root supergroup    9789248 2017-09-22 10:26 /user/root/input/age.txt
-rw-r--r--   3 root supergroup         71 2017-08-27 09:18 /user/root/input/books.txt
-rw-r--r--   3 root supergroup  264075431 2017-10-05 09:37 /user/root/input/cite75_99.txt
drwxr-xr-x   - root supergroup          0 2017-08-13 09:33 /user/root/input/emp.bak
drwxr-xr-x   - root supergroup          0 2017-09-24 04:08 /user/root/input/ml-1m
-rw-r--r--   3 root supergroup  871353053 2017-10-05 09:40 /user/root/input/ncdc.txt
[root@node1 data]#

2. hdfs fsck用法

[root@node1 data]# hdfs fsck 
Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]] [-includeSnapshots] [-storagepolicies] [-blockId <blk_Id>]
    <path>  start checking from this path
    -move   move corrupted files to /lost+found
    -delete delete corrupted files
    -files  print out files being checked
    -openforwrite   print out files opened for write
    -includeSnapshots   include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
    -list-corruptfileblocks print out list of missing blocks and files they belong to
    -blocks print out block report
    -locations  print out locations for every block
    -racks  print out network topology for data-node locations
    -storagepolicies    print out storage policy summary for the blocks
    -blockId    print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)

Please Note:
    1. By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually  tagged CORRUPT or HEALTHY depending on their block allocation status
    2. Option -includeSnapshots should not be used for comparing stats, should be used only for HEALTH check, as this may contain duplicates if the same file present in both original fs tree and inside snapshots.

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

[root@node1 data]# 

3. 查看block基本信息

hdfs fsck input/cite75_99.txt

[root@node1 data]# hdfs fsck input/cite75_99.txt
Connecting to namenode via http://node1:50070/fsck?ugi=root&path=%2Fuser%2Froot%2Finput%2Fcite75_99.txt
FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/cite75_99.txt at Thu Oct 05 09:41:58 EDT 2017
.Status: HEALTHY
 Total size:    264075431 B
 Total dirs:    0
 Total files:   1
 Total symlinks:        0
 Total blocks (validated):  2 (avg. block size 132037715 B)
 Minimally replicated blocks:   2 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 3.0
 Corrupt blocks:        0
 Missing replicas:      0 (0.0 %)
 Number of data-nodes:      3
 Number of racks:       1
FSCK ended at Thu Oct 05 09:41:58 EDT 2017 in 3 milliseconds


The filesystem under path '/user/root/input/cite75_99.txt' is HEALTHY
[root@node1 data]# hdfs fsck input/ncdc.txt
Connecting to namenode via http://node1:50070/fsck?ugi=root&path=%2Fuser%2Froot%2Finput%2Fncdc.txt
FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/ncdc.txt at Thu Oct 05 09:42:22 EDT 2017
.Status: HEALTHY
 Total size:    871353053 B
 Total dirs:    0
 Total files:   1
 Total symlinks:        0
 Total blocks (validated):  7 (avg. block size 124479007 B)
 Minimally replicated blocks:   7 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 3.0
 Corrupt blocks:        0
 Missing replicas:      0 (0.0 %)
 Number of data-nodes:      3
 Number of racks:       1
FSCK ended at Thu Oct 05 09:42:22 EDT 2017 in 2 milliseconds


The filesystem under path '/user/root/input/ncdc.txt' is HEALTHY
[root@node1 data]# 

4. -files -blocks -locations选项

hdfs fsck input/ncdc.txt -files -blocks hdfs fsck input/ncdc.txt -files -blocks -locations

[root@node1 data]# hdfs fsck input/ncdc.txt -files -blocks
Connecting to namenode via http://node1:50070/fsck?ugi=root&files=1&blocks=1&path=%2Fuser%2Froot%2Finput%2Fncdc.txt
FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/ncdc.txt at Thu Oct 05 09:47:14 EDT 2017
/user/root/input/ncdc.txt 871353053 bytes, 7 block(s):  OK
0. BP-766589174-192.168.80.131-1500731607717:blk_1073742821_2026 len=134217728 repl=3
1. BP-766589174-192.168.80.131-1500731607717:blk_1073742822_2027 len=134217728 repl=3
2. BP-766589174-192.168.80.131-1500731607717:blk_1073742823_2028 len=134217728 repl=3
3. BP-766589174-192.168.80.131-1500731607717:blk_1073742824_2029 len=134217728 repl=3
4. BP-766589174-192.168.80.131-1500731607717:blk_1073742825_2030 len=134217728 repl=3
5. BP-766589174-192.168.80.131-1500731607717:blk_1073742826_2031 len=134217728 repl=3
6. BP-766589174-192.168.80.131-1500731607717:blk_1073742827_2032 len=66046685 repl=3

Status: HEALTHY
 Total size:    871353053 B
 Total dirs:    0
 Total files:   1
 Total symlinks:        0
 Total blocks (validated):  7 (avg. block size 124479007 B)
 Minimally replicated blocks:   7 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 3.0
 Corrupt blocks:        0
 Missing replicas:      0 (0.0 %)
 Number of data-nodes:      3
 Number of racks:       1
FSCK ended at Thu Oct 05 09:47:14 EDT 2017 in 2 milliseconds


The filesystem under path '/user/root/input/ncdc.txt' is HEALTHY
[root@node1 data]# hdfs fsck input/ncdc.txt -files -blocks -locations
Connecting to namenode via http://node1:50070/fsck?ugi=root&files=1&blocks=1&locations=1&path=%2Fuser%2Froot%2Finput%2Fncdc.txt
FSCK started by root (auth:SIMPLE) from /192.168.80.131 for path /user/root/input/ncdc.txt at Thu Oct 05 09:47:45 EDT 2017
/user/root/input/ncdc.txt 871353053 bytes, 7 block(s):  OK
0. BP-766589174-192.168.80.131-1500731607717:blk_1073742821_2026 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK]]
1. BP-766589174-192.168.80.131-1500731607717:blk_1073742822_2027 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]]
2. BP-766589174-192.168.80.131-1500731607717:blk_1073742823_2028 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK], DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK]]
3. BP-766589174-192.168.80.131-1500731607717:blk_1073742824_2029 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]]
4. BP-766589174-192.168.80.131-1500731607717:blk_1073742825_2030 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]]
5. BP-766589174-192.168.80.131-1500731607717:blk_1073742826_2031 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK]]
6. BP-766589174-192.168.80.131-1500731607717:blk_1073742827_2032 len=66046685 repl=3 [DatanodeInfoWithStorage[192.168.80.131:50010,DS-602e79bf-d01e-4b6b-8712-f6293e394ab1,DISK], DatanodeInfoWithStorage[192.168.80.133:50010,DS-0056ec91-47b7-4c48-8f6e-89ca33be49c6,DISK], DatanodeInfoWithStorage[192.168.80.132:50010,DS-d3917eb8-31b4-49d6-b5eb-1316f7c0f310,DISK]]

Status: HEALTHY
 Total size:    871353053 B
 Total dirs:    0
 Total files:   1
 Total symlinks:        0
 Total blocks (validated):  7 (avg. block size 124479007 B)
 Minimally replicated blocks:   7 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 3.0
 Corrupt blocks:        0
 Missing replicas:      0 (0.0 %)
 Number of data-nodes:      3
 Number of racks:       1
FSCK ended at Thu Oct 05 09:47:45 EDT 2017 in 2 milliseconds


The filesystem under path '/user/root/input/ncdc.txt' is HEALTHY
[root@node1 data]#

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏搞前端的李蚊子

Html5模拟通讯录人员排序(sen.js)

// JavaScript Document  var PY_Json_Str = ""; var PY_Str_1 = ""; var PY_Str_...

5266
来自专栏pydata

python asynchrous network

541
来自专栏高爽的专栏

Notepad++列编辑、正则查找、替换

  目标:        将源数据转成初始化sql语句。 源数据: 104110040018,1,中国银行,中国银行天津琼州道支行,NULL,1100,天津市...

1830
来自专栏深度学习与数据挖掘实战

Graph application with Python, Neo4j, Gephi & Linkurious.js

I love Python, and to celebrate Packt Python week, I’ve spent some time developi...

402
来自专栏小樱的经验随笔

Codeforces 839E Mother of Dragons【__builtin_popcount()的使用】

E. Mother of Dragons time limit per test:2 seconds memory limit per test:256 meg...

2588
来自专栏菩提树下的杨过

Flash/Flex学习笔记(23):运动学原理

先写一个公用的小球类Ball: package{ import flash.display.Sprite; //小球 类 public class B...

22710
来自专栏Golang语言社区

Golang生产级可靠UDP库

kcp-go is a Production-Grade Reliable-UDP library for golang.

1392
来自专栏ASP.NET MVC5 后台权限管理系统

Bootstrap Metronic 学习记录(二)菜单栏

1.简介 1)  .环境配置 2)  .提取页面 2).动态生成菜单(无限级别树) 2.系统环境配置 项目需要程序数据支撑,这里选择MVC5.0+EF6.0[S...

1888
来自专栏生信技能树

肿瘤全外显子测序数据分析流程大放送

这个一个肿瘤外显子项目的文章发表并且公布的公共数据,我这里给出全套分析流程代码。只需要你肯实践,就可以运行成功。 PS:有些后起之秀自己运营公众号或者博客喜欢批...

6798
来自专栏西二旗一哥

Clang - Use Xcode to make first Clang plugin

995

扫码关注云+社区