Java API访问HA方式的HDFS

程裕强

发布于 2022-05-06 20:48:58

9230

发布于 2022-05-06 20:48:58

1、介绍

对于namenode是HA高可用集群时，客户端远程访问hdfs有两种实现方法：（1）将所有关于namenode的参数写入Configuration对象中（2）将配置文件core-site.xml和hdfs-site.xml文件复制到项目的src目录下

2、方式一

（1）样例代码

import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class HADemo {

    public static void main(String[] args) {
        // TODO Auto-generated method stub
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://my");
        conf.set("dfs.nameservices", "my");
        conf.set("dfs.ha.namenodes.my", "nn1,nn2");
        conf.set("dfs.namenode.rpc-address.my.nn1", "192.168.12.156:8020");
        conf.set("dfs.namenode.rpc-address.my.nn2", "192.168.157:8020");
        conf.set("dfs.client.failover.proxy.provider.my",
                "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
        try {
            FileSystem fs = FileSystem.get(new URI("hdfs://my"), conf,"root");
            boolean b=fs.mkdirs(new Path("/a"));
            System.out.println(b);
            fs.copyFromLocalFile(new Path("D:\\a.txt"), new Path("/a/a.txt"));
            fs.close();
        } catch (Exception e) {
            System.out.println(e);
        }

    }

}

（2）运行结果

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
true

（3）HDFS命令验证

[root@node1 ~]# hdfs dfs -ls /a
Found 2 items
-rw-r--r--   3 root supergroup         13 2018-08-08 09:11 /a/a.txt
[root@node1 ~]# hdfs dfs -cat /a/a.txt
Hello,Hadoop!
[root@node1 ~]#

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

原始发表：2018-08-08，如有侵权请联系 cloudcommunity@tencent.com 删除

xml