blocks|key|502243|text|看一下hadoop-test+jar|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|502244|<dependency>
++++<groupId>org.apache.hadoop</groupId>
++++<artifactId>hadoop-test</artifactId>
++++<version>0.20.205.0</version>
</dependency>|code-block|syntax|javascript|502245|它包含用于设置MiniDFSCluster和MiniMRCluster的类，因此您可以在没有Hadoop的情况下进行测试|502246|entityMap^0|3|B|0|0|7|E|M|D|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|S|8|@$9|T|A|U|B|C]|$9|V|A|W|B|C]]|D|@]|E|$]]|$1|M|3|-4|5|6|7|X|8|@]|D|@]|E|$]]]|N|$]]

Take a look at the <code>hadoop-test</code> jar 

<pre><code>&lt;dependency&gt;
 &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
 &lt;artifactId&gt;hadoop-test&lt;/artifactId&gt;
 &lt;version&gt;0.20.205.0&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>

it has classes for setting up a <code>MiniDFSCluster</code> and <code>MiniMRCluster</code> so you can test without Hadoop

blocks|key|502308|text|如果您使用的是hadoop+2.0.0及更高版本-请考虑使用hadoop-minicluster|type|unstyled|depth|inlineStyleRanges|entityRanges|data|502309|<dependency>
++++<groupId>org.apache.hadoop</groupId>
++++<artifactId>hadoop-minicluster</artifactId>
++++<version>2.5.0</version>
++++<scope>test</scope>
</dependency>|code-block|syntax|javascript|502310|使用它，您可以在本地计算机上创建一个临时hdfs，并在其上运行测试。setUp方法可能如下所示：|502311|baseDir+=+Files.createTempDirectory("test_hdfs").toFile().getAbsoluteFile();
Configuration+conf+=+new+Configuration();
conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR,+baseDir.getAbsolutePath());
MiniDFSCluster.Builder+builder+=+new+MiniDFSCluster.Builder(conf);
hdfsCluster+=+builder.build();

String+hdfsURI+=+"hdfs://localhost:"%2B+hdfsCluster.getNameNodePort()+%2B+"/";
DistributedFileSystem+fileSystem+=+hdfsCluster.getFileSystem();|502312|在tearDown方法中，您应该关闭mini+hdfs集群，并删除临时目录。|502313|hdfsCluster.shutdown();
FileUtil.fullyDelete(baseDir);|502314|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|S|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|T|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|U|8|@]|9|@]|A|$]]|$1|M|3|N|5|D|7|V|8|@]|9|@]|A|$E|F]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

If you're using hadoop 2.0.0 and above - consider using a hadoop-minicluster

<pre><code>&lt;dependency&gt;
 &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
 &lt;artifactId&gt;hadoop-minicluster&lt;/artifactId&gt;
 &lt;version&gt;2.5.0&lt;/version&gt;
 &lt;scope&gt;test&lt;/scope&gt;
&lt;/dependency&gt;
</code></pre>

With it, you can create a temporary hdfs on your local machine, and run your tests on it. A setUp method may look like this:

<pre><code>baseDir = Files.createTempDirectory("test_hdfs").toFile().getAbsoluteFile();
Configuration conf = new Configuration();
conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, baseDir.getAbsolutePath());
MiniDFSCluster.Builder builder = new MiniDFSCluster.Builder(conf);
hdfsCluster = builder.build();

String hdfsURI = "hdfs://localhost:"+ hdfsCluster.getNameNodePort() + "/";
DistributedFileSystem fileSystem = hdfsCluster.getFileSystem();
</code></pre>

And in a tearDown method you should shut down your mini hdfs cluster, and remove temporary directory.

<pre><code>hdfsCluster.shutdown();
FileUtil.fullyDelete(baseDir);
</code></pre>

blocks|key|718092|text|为什么不使用Mockito或PowerMock这样的模拟框架来模拟你与FileSystem的交互呢？您的单元测试不应该依赖于实际的FileSystem，而应该只是在与FileSystem交互时验证代码中的行为。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|718093|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Why not use a mocking framework like Mockito or PowerMock to mock your interations with the FileSystem? Your unit tests should not depend on an actual FileSystem, but should just be verifying behavior in your code in interacting with the FileSystem.

blocks|key|502111|text|我所做的(直到我找到更好的解决方案)扩展了FileSystem。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|502112|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

What I have done (until I will find better solution) I extended the FileSystem.

blocks|key|718508|text|我的解决方案是创建一个DummyFileSystem``+that+extends+abstract+Hadoop文件系统`，这样我就可以伪造一个文件是否存在，等等。“所有文件都存在”的例子：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|718509|@Override
public+FileStatus+getFileStatus(Path+f)+throws+IOException+{
++++return+new+FileStatus(10,+false,+3,+128*1024*1024,1,1,+null,+null,+null,+f);
}|code-block|syntax|javascript|718510|我发现更容易控制伪造的数据。|718511|entityMap^0|B|1A|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|S|8|@]|D|@]|E|$]]|$1|M|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|N|$]]

My solution is to create a <code>DummyFileSystem`` that extends abstract Hadoop</code>FileSystem`, so I can fake if a file exists or not, etc. Example of "all file exists":

<pre><code>@Override
public FileStatus getFileStatus(Path f) throws IOException {
 return new FileStatus(10, false, 3, 128*1024*1024,1,1, null, null, null, f);
}
</code></pre>

I found easier to keep control on faked data.

blocks|key|502639|text|我使用sbt尝试了Thirupathi+Chavati和Alexander+Tokarev解决方案，并且：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|502640|import+org.apache.hadoop.hdfs.MiniDFSCluster|offset|length|style|CODE|502641|只有添加以下内容才能正常工作：|502642|libraryDependencies+%2B=+"org.apache.hadoop"+%25+"hadoop-hdfs"+%25+"2.8.1"+classifier+"tests"+libraryDependencies+%2B=+"org.apache.hadoop"+%25+"hadoop-common"+%25+"2.8.1"+classifier+"tests"|502643|entityMap^0|0|0|18|0|0|0|2F|2G|2H|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|O|8|@$D|P|E|Q|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|R|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|S|8|@$D|T|E|U|F|G]|$D|V|E|W|F|G]]|9|@]|A|$]]|$1|L|3|-4|5|6|7|X|8|@]|9|@]|A|$]]]|M|$]]

I tried Thirupathi Chavati and Alexander Tokarev solutions with sbt, and :

<code>import org.apache.hadoop.hdfs.MiniDFSCluster</code> 

will only work by adding: 

<code>libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.8.1" classifier "tests"</code>
<code>libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.8.1" classifier "tests"</code>

blocks|key|502378|text|你可能想看看RawLocalFileSystem。不过，我觉得你最好还是嘲笑它。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|502379|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

You might want to take a look at RawLocalFileSystem. Though I think you'd better just mock it.

blocks|key|718310|text|您可以使用HBaseTestingUtility|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|718311|public+class+SomeTest+{
++++private+HBaseTestingUtility+testingUtil+=+new+HBaseTestingUtility();

++++@Before
++++public+void+setup()+throws+Exception+{
++++++++testingUtil.startMiniDFSCluster(1);
++++}

++++@After
++++public+void+tearDown()+throws+IOException+{
++++++++testingUtil.shutdownMiniDFSCluster();
++++}

++++@Test
++++public+void+test()+throws+Exception+{
++++++++DistributedFileSystem+fs+=+testingUtil.getDFSCluster().getFileSystem();
++++++++final+Path+dstPath+=+new+Path("/your/path/file.txt);
++++++++final+Path+srcPath+=+new+Path(SomeTest.class.getResource("file.txt").toURI());
++++++++fs.copyFromLocalFile(srcPath,+dstPath);
++++++++...
++++}
}|code-block|syntax|javascript|718312|entityMap|0|LINK|mutability|MUTABLE|url|https://hbase.apache.org/testapidocs/org/apache/hadoop/hbase/HBaseTestingUtility.html^0|5|J|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

You can use <a href="https://hbase.apache.org/testapidocs/org/apache/hadoop/hbase/HBaseTestingUtility.html" rel="nofollow noreferrer">HBaseTestingUtility</a>:

<pre><code>public class SomeTest {
 private HBaseTestingUtility testingUtil = new HBaseTestingUtility();

 @Before
 public void setup() throws Exception {
 testingUtil.startMiniDFSCluster(1);
 }

 @After
 public void tearDown() throws IOException {
 testingUtil.shutdownMiniDFSCluster();
 }

 @Test
 public void test() throws Exception {
 DistributedFileSystem fs = testingUtil.getDFSCluster().getFileSystem();
 final Path dstPath = new Path("/your/path/file.txt);
 final Path srcPath = new Path(SomeTest.class.getResource("file.txt").toURI());
 fs.copyFromLocalFile(srcPath, dstPath);
 ...
 }
}
</code></pre>

blocks|key|502498|text|添加以下依赖项|type|unstyled|depth|inlineStyleRanges|entityRanges|data|502499|++++<dependency>
++++++++<groupId>org.apache.hadoop</groupId>
++++++++<artifactId>hadoop-minicluster</artifactId>
++++++++<version>2.7.3</version>
+++++++
++++</dependency>|code-block|syntax|javascript|502500|添加以下代码，它将创建FileSysetm。|502501|import+java.nio.file.{Files,+Paths}


import+org.apache.hadoop.conf.Configuration
import+org.apache.hadoop.fs.FileSystem
import+org.apache.hadoop.hdfs.MiniDFSCluster


object+MiniClusterDemo+extends+App+{
++def+sysDir:+String+=+System.getProperty("user.dir")

++if(miniCluster!=null)+println("Cluster+created+and+active")+else+println("something+went+wrong")

++def+miniCluster:+FileSystem+=+{
++++val+basePath+=+Paths.get(s"$sysDir")

++++val+baseDir+=+Files.createTempDirectory(basePath,"hdfs_test").toFile.getAbsoluteFile
++++val+conf+=+new+Configuration()
+++conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR,+baseDir.getAbsolutePath)
++++val+hdfsCluster+=+new+MiniDFSCluster.Builder(conf).build()
++++val+hdfsURI+=+s"hdfs://localhost:${hdfsCluster.getNameNodePort}/"
++++val+fileSystem+=+hdfsCluster.getFileSystem
++++//hdfsCluster.shutdown();
++++//FileUtil.fullyDelete(baseDir);
++++fileSystem
++}

}|502502|创建MiniCluster后，请参阅示例日志|502503|​|502504|📷|atomic|offset|length|502505|502506|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/3ba4239093ecd3092858f99c69e8ef69.jpeg|imageAlt^0|0|0|0|0|0|0|0|1|0|0|0^^$0|@$1|2|3|4|5|6|7|13|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|14|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|15|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|16|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|17|8|@]|9|@]|A|$]]|$1|M|3|N|5|6|7|18|8|@]|9|@]|A|$]]|$1|O|3|P|5|Q|7|19|8|@]|9|@$R|1A|S|1B|1|1C]]|A|$]]|$1|T|3|N|5|6|7|1D|8|@]|9|@]|A|$]]|$1|U|3|-4|5|6|7|1E|8|@]|9|@]|A|$]]]|V|$W|$5|X|Y|Z|A|$10|11|12|-4]]]]

add below dependency


<pre><code> &lt;dependency&gt;
 &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
 &lt;artifactId&gt;hadoop-minicluster&lt;/artifactId&gt;
 &lt;version&gt;2.7.3&lt;/version&gt;
 &lt;!-- &lt;scope&gt;test&lt;/scope&gt;--&gt;
 &lt;/dependency&gt;
</code></pre>

add the below code, it will create the FileSysetm.

<pre><code>import java.nio.file.{Files, Paths}


import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.hdfs.MiniDFSCluster


object MiniClusterDemo extends App {
 def sysDir: String = System.getProperty("user.dir")

 if(miniCluster!=null) println("Cluster created and active") else println("something went wrong")

 def miniCluster: FileSystem = {
 val basePath = Paths.get(s"$sysDir")

 val baseDir = Files.createTempDirectory(basePath,"hdfs_test").toFile.getAbsoluteFile
 val conf = new Configuration()
 conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, baseDir.getAbsolutePath)
 val hdfsCluster = new MiniDFSCluster.Builder(conf).build()
 val hdfsURI = s"hdfs://localhost:${hdfsCluster.getNameNodePort}/"
 val fileSystem = hdfsCluster.getFileSystem
 //hdfsCluster.shutdown();
 //FileUtil.fullyDelete(baseDir);
 fileSystem
 }

}
</code></pre>

See the sample logs after creation of MiniCluster

<a href="https://i.stack.imgur.com/6d9m1.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6d9m1.jpg" alt="enter image description here"></a>

I want to run unit test but I need to have a org.apache.hadoop.fs.FileSystem instance.
Are there any mock or any other solution for creating FileSystem?

Hadoop: How to unit test FileSystem

单元测试

文件系统

Hadoop 

我想运行单元测试，但我需要有一个org.apache.hadoop.fs.FileSystem实例。是否有任何模拟或任何其他解决方案来创建FileSystem？

问Hadoop:如何对FileSystem进行单元测试
EN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Hadoop:如何对FileSystem进行单元测试EN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Hadoop:如何对FileSystem进行单元测试
EN