blocks|key|311186|text|你有没有试过这个：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|311187|import+java.io.*;
import+java.util.*;
import+java.net.*;
import+org.apache.hadoop.fs.*;
import+org.apache.hadoop.conf.*;
import+org.apache.hadoop.io.*;
import+org.apache.hadoop.mapred.*;
import+org.apache.hadoop.util.*;

public+class+cat{
++++public+static+void+main+(String+[]+args)+throws+Exception{
++++++++try{
++++++++++++FileSystem+fs+=+FileSystem.get(new+Configuration());
++++++++++++FileStatus[]+status+=+fs.listStatus(new+Path("hdfs://test.com:9000/user/test/in"));++//+you+need+to+pass+in+your+hdfs+path

++++++++++++for+(int+i=0;i<status.length;i%2B%2B){
++++++++++++++++BufferedReader+br=new+BufferedReader(new+InputStreamReader(fs.open(status[i].getPath())));
++++++++++++++++String+line;
++++++++++++++++line=br.readLine();
++++++++++++++++while+(line+!=+null){
++++++++++++++++++++System.out.println(line);
++++++++++++++++++++line=br.readLine();
++++++++++++++++}
++++++++++++}
++++++++}catch(Exception+e){
++++++++++++System.out.println("File+not+found");
++++++++}
++++}
}|code-block|syntax|javascript|311188|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Have you tried this:

<pre><code>import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class cat{
 public static void main (String [] args) throws Exception{
 try{
 FileSystem fs = FileSystem.get(new Configuration());
 FileStatus[] status = fs.listStatus(new Path("hdfs://test.com:9000/user/test/in")); // you need to pass in your hdfs path

 for (int i=0;i&lt;status.length;i++){
 BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(status[i].getPath())));
 String line;
 line=br.readLine();
 while (line != null){
 System.out.println(line);
 line=br.readLine();
 }
 }
 }catch(Exception e){
 System.out.println("File not found");
 }
 }
}
</code></pre>

blocks|key|311227|text|您将需要使用FileSystem对象并对生成的FileStatus对象执行一些逻辑，以便手动递归到子目录中。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|311228|您还可以使用listStatus(Path,+PathFilter)方法将PathFilter应用于仅返回xml文件|311229|hadoop+FsShell类提供了hadoop+fs+-lsr命令的示例，该命令是一个递归ls+-请参阅the+source，在第590行(递归步骤在第635行触发)|311230|entityMap|0|LINK|mutability|MUTABLE|url|http://hadoop.apache.org/common/docs/r1.0.3/api/org/apache/hadoop/fs/FileSystem.html|1|http://hadoop.apache.org/common/docs/r1.0.3/api/org/apache/hadoop/fs/FileSystem.html#listStatus%2528org.apache.hadoop.fs.Path,%2520org.apache.hadoop.fs.PathFilter%2529|2|http://svn.apache.org/viewvc/hadoop/common/tags/release-1.0.3/src/core/org/apache/hadoop/fs/FsShell.java?view=markup^0|6|A|0|0|6|S|1|0|1H|A|2|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@$A|U|B|V|1|W]]|C|$]]|$1|D|3|E|5|6|7|X|8|@]|9|@$A|Y|B|Z|1|10]]|C|$]]|$1|F|3|G|5|6|7|11|8|@]|9|@$A|12|B|13|1|14]]|C|$]]|$1|H|3|-4|5|6|7|15|8|@]|9|@]|C|$]]]|I|$J|$5|K|L|M|C|$N|O]]|P|$5|K|L|M|C|$N|Q]]|R|$5|K|L|M|C|$N|S]]]]

You'll need to use the <a href="http://hadoop.apache.org/common/docs/r1.0.3/api/org/apache/hadoop/fs/FileSystem.html" rel="noreferrer">FileSystem</a> object and perform some logic on the resultant FileStatus objects to manually recurse into the subdirectories.

You can also apply a PathFilter to only return the xml files using the <a href="http://hadoop.apache.org/common/docs/r1.0.3/api/org/apache/hadoop/fs/FileSystem.html#listStatus%28org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.PathFilter%29" rel="noreferrer">listStatus(Path, PathFilter)</a> method

The hadoop FsShell class has examples of this for the hadoop fs -lsr command, which is a recursive ls - see <a href="http://svn.apache.org/viewvc/hadoop/common/tags/release-1.0.3/src/core/org/apache/hadoop/fs/FsShell.java?view=markup" rel="noreferrer">the source</a>, around line 590 (the recursive step is triggered on line 635)

blocks|key|311300|text|下面是一段计算特定HDFS目录中文件数量的代码片段(我用它来确定在特定ETL代码中使用多少个reducers+)。您可以轻松地对其进行修改以满足您的需求。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|311301|private+int+calculateNumberOfReducers(String+input)+throws+IOException+{
++++int+numberOfReducers+=+0;
++++Path+inputPath+=+new+Path(input);
++++FileSystem+fs+=+inputPath.getFileSystem(getConf());
++++FileStatus[]+statuses+=+fs.globStatus(inputPath);
++++for(FileStatus+status:+statuses)+{
++++++++if(status.isDirectory())+{
++++++++++++numberOfReducers+%2B=+getNumberOfInputFiles(status,+fs);
++++++++}+else+if(status.isFile())+{
++++++++++++numberOfReducers+%2B%2B;
++++++++}
++++}
++++return+numberOfReducers;
}

/**
+*+Recursively+determines+number+of+input+files+in+an+HDFS+directory
+*
+*+@param+status+instance+of+FileStatus
+*+@param+fs+instance+of+FileSystem
+*+@return+number+of+input+files+within+particular+HDFS+directory
+*+@throws+IOException
+*/
private+int+getNumberOfInputFiles(FileStatus+status,+FileSystem+fs)+throws+IOException++{
++++int+inputFileCount+=+0;
++++if(status.isDirectory())+{
++++++++FileStatus[]+files+=+fs.listStatus(status.getPath());
++++++++for(FileStatus+file:+files)+{
++++++++++++inputFileCount+%2B=+getNumberOfInputFiles(file,+fs);
++++++++}
++++}+else+{
++++++++inputFileCount+%2B%2B;
++++}

++++return+inputFileCount;
}|code-block|syntax|javascript|311302|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Here is a code snippet, that counts number of files in a particular HDFS directory (I used this to determine how many reducers to use in a particular ETL code). You can easily modify this to suite your needs.

<pre><code>private int calculateNumberOfReducers(String input) throws IOException {
 int numberOfReducers = 0;
 Path inputPath = new Path(input);
 FileSystem fs = inputPath.getFileSystem(getConf());
 FileStatus[] statuses = fs.globStatus(inputPath);
 for(FileStatus status: statuses) {
 if(status.isDirectory()) {
 numberOfReducers += getNumberOfInputFiles(status, fs);
 } else if(status.isFile()) {
 numberOfReducers ++;
 }
 }
 return numberOfReducers;
}

/**
 * Recursively determines number of input files in an HDFS directory
 *
 * @param status instance of FileStatus
 * @param fs instance of FileSystem
 * @return number of input files within particular HDFS directory
 * @throws IOException
 */
private int getNumberOfInputFiles(FileStatus status, FileSystem fs) throws IOException {
 int inputFileCount = 0;
 if(status.isDirectory()) {
 FileStatus[] files = fs.listStatus(status.getPath());
 for(FileStatus file: files) {
 inputFileCount += getNumberOfInputFiles(file, fs);
 }
 } else {
 inputFileCount ++;
 }

 return inputFileCount;
}
</code></pre>

blocks|key|528280|text|如果您使用的是hadoop+2.*+API，则有更好的解决方案：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|528281|++++Configuration+conf+=+getConf();
++++Job+job+=+Job.getInstance(conf);
++++FileSystem+fs+=+FileSystem.get(conf);

++++//the+second+boolean+parameter+here+sets+the+recursion+to+true
++++RemoteIterator<LocatedFileStatus>+fileStatusListIterator+=+fs.listFiles(
++++++++++++new+Path("path/to/lib"),+true);
++++while(fileStatusListIterator.hasNext()){
++++++++LocatedFileStatus+fileStatus+=+fileStatusListIterator.next();
++++++++//do+stuff+with+the+file+like+...
++++++++job.addFileToClassPath(fileStatus.getPath());
++++}|code-block|syntax|javascript|528282|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

If you are using hadoop 2.* API there are more elegant solutions:

<pre><code> Configuration conf = getConf();
 Job job = Job.getInstance(conf);
 FileSystem fs = FileSystem.get(conf);

 //the second boolean parameter here sets the recursion to true
 RemoteIterator&lt;LocatedFileStatus&gt; fileStatusListIterator = fs.listFiles(
 new Path("path/to/lib"), true);
 while(fileStatusListIterator.hasNext()){
 LocatedFileStatus fileStatus = fileStatusListIterator.next();
 //do stuff with the file like ...
 job.addFileToClassPath(fileStatus.getPath());
 }
</code></pre>

blocks|key|311389|text|/**
+*+@param+filePath
+*+@param+fs
+*+@return+list+of+absolute+file+path+present+in+given+path
+*+@throws+FileNotFoundException
+*+@throws+IOException
+*/
public+static+List<String>+getAllFilePath(Path+filePath,+FileSystem+fs)+throws+FileNotFoundException,+IOException+{
++++List<String>+fileList+=+new+ArrayList<String>();
++++FileStatus[]+fileStatus+=+fs.listStatus(filePath);
++++for+(FileStatus+fileStat+:+fileStatus)+{
++++++++if+(fileStat.isDirectory())+{
++++++++++++fileList.addAll(getAllFilePath(fileStat.getPath(),+fs));
++++++++}+else+{
++++++++++++fileList.add(fileStat.getPath().toString());
++++++++}
++++}
++++return+fileList;
}|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|311390|快速示例:假设您具有以下文件结构：|unstyled|311391|a++->++b
+++->++c++->+d
++++++++++->+e+
+++->++d++->+f|311392|使用上面的代码，您将获得：|311393|a/b
a/c/d
a/c/e
a/d/f|311394|如果您只需要叶(即fileNames)，请在else块中使用以下代码：|offset|length|style|CODE|311395|+...
++++}+else+{
++++++++String+fileName+=+fileStat.getPath().toString();+
++++++++fileList.add(fileName.substring(fileName.lastIndexOf("/")+%2B+1));
++++}|311396|这将提供：|311397|b
d
e
f|311398|entityMap^0|0|0|0|0|0|M|4|0|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|11|8|@]|9|@]|A|$]]|$1|G|3|H|5|6|7|12|8|@]|9|@]|A|$B|C]]|$1|I|3|J|5|F|7|13|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|14|8|@]|9|@]|A|$B|C]]|$1|M|3|N|5|F|7|15|8|@$O|16|P|17|Q|R]]|9|@]|A|$]]|$1|S|3|T|5|6|7|18|8|@]|9|@]|A|$B|C]]|$1|U|3|V|5|F|7|19|8|@]|9|@]|A|$]]|$1|W|3|X|5|6|7|1A|8|@]|9|@]|A|$B|C]]|$1|Y|3|-4|5|F|7|1B|8|@]|9|@]|A|$]]]|Z|$]]

<pre><code>/**
 * @param filePath
 * @param fs
 * @return list of absolute file path present in given path
 * @throws FileNotFoundException
 * @throws IOException
 */
public static List&lt;String&gt; getAllFilePath(Path filePath, FileSystem fs) throws FileNotFoundException, IOException {
 List&lt;String&gt; fileList = new ArrayList&lt;String&gt;();
 FileStatus[] fileStatus = fs.listStatus(filePath);
 for (FileStatus fileStat : fileStatus) {
 if (fileStat.isDirectory()) {
 fileList.addAll(getAllFilePath(fileStat.getPath(), fs));
 } else {
 fileList.add(fileStat.getPath().toString());
 }
 }
 return fileList;
}
</code></pre>

Quick Example : Suppose you have the following file structure:

<pre><code>a -&gt; b
 -&gt; c -&gt; d
 -&gt; e 
 -&gt; d -&gt; f
</code></pre>

Using the code above, you get:

<pre><code>a/b
a/c/d
a/c/e
a/d/f
</code></pre>

If you want only the leaf (i.e. fileNames), use the following code in <code>else</code> block :

<pre><code> ...
 } else {
 String fileName = fileStat.getPath().toString(); 
 fileList.add(fileName.substring(fileName.lastIndexOf("/") + 1));
 }
</code></pre>

This will give:

<pre><code>b
d
e
f
</code></pre>

blocks|key|311468|text|不要使用递归方法(堆问题)+:)使用队列|type|unstyled|depth|inlineStyleRanges|entityRanges|data|311469|queue.add(param_dir)
while+(queue+is+not+empty){

++directory=++queue.pop
+-+get+items+from+current+directory
+-+if+item+is+file+add+to+a+list+(final+list)
+-+if+item+is+directory+=>+queue.push
}|code-block|syntax|javascript|311470|这很简单，尽情享受吧！|311471|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

don't use recursive approach (heap issues) :)
use a queue

<pre><code>queue.add(param_dir)
while (queue is not empty){

 directory= queue.pop
 - get items from current directory
 - if item is file add to a list (final list)
 - if item is directory =&gt; queue.push
}
</code></pre>

that was easy, enjoy!

blocks|key|311527|text|感谢Radu+Adrian+Moldovan的建议。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|311528|下面是一个使用queue的实现：|311529|private+static+List<String>+listAllFilePath(Path+hdfsFilePath,+FileSystem+fs)
throws+FileNotFoundException,+IOException+{
++List<String>+filePathList+=+new+ArrayList<String>();
++Queue<Path>+fileQueue+=+new+LinkedList<Path>();
++fileQueue.add(hdfsFilePath);
++while+(!fileQueue.isEmpty())+{
++++Path+filePath+=+fileQueue.remove();
++++if+(fs.isFile(filePath))+{
++++++filePathList.add(filePath.toString());
++++}+else+{
++++++FileStatus[]+fileStatus+=+fs.listStatus(filePath);
++++++for+(FileStatus+fileStat+:+fileStatus)+{
++++++++fileQueue.add(fileStat.getPath());
++++++}
++++}
++}
++return+filePathList;
}|code-block|syntax|javascript|311530|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

Thanks Radu Adrian Moldovan for the suggestion.

Here is an implementation using queue:

<pre><code>private static List&lt;String&gt; listAllFilePath(Path hdfsFilePath, FileSystem fs)
throws FileNotFoundException, IOException {
 List&lt;String&gt; filePathList = new ArrayList&lt;String&gt;();
 Queue&lt;Path&gt; fileQueue = new LinkedList&lt;Path&gt;();
 fileQueue.add(hdfsFilePath);
 while (!fileQueue.isEmpty()) {
 Path filePath = fileQueue.remove();
 if (fs.isFile(filePath)) {
 filePathList.add(filePath.toString());
 } else {
 FileStatus[] fileStatus = fs.listStatus(filePath);
 for (FileStatus fileStat : fileStatus) {
 fileQueue.add(fileStat.getPath());
 }
 }
 }
 return filePathList;
}
</code></pre>

blocks|key|528555|text|现在，人们可以使用Spark来做同样的事情，而且它的速度比其他方法(如Hadoop+MR)更快。下面是代码片段。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|528556|def+traverseDirectory(filePath:String,recursiveTraverse:Boolean,filePaths:ListBuffer[String])+{
++++val+files+=+FileSystem.get(+sparkContext.hadoopConfiguration+).listStatus(new+Path(filePath))
++++++++++++files.foreach+{+fileStatus+=>+{
++++++++++++++++if(!fileStatus.isDirectory()+&&+fileStatus.getPath().getName().endsWith(".xml"))+{++++++++++++++++
++++++++++++++++++++filePaths%2B=fileStatus.getPath().toString()++++++
++++++++++++++++}
++++++++++++++++else+if(fileStatus.isDirectory())+{
++++++++++++++++++++traverseDirectory(fileStatus.getPath().toString(),+recursiveTraverse,+filePaths)
++++++++++++++++}
++++++++++++}
++++}+++
}|code-block|syntax|javascript|528557|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Now, one can use Spark to do the same and its way faster than other approaches (such as Hadoop MR). Here is the code snippet.

<pre><code>def traverseDirectory(filePath:String,recursiveTraverse:Boolean,filePaths:ListBuffer[String]) {
 val files = FileSystem.get( sparkContext.hadoopConfiguration ).listStatus(new Path(filePath))
 files.foreach { fileStatus =&gt; {
 if(!fileStatus.isDirectory() &amp;&amp; fileStatus.getPath().getName().endsWith(".xml")) { 
 filePaths+=fileStatus.getPath().toString() 
 }
 else if(fileStatus.isDirectory()) {
 traverseDirectory(fileStatus.getPath().toString(), recursiveTraverse, filePaths)
 }
 }
 } 
}
</code></pre>

blocks|key|528601|text|用于递归和非递归方法的代码片段：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|528602|//helper+method+to+get+the+list+of+files+from+the+HDFS+path
public+static+List<String>
++++listFilesFromHDFSPath(Configuration+hadoopConfiguration,
++++++++++++++++++++++++++String+hdfsPath,
++++++++++++++++++++++++++boolean+recursive)+throws+IOException,
++++++++++++++++++++++++++++++++++++++++IllegalArgumentException
{
++++//resulting+list+of+files
++++List<String>+filePaths+=+new+ArrayList<String>();

++++//get+path+from+string+and+then+the+filesystem
++++Path+path+=+new+Path(hdfsPath);++//throws+IllegalArgumentException
++++FileSystem+fs+=+path.getFileSystem(hadoopConfiguration);

++++//if+recursive+approach+is+requested
++++if(recursive)
++++{
++++++++//(heap+issues+with+recursive+approach)+=>+using+a+queue
++++++++Queue<Path>+fileQueue+=+new+LinkedList<Path>();

++++++++//add+the+obtained+path+to+the+queue
++++++++fileQueue.add(path);

++++++++//while+the+fileQueue+is+not+empty
++++++++while+(!fileQueue.isEmpty())
++++++++{
++++++++++++//get+the+file+path+from+queue
++++++++++++Path+filePath+=+fileQueue.remove();

++++++++++++//filePath+refers+to+a+file
++++++++++++if+(fs.isFile(filePath))
++++++++++++{
++++++++++++++++filePaths.add(filePath.toString());
++++++++++++}
++++++++++++else+++//else+filePath+refers+to+a+directory
++++++++++++{
++++++++++++++++//list+paths+in+the+directory+and+add+to+the+queue
++++++++++++++++FileStatus[]+fileStatuses+=+fs.listStatus(filePath);
++++++++++++++++for+(FileStatus+fileStatus+:+fileStatuses)
++++++++++++++++{
++++++++++++++++++++fileQueue.add(fileStatus.getPath());
++++++++++++++++}+//+for
++++++++++++}+//+else

++++++++}+//+while

++++}+//+if
++++else++++++++//non-recursive+approach+=>+no+heap+overhead
++++{
++++++++//if+the+given+hdfsPath+is+actually+directory
++++++++if(fs.isDirectory(path))
++++++++{
++++++++++++FileStatus[]+fileStatuses+=+fs.listStatus(path);

++++++++++++//loop+all+file+statuses
++++++++++++for(FileStatus+fileStatus+:+fileStatuses)
++++++++++++{
++++++++++++++++//if+the+given+status+is+a+file,+then+update+the+resulting+list
++++++++++++++++if(fileStatus.isFile())
++++++++++++++++++++filePaths.add(fileStatus.getPath().toString());
++++++++++++}+//+for
++++++++}+//+if
++++++++else++++++++//it+is+a+file+then
++++++++{
++++++++++++//return+the+one+and+only+file+path+to+the+resulting+list
++++++++++++filePaths.add(path.toString());
++++++++}+//+else

++++}+//+else

++++//close+filesystem;+no+more+operations
++++fs.close();

++++//return+the+resulting+list
++++return+filePaths;
}+//+listFilesFromHDFSPath|code-block|syntax|javascript|528603|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Code snippet for both recursive and non-recursive approaches:

<pre><code>//helper method to get the list of files from the HDFS path
public static List&lt;String&gt;
 listFilesFromHDFSPath(Configuration hadoopConfiguration,
 String hdfsPath,
 boolean recursive) throws IOException,
 IllegalArgumentException
{
 //resulting list of files
 List&lt;String&gt; filePaths = new ArrayList&lt;String&gt;();

 //get path from string and then the filesystem
 Path path = new Path(hdfsPath); //throws IllegalArgumentException
 FileSystem fs = path.getFileSystem(hadoopConfiguration);

 //if recursive approach is requested
 if(recursive)
 {
 //(heap issues with recursive approach) =&gt; using a queue
 Queue&lt;Path&gt; fileQueue = new LinkedList&lt;Path&gt;();

 //add the obtained path to the queue
 fileQueue.add(path);

 //while the fileQueue is not empty
 while (!fileQueue.isEmpty())
 {
 //get the file path from queue
 Path filePath = fileQueue.remove();

 //filePath refers to a file
 if (fs.isFile(filePath))
 {
 filePaths.add(filePath.toString());
 }
 else //else filePath refers to a directory
 {
 //list paths in the directory and add to the queue
 FileStatus[] fileStatuses = fs.listStatus(filePath);
 for (FileStatus fileStatus : fileStatuses)
 {
 fileQueue.add(fileStatus.getPath());
 } // for
 } // else

 } // while

 } // if
 else //non-recursive approach =&gt; no heap overhead
 {
 //if the given hdfsPath is actually directory
 if(fs.isDirectory(path))
 {
 FileStatus[] fileStatuses = fs.listStatus(path);

 //loop all file statuses
 for(FileStatus fileStatus : fileStatuses)
 {
 //if the given status is a file, then update the resulting list
 if(fileStatus.isFile())
 filePaths.add(fileStatus.getPath().toString());
 } // for
 } // if
 else //it is a file then
 {
 //return the one and only file path to the resulting list
 filePaths.add(path.toString());
 } // else

 } // else

 //close filesystem; no more operations
 fs.close();

 //return the resulting list
 return filePaths;
} // listFilesFromHDFSPath
</code></pre>

I have a folder in hdfs which has two subfolders each one has about 30 subfolders which,finally,each one contains xml files.
I want to list all xml files giving only the main folder's path.
Locally I can do this with <a href="http://commons.apache.org/io/" rel="noreferrer">apache commons-io's</a> FileUtils.listFiles().
I have tried this

<pre><code>FileStatus[] status = fs.listStatus( new Path( args[ 0 ] ) );
</code></pre>

but it only lists the two first subfolders and it doesn't go further.
Is there any way to do this in hadoop?

How to list all files in a directory and its subdirectories in hadoop hdfs

Hadoop 

Spark 

HDFS

我在hdfs中有一个文件夹，其中有两个子文件夹，每个子文件夹大约有30个子文件夹，最后，每个子文件夹都包含xml文件。我想列出所有的xml文件，只给出主文件夹的路径。在本地，我可以使用 FileUtils.listFiles()来做这件事。我已经试过了FileStatus[] status = fs.listStatu...

问如何在hadoop hdfs中列出目录及其子目录中的所有文件
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在hadoop hdfs中列出目录及其子目录中的所有文件EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在hadoop hdfs中列出目录及其子目录中的所有文件
EN