Selection of HDFS Federation Management Type
Note
The current HDFS Federation Management is open to a whitelist. If needed, you can submit a service ticket to enable it.
HDFS Federation Management Architecture
HDFS Federation utilizes multiple independent NameNode/Namespaces to enable horizontal scaling of HDFS naming services. In the federation, each DataNode is used by all NameNodes as a common storage block location; every DataNode registers with all NameNodes in its cluster, periodically sending heartbeat and block information reports, while also processing instructions from all NameNodes.
ViewFs Federation Principle
To facilitate the management of multiple namespaces, HDFS ViewFs Federation employs the classic Client Side Mount Table (community ViewFs feature). This maps different application paths to specific NameServices on the client side, thereby achieving storage separation or performance separation scenarios. HDFS ViewFs Federation uses a Client-Side Mount Table to distribute files and load, a method that requires more manual intervention (explicit planning) to achieve optimal load balancing.
Router-based Federation Principle
Router-based Federation provides a software layer to manage multiple NameSpace spaces. Compared to ViewFs, which maintains mount table information on the client side, Router-based Federation truly achieves complete transparency for the client. This is because this part of the mapping information will be additionally saved and persisted.
HDFS Federation Management Configuration
Two main aspects need to be considered: the required number of NameServices and the method of mounting business data directories.
Principles for Planning the Number of NameServices
1. The secure storage capacity limit for a single NameService is 100 million files. If this number is exceeded, the access speed and read/write throughput of the NameService will be significantly reduced. Therefore, if the file storage volume (anticipated) exceeds 100 million files, a new NameService needs to be added.
2. For heavy usage of HDFS (i.e., extensive reading and writing of files in HDFS), it is necessary to allocate a separate NameService to handle its requests. This ensures that the application's data can monopolize all the processing capabilities of the NameService, and also avoids the application's impact on other applications.
3. For applications with stringent reliability requirements, a separate NameService can be allocated to prevent high read/write frequencies from other applications causing this application to be unable to access HDFS, resulting in instability of the application's business operations.
Principles for Planning the Mounting Method of Business Data Directories
1. Data directories related to business data services should ideally be mounted under the same NameService. Otherwise, the file read/write speed across NameServices is slower, which can reduce the storage performance of the application.
2. For large volumes of data that are unrelated to the business of other services, a single NameService can be directly utilized.
3. For businesses with smaller volumes, it is recommended to directly mount their directories in the default NameService (HDFS${clusterid}, HDFS concatenated with numeric form clusterid). This eliminates the need for data migration and reduces the complexity of configuring for Federation.
4. It is recommended to map only the global top-level directories to NameService in order to reduce configuration complexity.
Comparative Analysis of Solutions
ViewFs Federation Type
1. Direct Use of ViewFs
Advantages: Unified view, different applications have the same usage method.
Disadvantages: Changes to the ViewFs mount table require all applications using the cluster to synchronously read the latest mount points.
2. Specifying NameService
Preference: There is no need to synchronously change the configuration of all applications.
Drawback: Different components and applications need to clearly define their respective usage directories. Scenarios where component paths are interlinked will become more complex.
Configuration Method for Router-based Federation Type
1. Direct Use of Router-based Federation
Advantages: Unified view, different applications have the same usage method. Compared to ViewFs, changes to the Router-based Federation mount table take effect directly, without the need for applications using the cluster to synchronously update the mount table.
Disadvantages: When the client accesses, an additional DFSRouter forwarding layer is added, which slightly impacts performance.
2. Specifying NameService
Advantages: There is no need to synchronously change the configuration of all applications.
Disadvantages: Different components and applications need to clearly define their respective usage directories. Scenarios where component paths are interlinked will become more complex.