If you need to migrate your HDFS raw data to EMR, you can achieve using either of the following: migrate data with Tencent Cloud Object Storage (COS) service as a transfer stop; migrate data with DistCp, a built-in tool of Hadoop for large inter/intra-cluster copying. This document describes how to migrate data with the first method.
Non-HDFS Original Data
If your original data is not an HDFS file but rather from another formatted source, you can upload it to COS using either the COS web console or the API provided by COS. Afterward, the uploaded data can be analyzed in the EMR cluster. Please refer to Local Data Migration to COS for more information about data transfer via COS.
Migrating an HDFS file
1. Acquire the COS migration tool.
Obtain the migration tool, for more migration tools, please refer to the Tool Overview.
2. Tool configuration.
The configuration files are all placed in the conf directory of the tool directory, copy the core-site.xml of the HDFS cluster that needs to be synchronized to the conf, which includes the configuration information of the NameNode, edit the configuration file cos_info.conf, including appid, bucket, region, and key information.
Caution
It's advised that users use the sub-account key, abide by the principle of minimal authority explanation, to prevent the leakage of resources outside of the intended storage bucket or object.
If you absolutely must use a permanent key, it's recommended to abide by the principle of minimal authority explanation to limit the scope of permissions for the permanent key.
Explanation of Command Parameters:
-ak <ak> the cos secret id // User's SecretId. It is recommended to use a sub-account key, adhering to the principle of least privilege to minimize usage risks. For obtaining a sub-account key, refer to: https://cloud.tencent.com/document/product/598/37140-appid,--appid <appid> the cos appid-bucket,--bucket <bucket_name> the cos bucket name-cos_info_file,--cos_info_file <arg> the cos user info config default is ./conf/cos_info.conf-cos_path,--cos_path <cos_path> the absolute cos folder path-h,--help print help message-hdfs_conf_file,--hdfs_conf_file <arg> the hdfs info config default is ./conf/core-site.xml-hdfs_path,--hdfs_path <hdfs_path> the hdfs path-region,--region <region> the cos region. legal value cn-south, cn-east, cn-north, sg-sk <sk> the cos secret key // User's SecretKey. It is recommended to use a sub-account key, adhering to the principle of least privilege to minimize usage risks. For obtaining a sub-account key, refer to: https://cloud.tencent.com/document/product/598/37140-skip_if_len_match,--skip_if_len_match skip upload if hadoop file length match cos
3. Executing Migration:
# All operations must be performed within the tool directory. If both the configuration file and command line parameters are set, the command line parameters will take precedence../hdfs_to_cos_cmd -h# Copying from HDFS to COS (If a file already exists on COS, it will be overwritten)./hdfs_to_cos_cmd --hdfs_path=/tmp/hive --cos_path=/hdfs/20170224/# Copying from HDFS to COS, if the length of the file to be copied and COS is consistent, the upload is ignored (applicable for re-copying after the initial copy).# Here, only the length is evaluated, as calculating the file summary on Hadoop would be too costly../hdfs_to_cos_cmd --hdfs_path=/tmp/hive --cos_path=/hdfs/20170224/ -skip_if_len_match# Setting parameters entirely through the command line../hdfs_to_cos_cmd -appid 1252xxxxxx -akAKIDVt55xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -skKS08jDVbVElxxxxxxxxxxxxxxxxxxxxxxxxxx -bucket test -cos_path /hdfs-hdfs_path /data/data -region cn-south -hdfs_conf_file/home/hadoop/hadoop-2.8.1/etc/hadoop/core-site.xml
4. Upon executing the verification command, the following logs are outputted:
[Folder Operation Result : [ 53(sum)/ 53(ok) / 0(fail)]][File Operation Result: [22(sum)/ 22(ok) / 0(fail) / 0(skip)]][Used Time: 3 s]
Sum denotes the total number of files that need to be migrated.
Ok signifies the number of files successfully migrated.
Fail represents the number of files for which the migration failed.
Skip denotes the count of files skipped due to the uploaded file and the identically named file having the same length, after adding the skip_if_len_match parameter.
You may also sign in to the COS console to determine if the data has been rightly migrated. For the guidelines on using COS, see Getting Started.
FAQs
Please ensure that the configuration information you provide, including the appID, key information, bucket, and region, is accurate, and that the machine's time aligns with Beijing time (a discrepancy of about 1 minute is normal). If the difference is substantial, please adjust the machine's time.
Please ensure that the machine where the copy program is located can also connect to the DateNode. Although the NameNode can be connected via an external IP, the DateNode machine where the block is located has an internal IP and cannot be connected. Therefore, it is recommended to execute the synchronization program on a node of Hadoop to ensure access to both the NameNode and DateNode.
Regarding permission issues, use the current account to download files using Hadoop commands to check for normalcy, then use the synchronization tool to sync data on Hadoop.
For files that already exist on COS, they are overwritten by default unless the user explicitly specifies -skip_if_len_match, in which case the upload is skipped if the file lengths match.
The COS path is considered a directory, and all files copied from HDFS will ultimately be stored in this directory.