COS Migration is an all-in-one tool that integrates the COS data migration feature. You can use it to migrate local data to COS through simple configurations and steps. It has the following features:
Checkpoint restart: restarting uploads from checkpoints is supported. For large files, if the upload exits halfway or service failure occur, you can run the tool again to restart the upload.
Multipart upload: An object can be upload to COS in multiple parts.
Parallel upload: Multiple objects can be upload at the same time.
Migration verification: Migrate objects can be verified.
Note
COS Migration only support UF-8 for encoding.
If you use this tool to upload a file that already has the same name, the existing file will be overwrite. You need to configure the tool to skip files with the same name.
COS Migration is used for one-time migration but is not suitable for continuous sync. For example, if files are added locally every day and need to be continuously synced to COS, then in order to avoid repeated migration tasks, COS Migration will save the records of successful migration. In case of continuous sync, the record scanning time will keep increasing. We recommend you use COSBrowser as described in User Guide for Desktop Version for this scenario.
Decompress the package and save it to a directory, for example:
C:\Users\Administrator\Downloads\cos_migrate
Linux
Decompress the package and save it to a directory, for example:
unzip cos_migrate_tool_v5-master.zip && cd cos_migrate_tool_v5-master
Migration tool structure
The structure of the properly decompressed COS Migration folder is as follow:
COS_Migrate_tool
|——conf # Directory of the configuration file
| |——config.ini # Migration configuration file
|——db # Record of storage migration successes
|——dep #Main logic compiled JAR package
|——log # Log generated during tool execution
|——opbin # Script for compiling
|——result #Directory for saving successful migration records, with the file name format "date.out" and the record format "absolute path\tfile size\tlast modified time"
|——src # Source code of the tool
|——tmp # Temporary file storage directory
|── .gitignore # Files and folders ignored by Git version control
|——pom.xml # Project configuration file
|——README # Readme document
|——start_migrate.sh # Migration startup script for Linux
|——start_migrate.bat # Migration startup script for Windows
Note
The db directory mainly records the IDs of files successfully migrated by the tool. For each migration task, the IDs in db will be compared first, and if the ID of the current file has already been recorded, the current file will be skipped; otherwise, it will be migrated.
The log directory keeps all the logs generated during migration. If an error occurs, please check the error.log in this directory first.
3. Modify the config.ini configuration file
Before running the migration startup script, you need to modify the config.ini file (path: ./conf/config.ini) first. This file contains the following parts:
3.1 Configure the migration type
The "type" represents the migration type, and users should fill in the corresponding identifier based on their migration requirements. For example, if you need to migrate local data to COS, the configuration content for [migrateType] is type=migrateLocal.
[migrateType]
type=migrateLocal
Currently supported migration types include:
migrateType
Description
migrateLocal
From local system to COS
3.2 Configure the migration task
You can configure a migration task based on your actual migration needs. Main configuration items include the destination COS bucket and task properties.
# The public configuration section of the migration tool contains the account information for the target COS to which the data will be migrated.
[common]
secretId=COS_SECRETID
secretKey=COS_SECRETKEY
bucketName=examplebucket-1250000000
region=ap-guangzhou
storageClass=Standard
cosPath=/
https=off
tmpFolder=./tmp
smallFileThreshold=5242880
smallFileExecutorNum=64
bigFileExecutorNum=8
entireFileMd5Attached=on
daemonMode=off
daemonModeInterVal=60
executeTimeWindow=00:00,24:00
outputFinishedFileFolder=./result
resume=false
skipSamePath=false
Name
Description
Default value
secretId
SecretId of your key. Replace COS_SECRETID with your real key information, which can be obtained on the TencentCloud API key page in the CAM Console
-
secretKey
SecretKey of your key. Replace COS_SECRETKEY with your real key information, which can be obtained on the TencentCloud API key page in the CAM Console
-
bucketName
Destination bucket name: The naming format is <BucketName-APPID>, meaning the bucket name must include the APPID, such as examplebucket-1250000000.
-
region
Region information of the destination bucket. For the abbreviations for COS regions, see Regions and Access Endpoints.
-
storageClass
After data migration, the available storage types are Standard (STANDARD storage), Standard_IA (IA storage), Archive (ARCHIVE storage), Maz_Standard (Multi-AZ STANDARD storage), and Maz_Standard_IA (Multi-AZ IA storage). For more information, see Storage Class Overview.
Standard
cosPath
The COS path to migrate to. / represents migrating to the root path of the Bucket, while /folder/doc/ indicates migrating to the /folder/doc/ directory within the Bucket. If /folder/doc/ does not exist, the path will be created automatically.
/
https
Whether to transfer via HTTPS. on: Yes, off: No. It takes time to enable transfer via HTTPS, which is suitable for scenarios that demand high security.
off
tmpFolder
During the migration process from other cloud storage to COS, a temporary file directory is used to store intermediate files, which will be deleted after the migration is complete. The required format is an absolute path:
In Linux, the separator is a single forward slash, for example, /a/b/c.
In Windows, the separator is two backslashes, for example: E:\\a\\b\\c
By default, it is the tmp directory in the tool's location path.
./tmp
smallFileThreshold
Number of bytes as the threshold for small files. If the data size is higher than or equal to this threshold, multipart upload is used; otherwise, simple upload is used. The default value is 5 MB
5242880
smallFileExecutorNum
Concurrency for uploading small files (smaller than smallFileThreshold) via simple upload. Decrease the concurrency if files are uploaded to COS via public network with low bandwidth
64
bigFileExecutorNum
Concurrency for uploading large files (greater than or equal to smallFileThreshold) via multipart upload. Decrease the concurrency if files are uploaded to COS via public network with low bandwidth
8
entireFileMd5Attached
This indicates whether the migration tool calculates the MD5 of the entire file and stores it in the custom header "x-cos-meta-md5" of the file for subsequent verification, because the etag of a large file uploaded to COS in multiple parts is not the MD5 of the entire file
on
daemonMode
Whether to enable daemon mode. on: yes; off: no. In daemon mode, the program will keep performing synchronization. The synchronization interval is configured by the daemonModeInterVal parameter
off
daemonModeInterVal
This indicates the interval in seconds between two rounds of synchronization
60
executeTimeWindow
Execution time window: The time granularity is in minutes, and this parameter defines the time period during which the migration tool runs daily. For example:
Parameters 03:30, 21:00 indicate that the task will be executed between 03:30 AM and 09:00 PM. During other times, the task will enter a sleep state, pausing the migration while preserving its progress until the next time window when it will automatically resume. Note that the latter time must be greater than the former time.
00:00,24:00
outputFinishedFileFolder
This directory stores the successful migration results. The result files are named by date, such as ./result/2021-05-27.out, where ./result is the created directory. Each line in the file content is formatted as: absolute path\tfile size\tlast modified time. If set to empty, no results will be output.
./result
resume
Whether to continue with the result of the last run and traverse through the list of files from the source. The tool starts from scratch by default.
false
skipSamePath
Whether to skip the current file if a file with the same name already exists in COS. By default, the tool does not skip the current file: it overwrites the existing file.
false
requestTryCount
Total number of attempts for each file upload.
5
3.3 Configure the data source
Configure the corresponding sections based on the [migrateType] migration type. For example, if the configuration content of [migrateType] is type=migrateLocal, the user only needs to configure the [migrateLocal] section.
3.3.1 Configure a local data source "migrateLocal"
If you are migrating from a local system to COS, configure this section. The specific configuration items and descriptions are as follows:
Local directory, required in absolute path format:
In Linux, the separator is a single forward slash, for example, /a/b/c.
In Windows, the separator is two backslashes, for example: E:\\a\\b\\c
Caution: Only the directory path can be entered for this parameter, not the specific file path. Otherwise, it will cause an error in parsing the target object name. In the case of cosPath=/, it will be incorrectly parsed as a bucket creation request.
excludes
Absolute path of the directory or file to be excluded, which means that some directories or files in the localPath are not to be migrated. Multiple absolute paths are separated by semicolons. If this is left blank, all files in the localPath are to be migrated
ignoreModifiedTimeLessThanSeconds
Exclude files that have an update time less than a certain period of time from the current time (in seconds). This item is left blank by default, indicating files are not to be filtered by the time specified by lastmodified. It is suitable for scenarios where you run the migration tool while updating files and don't want files being updated to be migrated to COS. For example, if it is configured as 300, only files updated at least 5 minutes ago will be uploaded.
4. Run the migration tool
Windows
Double-click start_migrate.bat to run it.
Linux
1. Read the configuration from the config.ini file. The run command is:
sh start_migrate.sh
2. Read the configuration of some parameters from the command line. The run command is:
sh start_migrate.sh -Dcommon.cosPath=/savepoint0403_10/
Note
The tool supports reading configuration items in two ways: command line or configuration file.
The command line takes precedence over the configuration file, i.e., for the same configuration item, the parameter in the command line takes precedence.
Using configuration options in the command line allows users to run different migration tasks simultaneously, provided that the key configuration items in the two tasks are not identical, such as bucket names, COS paths, and source paths to be migrated. Different migration tasks write to different db directories, ensuring concurrent migration. Please refer to the db information in the tool structure mentioned earlier.
The configuration item format is -D{sectionName}.{sectionKey}={sectionValue}, where sectionName is the section name in the configuration file, sectionKey represents the configuration item name within the section, and sectionValue represents the configuration item value within the section. For example, to set the COS path for migration, use -Dcommon.cosPath=/bbb/ddd.
Migration Mechanism and Process
How Migration Works
COS Migration has a status. Successful migrations will be recorded in the format of KV in the leveldb file under the db directory. Before each migration, check whether the path to which data is migrated has been recorded in the db directory. If yes, and its attribute is the same as that in db, the migration will be skipped; otherwise, the migration will be executed. The attribute varies by migration type. For local migration, mtime determines whether to migrate. For migration from other cloud storage services and bucket replication, the etag and length of the source file determine whether to migrate. Therefore, we search for records of successful migrations in the db directory rather than in COS. If a file is deleted or modified via COSCMD or the console rather than the migration tool, the migration tool cannot detect this change, and the file will not be re-migrated.
Migration Steps
1. The configuration file is read, the corresponding configuration section is read according to the migration type, and parameters are checked.
2. The IDs of the files to be migrated are scanned for and compared in the db directory according to the migration type to decide whether upload is allowed.
3. During the migration process, the execution results will be printed, where "inprogress" indicates ongoing migration, "skip" means skipped, "fail" denotes failure, "ok" signifies success, and "condition_not_match" represents files skipped due to not meeting migration conditions (such as lastmodifed and excludes). Detailed information about failures can be found in the error logs. The execution process is illustrated in the following diagram:
4. Upon completion of the entire migration, statistical information will be printed, including the cumulative number of successful migrations, failures, skipped files, and time consumed. For failed cases, please check the error log or rerun the migration tool, as it will skip successfully migrated files and retry the unsuccessful ones. The result illustration is shown in the following figure:
FAQs
If you encounter migration failures, runtime errors, or other abnormal situations while using the COS Migration tool, please refer to Common Issues with COS Migration Tool for solutions.
Summary
Of course, COS not only offers the above applications and services but also provides a variety of popular open-source applications integrated with Tencent Cloud COS plugins. Click here to launch and start using them immediately!
