Use Cases
During your use of CFS, you may need to copy data in many scenarios. This document provides recommended solutions for fast data copying in Linux operating systems.
Scenario One: Data synchronization between CFS file systems, different directories within a file system, and between a file system and a cloud disk.
In daily use, data copying is frequently required. The above three scenarios at the Linux operating system level are all data copying between different directories, and the operation methods are similar. Normally, you can execute the cp command for simple and quick copying. However, if it involves a large number of files, the single-threaded operation mode of cp will seriously slow down the copying progress. In this case, it is recommended to use the method recommended in this document for copying, which can achieve self-concurrent acceleration.
Scenario Two: Data synchronization between the file system and object storage.
When it involves accessing storage or importing data via the public network, it is recommended to use object storage as a transit. You can use various basic tools provided by object storage to upload and download data. The recommended tools are as follows: COSBrowser, COSCMD.
Notes
These operation steps are intended for use in Scenario One. For Scenario Two, please refer to the usage pages of COSBrowser and COSCMD in Object Storage.
Prerequisites
There are source and destination directories that can be migrated on the cloud server.
Note
CFS supports cross-VPC access. You can connect multiple VPCs via Cloud Connect Network (CCN) and then mount them for access.
Instructions
1. Download and install the Rclone tool.
wget https://downloads.rclone.org/v1.53.4/rclone-v1.53.4-linux-amd64.zip --no-check-certificateunzip rclone-v1.53.4-linux-amd64.zipchmod 0755 ./rclone-*/rclonecp ./rclone-*/rclone /usr/bin/rm -rf ./rclone-*
2. Run the following command to copy data:
rclone copy /mnt/src /mnt/dst -Pvv --transfers 32 --checkers 64 --links --create-empty-src-dirs
This tool does not copy metadata information such as owner, ctime, and atime, by default. If you need to copy metadata information, use the
rsync command.Note
The parameters are described as follows; the number of transfers and checkers can be configured according to the system specifications:
transfers: The number of concurrent file transfers (recommended to be no more than twice the number of cores).
checkers: The number of concurrent scans of local files (recommended to be no more than twice the number of cores).
P: Real-time display of the progress of data copying (the progress is refreshed every 500 ms. If
P is not added to the command, the progress is refreshed every minute).links: Soft links for copying.
vv: Print copying logs.
create-empty-src-dirs: Copy empty directories.
3. After data copying is complete, you can view logs to check the results for different files.