The EMR console supports automatic detection of local disk replacement events. After the disk is replaced, you can independently initialize the new disk operation via the console.
Note
Upon receiving a CVM disk failure notification and subsequently repairing the physical disk or replacing the disk according to the CVM notification content, the EMR console can then initiate the 'Disk Repair' operation.
Following a disk replacement, the data on the disk will be lost. Please ensure that the data on the disk has been backed up.
Instructions
1. Log in to the EMR Console, and in the cluster list, click on the corresponding Cluster ID/Name to enter the cluster details page.
2. In the cluster details page, select Cluster Resources > Resource Management, and perform the Disk Repair operation on the node for which the disk has been replaced.
3. During the operation, services on the current node will be restarted, rendering the services and node unavailable during the restart process. It is recommended to perform the repair operation during off-peak business hours.
Kudu Service Recovery
Note
When there are multiple local disks, and one or more disks have been repaired and the EMR disk repair function is used, the node where the KuduServer service is deployed;
Due to the capabilities of Kudu's fs_data_dirs, if one or more disks have been formatted, to ensure the normal startup of KuduServer, it only supports that all data directories configured on the KuduServer node are empty. Customers need to assist in confirming these data directories, and apart from Kudu storage data, they have not been misused by other customer's own businesses.
Scenario:
Specifically, in the EMR console cluster service, the health status of the KuduServer in the node where the disk has been replaced is "Unavailable".
Confirm Data Consistency and Recovery:
1.1 Confirm that the data in the directory (specific viewing method is as follows) has no other use besides Kudu; if there are other uses, please first migrate the relevant data to other directories not configured by fs_data_dirs, and then perform the following operations.
Specific directory: View the file /usr/local/service/kudu/conf/tserver.gflags:
1.2 Log into the node with abnormal local disk to view the log: /data/emr/kudu/log/kudu-tserver.INFO:
Execute the following command as root user to clean up the related inconsistent data:
rm -rf /data/emr/kudu/tserver/*
rm -rf /data1/emr/kudu/tserver/*
This command assumes that fs_data_dirs is configured as /data/emr/kudu/tserver/, /data1/emr/kudu/tserver/. The specifics can be viewed according to /usr/local/service/kudu/conf/tserver.gflags.
1.3 Observe the status of the KuduServer service.
Note
If you encounter any issues during the operation, please submit a ticket promptly for us to verify and address.
