The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.
When the disk utilization exceeds 85% or reaches 100%, the ES cluster or Kibana cannot provide services normally, and the following problems may occur:
When an index request is made, an error similar to {[FORBIDDEN/12/index read-only/allow delete(api)];","type":"cluster_block_exception"} is returned.
1. When an operation is performed on the cluster, an error similar to [FORBIDDEN/13/cluster read-only / allow delete (api)] is returned.
2. The cluster is in the red status. In severe cases, the node may not join the cluster (which can be viewed through the GET _cat/allocation?v command), and there are unassigned shards (which can be viewed through the GET _cat/allocation?v command).
3. The node monitoring page in the ES console shows that the disk utilization of cluster nodes has reached or approached 100%.
Note
If the disk is full, log in to failure may occur, making it impossible to manually clean data. Therefore, it is recommended to configure an alert policy to promptly clean data.
Analysis
The above problems are caused by high disk utilization. The disk utilization of data nodes has the following three thresholds. Exceeding them may affect Elasticsearch or Kibana services.
When the cluster disk utilization exceeds 85%, new shards cannot be assigned.
When the cluster disk utilization exceeds 90%, Elasticsearch will try to migrate the shards on the corresponding node to other data nodes with lower disk utilization.
When the cluster disk utilization exceeds 95%, the system will forcibly set the read_only_allow_delete attribute for each index on the corresponding nodes in the Elasticsearch cluster. At this time, all indexes on the node cannot write data, only read and delete the corresponding indexes.
Solution
Clearing expired cluster data
1. Users can delete expired indexes to free up disk space by accessing Kibana > Dev Tools. Steps are as follows:
Warning
Data cannot be recovered after deletion; therefore, please do so with caution. You can also choose to keep the data, but you need to expand the disk space.
Step 1. Enable batch operation for cluster indexes.
PUT _cluster/settings
{
"persistent":{
"action.destructive_requires_name":"false"
}
}
Step 2. Delete data, such as DELETE NginxLog-12*.
DELETE index-name-*
2. After completing the above steps, if the user's Tencent Cloud Elasticsearch version is earlier than 7.5.1, execute the following command in the Dev Tools page of Kibana:
Disable the read-only status of the index by running the following command:
PUT _all/_settings
{
"index.blocks.read_only_allow_delete": null
}
Disable the read-only status of the cluster by running the following command:
PUT _cluster/settings
{
"persistent":{
"cluster.blocks.read_only_allow_delete": null
}
}
3. Check whether the cluster index is still in the read_only status and whether the index writing has returned to normal.
4. If the cluster is still in the red status, run the following command to check whether there are unassigned shards in the cluster.
GET /_cluster/allocation/explain
5. After the distribution of the shards is completed, check the cluster status. If the cluster status is still red, please contact Tencent Cloud Technical Support through After-sales Support.
6. In order to avoid high disk utilization from affecting Elasticsearch services, we recommend you enable disk utilization monitoring alarms, check the alarm SMS in time, and take precautionary measures. For more information, please see Monitoring and Alert Configuration Suggestions.
Expanding cloud disk space
If users do not want to clean up cluster data, they can also expand disk space in the Cluster Configuration Interface of the Tencent Cloud Elasticsearch Console. The steps are as follows:
1. In the cluster list, click the instance ID/Name you want to configure to enter the instance details page. In the Basic Configuration tab, click Adjust Configuration.
2. Enter the Adjust Configuration page, select the Single Node Data Disk you need to scale out, click Next, and submit the task.
3. If the user's Tencent Cloud Elasticsearch version is before 7.5.1, you also need to execute the following commands in the Dev Tools interface of Kibana:
Disable the read-only status of the index by running the following command:
PUT _all/_settings
{
"index.blocks.read_only_allow_delete": null
}
Disable the read-only status of the cluster by running the following command:
