SonarQube 数据清理，从100G 到9G
SonarQube 数据清理，从100G 到9G
donghui
发布于 2019-07-16 15:14:16
发布于 2019-07-16 15:14:16
背景描述
SonarQube 自去年使用开始，已运行一年有余 它上面的 Project 数量已超过 1000 因为每个应用的每个 git 分支的每次 push 事件都会触发 Sonar 分析任务 这样会有很多的数据都存储下来，从而导致数据量很大 目前 Mysql 数据库占用磁盘空间达到100G 并且随着时间推移会继续增加 于是在想是否可以进行数据清理，以降低所使用的磁盘空间
如何进行数据清理？
那么如何进行数据清理呢？ 经过不断探索与实践，形成如下解决方案： 1、调小【数据库清理器】相关参数 2、设置 master 为长期分支，并修改【保留短期分支】参数 3、清理非主分支的长期分支 4、MySQL Innodb DATA_FREE 清理
调小【数据库清理器】相关参数
配置—>通用配置 —> 数据库清理器 调小【数据库清理器】相关参数，缩短数据保留时间，以便尽早释放空间
设置 master 为长期分支
配置—>通用配置 —> Branches 默认情况下，名称以 “branch” 或 “release” 开头的分支将被视为长期分支，长期分支系统不会做自动清理。 将 master 设置为长期分支，那么其它分支就是短期分支，短期分支系统会做自动清理。 当然已经存在的长期分支不会自动更改为短期分支，需要先删除它，然后重新在这个分支执行 sonar 分析任务，那么它就会变为短期分支 此外，修改【清除不活跃的短分支前的保留天数】参数，改为10
使用长期分支模式后，每个 Project 分支页面如下：
短期分支会显示和长期分支有差异的 issue
清理非主分支的长期分支
修改为长期分支模式的两个月后，清理非主分支的长期分支 这里使用了 SonarQube API 进行了批量清理，清理任务花费了「三个多小时」～ 相关 python 脚本可以参考：https://github.com/donhui/python-sonarqube/blob/master/sonarqube_branches_clean.py
MySQL Innodb DATA_FREE 清理
对 SonarQube 数据库进行分析：
mysql> SELECT table_schema "DB Name", Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB" FROM information_schema.tables GROUP BY table_schema;
+--------------------------+---------------+
| DB Name | DB Size in MB |
+--------------------------+---------------+
| dingtalk_develop_members | 0.1 |
| information_schema | 0.2 |
| sonar | 90477.8 |
| sonar_statistics | 5.1 |
+--------------------------+---------------+
4 rows in set (0.30 sec)
mysql> SELECT table_name AS `Table`, round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB` FROM information_schema.TABLES WHERE table_schema = 'sonar';
+--------------------------+------------+
| Table | Size in MB |
+--------------------------+------------+
| active_rule_parameters | 0.03 |
| active_rules | 0.42 |
| analysis_properties | 0.02 |
| ce_activity | 32.55 |
| ce_queue | 0.05 |
| ce_scanner_context | 619.05 |
| ce_task_characteristics | 51.19 |
| ce_task_input | 1.52 |
| default_qprofiles | 0.03 |
| duplications_index | 0.05 |
| es_queue | 0.03 |
| events | 6.20 |
| file_sources | 33973.41 |
| group_roles | 2.53 |
| groups | 0.02 |
| groups_users | 0.23 |
| internal_properties | 0.02 |
| issue_changes | 160.84 |
| issues | 372.92 |
| loaded_templates | 0.03 |
| manual_measures | 0.02 |
| metrics | 0.08 |
| notifications | 0.02 |
| org_qprofiles | 0.05 |
| organization_members | 0.05 |
| organizations | 0.03 |
| perm_templates_groups | 0.02 |
| perm_templates_users | 0.02 |
| perm_tpl_characteristics | 0.02 |
| permission_templates | 0.02 |
| plugins | 0.03 |
| project_branches | 1.80 |
| project_links | 4.52 |
| project_measures | 49670.27 |
| project_qprofiles | 0.02 |
| projects | 5483.33 |
| properties | 0.03 |
| qprofile_changes | 1.92 |
| qprofile_edit_groups | 0.02 |
| qprofile_edit_users | 0.02 |
| quality_gate_conditions | 0.02 |
| quality_gates | 0.02 |
| rule_repositories | 0.02 |
| rules | 3.70 |
| rules_metadata | 0.02 |
| rules_parameters | 0.08 |
| rules_profiles | 0.03 |
| schema_migrations | 0.02 |
| snapshots | 7.69 |
| user_roles | 0.78 |
| user_tokens | 0.05 |
| users | 0.14 |
| webhook_deliveries | 0.05 |
+--------------------------+------------+
53 rows in set (0.02 sec)数据量大的表：
| ce_scanner_context | 619.05 |
| file_sources | 33973.41 |
| issue_changes | 160.84 |
| issues | 372.92 |
| project_measures | 49670.27 |
| projects | 5483.33 |清理非主分支的长期分支后，发现 data_length + index_length 下降，但是磁盘空间并没有减少 向 DBA 求助，得到的解决方案为：清理数据库 DATA_FREE，回收表空间
查看表相关的元数据信息，包括 TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE ，SQL 语句及输出如下：
mysql> SELECT TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA='sonar';
+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+
| TABLE_NAME | ROW_FORMAT | TABLE_ROWS | DATA_LENGTH | INDEX_LENGTH | MAX_DATA_LENGTH | DATA_FREE | ENGINE |
+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+
| active_rule_parameters | Dynamic | 138 | 16384 | 16384 | 0 | 0 | InnoDB |
| active_rules | Dynamic | 4385 | 294912 | 147456 | 0 | 0 | InnoDB |
| analysis_properties | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| ce_activity | Dynamic | 74861 | 16793600 | 17334272 | 0 | 4194304 | InnoDB |
| ce_queue | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB |
| ce_scanner_context | Dynamic | 17122 | 649117696 | 0 | 0 | 1286602752 | InnoDB |
| ce_task_characteristics | Dynamic | 228069 | 36290560 | 17383424 | 0 | 5242880 | InnoDB |
| ce_task_input | Dynamic | 0 | 344064 | 0 | 0 | 113246208 | InnoDB |
| default_qprofiles | Dynamic | 14 | 16384 | 16384 | 0 | 0 | InnoDB |
| duplications_index | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB |
| es_queue | Dynamic | 0 | 16384 | 16384 | 0 | 5242880 | InnoDB |
| events | Dynamic | 2882 | 3342336 | 2457600 | 0 | 5242880 | InnoDB |
| file_sources | Dynamic | 168412 | 8828141568 | 281624576 | 0 | 33689698304 | InnoDB |
| group_roles | Dynamic | 7000 | 507904 | 2146304 | 0 | 6291456 | InnoDB |
| groups | Dynamic | 48 | 16384 | 0 | 0 | 0 | InnoDB |
| groups_users | Dynamic | 1060 | 81920 | 163840 | 0 | 0 | InnoDB |
| internal_properties | Dynamic | 3 | 16384 | 0 | 0 | 0 | InnoDB |
| issue_changes | Dynamic | 762484 | 104284160 | 64372736 | 0 | 6291456 | InnoDB |
| issues | Dynamic | 184354 | 200441856 | 164954112 | 0 | 35651584 | InnoDB |
| loaded_templates | Dynamic | 2 | 16384 | 16384 | 0 | 0 | InnoDB |
| manual_measures | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| metrics | Dynamic | 169 | 65536 | 16384 | 0 | 0 | InnoDB |
| notifications | Dynamic | 0 | 16384 | 0 | 0 | 6291456 | InnoDB |
| org_qprofiles | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB |
| organization_members | Dynamic | 393 | 49152 | 0 | 0 | 0 | InnoDB |
| organizations | Dynamic | 1 | 16384 | 16384 | 0 | 0 | InnoDB |
| perm_templates_groups | Dynamic | 4 | 16384 | 0 | 0 | 0 | InnoDB |
| perm_templates_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| perm_tpl_characteristics | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| permission_templates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| plugins | Dynamic | 23 | 16384 | 16384 | 0 | 0 | InnoDB |
| project_branches | Dynamic | 1673 | 229376 | 114688 | 0 | 0 | InnoDB |
| project_links | Dynamic | 26829 | 4734976 | 0 | 0 | 4194304 | InnoDB |
| project_measures | Dynamic | 14545516 | 7061143552 | 10892787712 | 0 | 36237737984 | InnoDB |
| project_qprofiles | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| projects | Dynamic | 398678 | 1495941120 | 1684144128 | 0 | 2999975936 | InnoDB |
| properties | Dynamic | 125 | 16384 | 16384 | 0 | 0 | InnoDB |
| qprofile_changes | Dynamic | 5049 | 1589248 | 425984 | 0 | 4194304 | InnoDB |
| qprofile_edit_groups | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| qprofile_edit_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| quality_gate_conditions | Dynamic | 12 | 16384 | 0 | 0 | 0 | InnoDB |
| quality_gates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| rule_repositories | Dynamic | 37 | 16384 | 0 | 0 | 0 | InnoDB |
| rules | Dynamic | 3185 | 3686400 | 196608 | 0 | 4194304 | InnoDB |
| rules_metadata | Dynamic | 5 | 16384 | 0 | 0 | 0 | InnoDB |
| rules_parameters | Dynamic | 328 | 65536 | 16384 | 0 | 0 | InnoDB |
| rules_profiles | Dynamic | 33 | 16384 | 16384 | 0 | 0 | InnoDB |
| schema_migrations | Dynamic | 237 | 16384 | 0 | 0 | 0 | InnoDB |
| snapshots | Dynamic | 4715 | 1589248 | 327680 | 0 | 2097152 | InnoDB |
| user_roles | Dynamic | 6338 | 507904 | 311296 | 0 | 0 | InnoDB |
| user_tokens | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB |
| users | Dynamic | 438 | 114688 | 32768 | 0 | 0 | InnoDB |
| webhook_deliveries | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB |
+--------------------------+------------+------------+-------------+--------------+-----------------+-------------+--------+
53 rows in set (0.00 sec)数据库清理 DATA_FREE，回收表空间，SQL 语句及输出如下：
mysql> ALTER TABLE file_sources ENGINE=InnoDB;
Query OK, 0 rows affected (2 min 41.44 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE issue_changes ENGINE=InnoDB;
Query OK, 0 rows affected (4.39 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE ce_scanner_context ENGINE=InnoDB;
Query OK, 0 rows affected (14.82 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE project_measures ENGINE=InnoDB;
Query OK, 0 rows affected (3 min 55.74 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE projects ENGINE=InnoDB;
Query OK, 0 rows affected (25.27 sec)
Records: 0 Duplicates: 0 Warnings: 0清理 DATA_FREE 之后，再次查看表相关的元数据信息：
mysql> SELECT TABLE_NAME,ROW_FORMAT,TABLE_ROWS,DATA_LENGTH,INDEX_LENGTH,MAX_DATA_LENGTH,DATA_FREE,ENGINE FROM information_schema.TABLES WHERE TABLE_SCHEMA='sonar';
+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+
| TABLE_NAME | ROW_FORMAT | TABLE_ROWS | DATA_LENGTH | INDEX_LENGTH | MAX_DATA_LENGTH | DATA_FREE | ENGINE |
+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+
| active_rule_parameters | Dynamic | 138 | 16384 | 16384 | 0 | 0 | InnoDB |
| active_rules | Dynamic | 4385 | 294912 | 147456 | 0 | 0 | InnoDB |
| analysis_properties | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| ce_activity | Dynamic | 74971 | 16793600 | 17334272 | 0 | 4194304 | InnoDB |
| ce_queue | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB |
| ce_scanner_context | Dynamic | 17451 | 659079168 | 0 | 0 | 6291456 | InnoDB |
| ce_task_characteristics | Dynamic | 228289 | 36290560 | 17383424 | 0 | 5242880 | InnoDB |
| ce_task_input | Dynamic | 0 | 1589248 | 0 | 0 | 113246208 | InnoDB |
| default_qprofiles | Dynamic | 14 | 16384 | 16384 | 0 | 0 | InnoDB |
| duplications_index | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB |
| es_queue | Dynamic | 0 | 16384 | 16384 | 0 | 5242880 | InnoDB |
| events | Dynamic | 3300 | 3342336 | 2457600 | 0 | 5242880 | InnoDB |
| file_sources | Dynamic | 122205 | 3011493888 | 22626304 | 0 | 5242880 | InnoDB |
| group_roles | Dynamic | 7000 | 507904 | 2146304 | 0 | 6291456 | InnoDB |
| groups | Dynamic | 48 | 16384 | 0 | 0 | 0 | InnoDB |
| groups_users | Dynamic | 1060 | 81920 | 163840 | 0 | 0 | InnoDB |
| internal_properties | Dynamic | 3 | 16384 | 0 | 0 | 0 | InnoDB |
| issue_changes | Dynamic | 731182 | 101318656 | 43089920 | 0 | 6291456 | InnoDB |
| issues | Dynamic | 184366 | 200441856 | 164954112 | 0 | 35651584 | InnoDB |
| loaded_templates | Dynamic | 2 | 16384 | 16384 | 0 | 0 | InnoDB |
| manual_measures | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| metrics | Dynamic | 169 | 65536 | 16384 | 0 | 0 | InnoDB |
| notifications | Dynamic | 0 | 16384 | 0 | 0 | 6291456 | InnoDB |
| org_qprofiles | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB |
| organization_members | Dynamic | 393 | 49152 | 0 | 0 | 0 | InnoDB |
| organizations | Dynamic | 1 | 16384 | 16384 | 0 | 0 | InnoDB |
| perm_templates_groups | Dynamic | 4 | 16384 | 0 | 0 | 0 | InnoDB |
| perm_templates_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| perm_tpl_characteristics | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| permission_templates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| plugins | Dynamic | 23 | 16384 | 16384 | 0 | 0 | InnoDB |
| project_branches | Dynamic | 1686 | 229376 | 114688 | 0 | 0 | InnoDB |
| project_links | Dynamic | 26866 | 4734976 | 0 | 0 | 4194304 | InnoDB |
| project_measures | Dynamic | 10381786 | 2282733568 | 1306165248 | 0 | 51380224 | InnoDB |
| project_qprofiles | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| projects | Dynamic | 374448 | 314392576 | 132235264 | 0 | 4194304 | InnoDB |
| properties | Dynamic | 125 | 16384 | 16384 | 0 | 0 | InnoDB |
| qprofile_changes | Dynamic | 5049 | 1589248 | 425984 | 0 | 4194304 | InnoDB |
| qprofile_edit_groups | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| qprofile_edit_users | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| quality_gate_conditions | Dynamic | 12 | 16384 | 0 | 0 | 0 | InnoDB |
| quality_gates | Dynamic | 0 | 16384 | 0 | 0 | 0 | InnoDB |
| rule_repositories | Dynamic | 37 | 16384 | 0 | 0 | 0 | InnoDB |
| rules | Dynamic | 3185 | 3686400 | 196608 | 0 | 4194304 | InnoDB |
| rules_metadata | Dynamic | 5 | 16384 | 0 | 0 | 0 | InnoDB |
| rules_parameters | Dynamic | 328 | 65536 | 16384 | 0 | 0 | InnoDB |
| rules_profiles | Dynamic | 33 | 16384 | 16384 | 0 | 0 | InnoDB |
| schema_migrations | Dynamic | 237 | 16384 | 0 | 0 | 0 | InnoDB |
| snapshots | Dynamic | 4737 | 1589248 | 327680 | 0 | 4194304 | InnoDB |
| user_roles | Dynamic | 6338 | 507904 | 311296 | 0 | 0 | InnoDB |
| user_tokens | Dynamic | 34 | 16384 | 32768 | 0 | 0 | InnoDB |
| users | Dynamic | 438 | 114688 | 32768 | 0 | 0 | InnoDB |
| webhook_deliveries | Dynamic | 0 | 16384 | 32768 | 0 | 0 | InnoDB |
+--------------------------+------------+------------+-------------+--------------+-----------------+-----------+--------+
53 rows in set (0.06 sec)从上面可以看出，相关表（如：project_measures，projects，file_sources）的 DATA_FREE 的大小有很大幅度下降。 然后，查看数据库所占磁盘空间，也有大幅度下降：100G —> 9G 。
至此，SonarQube 的数据清理已取得很好的成果，以此为记。
(adsbygoogle = window.adsbygoogle || []).push({});
本文参与 腾讯云自媒体分享计划，分享自作者个人站点/博客。
如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
目录
相关产品与服务
云数据库 SQL Server
腾讯云数据库 SQL Server （TencentDB for SQL Server）是业界最常用的商用数据库之一，对基于 Windows 架构的应用程序具有完美的支持。TencentDB for SQL Server 拥有微软正版授权，可持续为用户提供最新的功能，避免未授权使用软件的风险。具有即开即用、稳定可靠、安全运行、弹性扩缩等特点。