MySQL 大数据操作注意事项

MySQL 大数据操作注意事项

http://netkiller.github.io/journal/mysql.parallel.html

摘要


目录

  • 1. 关于 delete
  • 2. 关于 update
  • 3. 关于创建索引
  • 4. 关于 OPTIMIZE
  • 5. 关于切换引擎
  • 6. 确保SELECT不被受阻

1. 关于 delete

delete from mytable 必死无疑,你需要分批删除,尽量缩小每个批次删除的记录数,delete 是可以并行执行的,你可以同时运行多个删除操作

		mysql> show processlist;
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
| Id     | User            | Host                | db        | Command | Time  | State                       | Info                                                   |
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
|      1 | event_scheduler | localhost           | NULL      | Daemon  |    52 | Waiting for next activation | NULL                                                   |
| 115986 | dba             | localhost           | example   | Query   |     0 | NULL                        | show processlist                                       |
| 117446 | dba             | localhost           | example   | Query   |    20 | updating                    | delete from mytable where OPEN_TIME like '2011.11.28%' |
| 117525 | dba             | localhost           | example   | Query   |     2 | updating                    | delete from mytable where OPEN_TIME like '2011.12.02%' |
| 117526 | dba             | localhost           | example   | Query   |    49 | updating                    | delete from mytable where OPEN_TIME like '2011.12.12%' |
| 117527 | dba             | localhost           | example   | Query   |     6 | updating                    | delete from mytable where OPEN_TIME like '2011.12.21%' |
| 117528 | dba             | localhost           | example   | Query   |    64 | updating                    | delete from mytable where OPEN_TIME like '2011.12.30%' |
| 117546 | dba             | localhost           | example   | Query   |    33 | updating                    | delete from mytable where OPEN_TIME like '2011.11.10%' |
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
23 rows in set (0.00 sec)		

2. 关于 update

在电商领域常常遇到一个问题“调价”,经常需要调整一批商品的价格, 程序猿一条语句搞定有没有?

update goods set price=price+10 where category_id = xxx		

在开发,测试环境是可以通过测试的,一旦部署到生产环境,必死无疑

3. 关于创建索引

大表创建索引需要很久的时间,通常要经历 manage keys 与 copy to tmp table 的过程

mysql> show processlist;
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
| Id     | User            | Host                | db       | Command | Time  | State                       | Info                                                             |
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
|      1 | event_scheduler | localhost           | NULL     | Daemon  |    47 | Waiting for next activation | NULL                                                             |
| 115986 | dba             | localhost           | example  | Query   |     0 | NULL                        | show processlist                                                 |
| 118814 | dba             | 192.168.6.20:50459  | example  | Query   |     8 | copy to tmp table           | ALTER TABLE `mytable` ADD INDEX `modifiy_time` (`MODIFY_TIME`)   |
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
17 rows in set (0.00 sec)		

删除索引,也需要经理 copy to tmp table 过程,漫长的等待

		mysql> show processlist;
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
| Id     | User            | Host                | db           | Command | Time  | State                       | Info                                            |
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
|      1 | event_scheduler | localhost           | NULL         | Daemon  |    11 | Waiting for next activation | NULL                                            |
| 115986 | dba             | localhost           | example      | Query   |     0 | NULL                        | show processlist                                |
| 118814 | dba             | 192.168.6.20:50459  | example      | Query   |     4 | copy to tmp table           | ALTER TABLE `mytable`	DROP INDEX `modifiy_time` |
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
17 rows in set (0.00 sec)		

所以数据设计要深思熟虑,做到提前未雨绸缪,不要亡羊补牢

4. 关于 OPTIMIZE

OPTIMIZE 的操作是将当前表复制到临时表操作后再删除当前表,最后将临时表改名

		mysql> show processlist;
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
| Id     | User            | Host                | db                        | Command | Time  | State                       | Info                     |
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
|      1 | event_scheduler | localhost           | NULL                      | Daemon  |    14 | Waiting for next activation | NULL                     |
| 115835 | dba             | 192.168.6.20:49664  | example                   | Query   |     9 | copy to tmp table           | OPTIMIZE TABLE `mytable` |
| 115986 | dba             | localhost           | example                   | Query   |     0 | NULL                        | show processlist         |
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
17 rows in set (0.00 sec)		

5. 关于切换引擎

转换ENGINE从MyISAM到InnoDB会经历creating table然后copy to tmp table在修改表名几个阶段,过程非常缓慢

		mysql> show processlist;
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| Id   | User            | Host                | db      | Command | Time  | State                       | Info                                     |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
|    1 | event_scheduler | localhost           | NULL    | Daemon  |    10 | Waiting for next activation | NULL                                     |
| 3167 | dba             | 192.168.6.20:56723  | example | Query   |     2 | creating table              | ALTER TABLE `mytable`	ENGINE=InnoDB   |
| 3172 | dba             | localhost           | example | Query   |     0 | NULL                        | show processlist                         |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
18 rows in set (0.00 sec)		

copy to tmp table 过程

		mysql> show processlist;
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| Id   | User            | Host                | db      | Command | Time  | State                       | Info                                     |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
|    1 | event_scheduler | localhost           | NULL    | Daemon  |    21 | Waiting for next activation | NULL                                     |
| 3167 | dba             | 192.168.6.20:56723  | example | Query   |    13 | copy to tmp table           | ALTER TABLE `mytable`	ENGINE=InnoDB   |
| 3172 | dba             | localhost           | example | Query   |     0 | NULL                        | show processlist                         |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
18 rows in set (0.00 sec)		

此时我们查看mysql data目录会看到临时表文件

# ll /var/lib/mysql/hx9999_real_history/
		
-rw-rw---- 1 mysql mysql      9522 May 16 17:17 #sql-c2f_c5f.frm
-rw-rw---- 1 mysql mysql        48 May 16 17:17 #sql-c2f_c5f.par
-rw-rw---- 1 mysql mysql 637534208 May 16 17:29 #sql-c2f_c5f#P#p0.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p1.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p2.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p3.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p4.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p5.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p6.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p7.ibd		

6. 确保SELECT不被受阻

使用各种手段保证select操作不被受阻,只要select一直可以查询网站前端就能提供80%的功能,一旦select受阻一切都是浮云。

保证 select 操作优先于其他操作

UPDATE [LOW_PRIORITY] [IGNORE] tbl_name  
SET col_name1=expr1 [, col_name2=expr2 ...]  
[WHERE where_definition]  
[ORDER BY ...]  
[LIMIT row_count]		

update的时候增加 LOW_PRIORITY 参数,可以降低更新语句的优先级。

my.cnf

[mysqld]		
low_priority_updates=1		

或者启动是添加--low-priority-updates参数

全局开启

SET @@global.low_priority_updates = 1;		

适用于本次会话连接

SET @@session.low_priority_updates = 1;

原文发布于微信公众号 - Netkiller(netkiller-ebook)

原文发表时间:2015-09-21

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏数据和云

书接上文:薛定谔的猫是如何诞生的?

编辑手记:注重细节,是DBA必要的基本素质要求。 书接上文(参考:空与非空 - 数据库中也有薛定谔的猫?),其实CBO的判断本身是没有问题的,问题在于,为什...

29310
来自专栏杨建荣的学习笔记

数据库文件的迁移

关于数据库中的文件迁移,需要考虑普通数据文件,redo日志文件(还需要考虑是否为current状态),undo表空间,临时表空间,system,sysaux表空...

3634
来自专栏pangguoming

Spring Security OAuth2 Demo

Spring Security OAuth2 Demo 项目使用的是MySql存储, 需要先创建以下表结构: CREATE SCHEMA IF NOT EXIS...

7857
来自专栏沃趣科技

应用示例荟萃 | performance_schema全方位介绍(下)

使用performance_schema中的语句当前事件记录表和语句事件历史记录表可以查询数据库中最近执行的一些SQL语句,以及语句相关的信息,这里我们以eve...

1853
来自专栏数据和云

【动手实践】Oracle 12.2新特性:多列列表分区和外部表分区

在Oracle 12.2版本中,增加了大量的分区新特性,这其中包括: 自动的列表分区创建 在线的普通表转换分区表 支持只读分区和读写分区混合 以下介绍的三个特...

3025
来自专栏我的博客

ubuntu安装puref-ftpd

1.安装pureftpd sudo apt-get install pure-ftpd sudo apt-get install pure-ftpd-mysql...

2914
来自专栏杨建荣的学习笔记

浅谈exp/imp(上) (r5笔记第81天)

作为DBA,经常需要在不同数据库环境间做数据的导入导出,exp/imp就是这样的轻便快捷的客户端工具,可以很方便的在不同数据库之间转移数据对象,即使数据库位于不...

2748
来自专栏Danny的专栏

无法删除数据库,因为该数据库当前正在使用

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/huyuyang6688/article/...

1992
来自专栏用户2442861的专栏

基于Spring + Spring MVC + Mybatis 高性能web构建

http://blog.csdn.net/zoutongyuan/article/details/41379851#comments

1701
来自专栏数据库新发现

Oracle9i新特性-索引监视及注意事项[修正版]

Last Updated: Saturday, 2004-12-04 10:28 Eygle

833

扫码关注云+社区