MySQL 大數據操作注意事項
http://netkiller.github.io/journal/mysql.parallel.html
版權聲明
轉載請與作者聯繫，轉載時請務必標明文章原始出處和作者信息及本聲明。
|
|
|
微信掃瞄二維碼進入 Netkiller 微信訂閲號 QQ群：128659835 請註明“讀者” |
2014-05-16
摘要
1. 關於 delete
delete from mytable 必死無疑，你需要分批刪除，儘量縮小每個批次刪除的記錄數，delete 是可以並行執行的，你可以同時運行多個刪除操作
mysql> show processlist; +--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 52 | Waiting for next activation | NULL | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | | 117446 | dba | localhost | example | Query | 20 | updating | delete from mytable where OPEN_TIME like '2011.11.28%' | | 117525 | dba | localhost | example | Query | 2 | updating | delete from mytable where OPEN_TIME like '2011.12.02%' | | 117526 | dba | localhost | example | Query | 49 | updating | delete from mytable where OPEN_TIME like '2011.12.12%' | | 117527 | dba | localhost | example | Query | 6 | updating | delete from mytable where OPEN_TIME like '2011.12.21%' | | 117528 | dba | localhost | example | Query | 64 | updating | delete from mytable where OPEN_TIME like '2011.12.30%' | | 117546 | dba | localhost | example | Query | 33 | updating | delete from mytable where OPEN_TIME like '2011.11.10%' | +--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+ 23 rows in set (0.00 sec)
2. 關於 update
在電商領域常常遇到一個問題“調價”，經常需要調整一批商品的價格, 程序猿一條語句搞定有沒有？
update goods set price=price+10 where category_id = xxx
在開發，測試環境是可以通過測試的，一旦部署到生產環境，必死無疑
3. 關於創建索引
大表創建索引需要很久的時間，通常要經歷 manage keys 與 copy to tmp table 的過程
mysql> show processlist; +--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 47 | Waiting for next activation | NULL | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | | 118814 | dba | 192.168.6.20:50459 | example | Query | 8 | copy to tmp table | ALTER TABLE `mytable` ADD INDEX `modifiy_time` (`MODIFY_TIME`) | +--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+ 17 rows in set (0.00 sec)
刪除索引，也需要經理 copy to tmp table 過程，漫長的等待
mysql> show processlist; +--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 11 | Waiting for next activation | NULL | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | | 118814 | dba | 192.168.6.20:50459 | example | Query | 4 | copy to tmp table | ALTER TABLE `mytable` DROP INDEX `modifiy_time` | +--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+ 17 rows in set (0.00 sec)
所以數據設計要深思熟慮，做到提前未雨綢繆，不要亡羊補牢
4. 關於 OPTIMIZE
OPTIMIZE 的操作是將當前表複製到臨時表操作後再刪除當前表，最後將臨時表改名
mysql> show processlist; +--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 14 | Waiting for next activation | NULL | | 115835 | dba | 192.168.6.20:49664 | example | Query | 9 | copy to tmp table | OPTIMIZE TABLE `mytable` | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | +--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+ 17 rows in set (0.00 sec)
5. 關於切換引擎
轉換ENGINE從MyISAM到InnoDB會經歷creating table然後copy to tmp table在修改表名幾個階段，過程非常緩慢
mysql> show processlist; +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 10 | Waiting for next activation | NULL | | 3167 | dba | 192.168.6.20:56723 | example | Query | 2 | creating table | ALTER TABLE `mytable` ENGINE=InnoDB | | 3172 | dba | localhost | example | Query | 0 | NULL | show processlist | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ 18 rows in set (0.00 sec)
copy to tmp table 過程
mysql> show processlist; +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 21 | Waiting for next activation | NULL | | 3167 | dba | 192.168.6.20:56723 | example | Query | 13 | copy to tmp table | ALTER TABLE `mytable` ENGINE=InnoDB | | 3172 | dba | localhost | example | Query | 0 | NULL | show processlist | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ 18 rows in set (0.00 sec)
此時我們查看mysql data目錄會看到臨時表檔案
# ll /var/lib/mysql/hx9999_real_history/ -rw-rw---- 1 mysql mysql 9522 May 16 17:17 #sql-c2f_c5f.frm -rw-rw---- 1 mysql mysql 48 May 16 17:17 #sql-c2f_c5f.par -rw-rw---- 1 mysql mysql 637534208 May 16 17:29 #sql-c2f_c5f#P#p0.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p1.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p2.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p3.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p4.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p5.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p6.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p7.ibd
6. 確保SELECT不被受阻
使用各種手段保證select操作不被受阻，只要select一直可以查詢網站前端就能提供80%的功能，一旦select受阻一切都是浮雲。
保證 select 操作優先於其他操作
UPDATE [LOW_PRIORITY] [IGNORE] tbl_name SET col_name1=expr1 [, col_name2=expr2 ...] [WHERE where_definition] [ORDER BY ...] [LIMIT row_count]
update的時候增加 LOW_PRIORITY 參數，可以降低更新語句的優先順序。
my.cnf
[mysqld] low_priority_updates=1
或者啟動是添加--low-priority-updates參數
全局開啟
SET @@global.low_priority_updates = 1;
適用於本次會話連接
SET @@session.low_priority_updates = 1;