首页
学习
活动
专区
圈层
工具
发布
首页
学习
活动
专区
圈层
工具
MCP广场
社区首页 >问答首页 >为什么当Hangfire并行运行多个作业时,MySQL InnoDB会造成这么多死锁?

为什么当Hangfire并行运行多个作业时,MySQL InnoDB会造成这么多死锁?
EN

Stack Overflow用户
提问于 2018-12-19 15:32:37
回答 1查看 872关注 0票数 1

在我的asp.net核心应用程序中,我使用了带有MySQL数据库存储的Hangfire。我有一个端点,当访问它时,它会在后台安排一个挂起的作业。当我对这个端点进行加载测试时,如果我发送了40多个并发请求,此代码BackgroundJob.Schedule<IJobSchedulerCallbacks>(s => s.ScheduleSomeCode();将开始抛出以下异常:

代码语言:javascript
运行
复制
Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details. ---> MySql.Data.MySqlClient.MySqlException: Deadlock found when trying to get lock; try restarting transaction
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int64& insertedId)
at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)
at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader)
at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command)
at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType)
at Hangfire.MySql.MySqlWriteOnlyTransaction.<>c__DisplayClass14_0.<AddToSet>b__0(MySqlConnection x)
at Hangfire.MySql.MySqlWriteOnlyTransaction.<Commit>b__29_0(MySqlConnection connection)
at Hangfire.MySql.MySqlStorage.<>c__DisplayClass18_0.<UseTransaction>b__0(MySqlConnection connection)
at Hangfire.MySql.MySqlStorage.UseConnection[T](Func`2 func)
at Hangfire.MySql.MySqlStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)
at Hangfire.MySql.MySqlStorage.UseTransaction(Action`1 action)
at Hangfire.MySql.MySqlWriteOnlyTransaction.Commit()
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass7_0.<CreateWithFilters>b__0()
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
--- End of inner exception stack trace ---
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
at Hangfire.BackgroundJobClientExtensions.Schedule[T](IBackgroundJobClient client, Expression`1 methodCall, TimeSpan delay)
at Hangfire.BackgroundJob.Schedule[T](Expression`1 methodCall, TimeSpan delay)

当我使用下面的命令:SHOW ENGINE INNODB STATUS检查innodb日志时,我得到了以下日志:

代码语言:javascript
运行
复制
=====================================
2018-12-19 14:37:29 0x2ab9c5591700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 53 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 2441 srv_active, 0 srv_shutdown, 13392 srv_idle
srv_master_thread log flush and writes: 15830
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 7531
OS WAIT ARRAY INFO: signal count 8029
RW-shared spins 0, rounds 15152, OS waits 6763
RW-excl spins 0, rounds 15133, OS waits 270
RW-sx spins 58, rounds 1734, OS waits 37
Spin rounds per wait: 15152.00 RW-shared, 15133.00 RW-excl, 29.90 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2018-12-19 13:41:01 0x2aba11f50700
*** (1) TRANSACTION:
TRANSACTION 88410, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 443, OS thread handle 46979012679424, query id 374494 172.31.25.222 cpdbuser update
INSERT INTO `Set` (`Key`, `Value`, `Score`) VALUES (''schedule'', ''475'', 1545313257) ON DUPLICATE KEY UPDATE `Score` = 1545313257
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 147 page no 4 n bits 176 index IX_Set_Key_Value of table `cp-hangfire`.`Set` trx id 88410 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 103 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 0: len 8; hex 7363686564756c65; asc schedule;;
 1: len 3; hex 343736; asc 476;;
 2: len 4; hex 80000088; asc     ;;

*** (2) TRANSACTION:
TRANSACTION 88408, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 457, OS thread handle 46978653554432, query id 374490 172.31.25.222 cpdbuser update
INSERT INTO `Set` (`Key`, `Value`, `Score`) VALUES (''schedule'', ''474'', 1545313257) ON DUPLICATE KEY UPDATE `Score` = 1545313257
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 147 page no 4 n bits 176 index IX_Set_Key_Value of table `cp-hangfire`.`Set` trx id 88408 lock_mode X locks gap before rec
Record lock, heap no 103 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 0: len 8; hex 7363686564756c65; asc schedule;;
 1: len 3; hex 343736; asc 476;;
 2: len 4; hex 80000088; asc     ;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 147 page no 4 n bits 176 index IX_Set_Key_Value of table `cp-hangfire`.`Set` trx id 88408 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 103 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 0: len 8; hex 7363686564756c65; asc schedule;;
 1: len 3; hex 343736; asc 476;;
 2: len 4; hex 80000088; asc     ;;

*** WE ROLL BACK TRANSACTION (1)

请注意,这两个非常简单的事务只使用一个insert命令就创建了死锁:

代码语言:javascript
运行
复制
INSERT INTO `Set` (`Key`, `Value`, `Score`) VALUES (''schedule'', ''475'', 1545313257) ON DUPLICATE KEY UPDATE `Score` = 1545313257
INSERT INTO `Set` (`Key`, `Value`, `Score`) VALUES (''schedule'', ''474'', 1545313257) ON DUPLICATE KEY UPDATE `Score` = 1545313257

以下是Set表模式:

下面是Set表对Value和Score列的唯一索引:

我找到了这种堆叠溢出的答案,上面说即使在完全正常的情况下,mysql也会造成死锁,而我觉得这很奇怪。无论如何,作为一种解决方案,我尝试使用波莉实现指数备份重试策略,这是一个很好的库。但是这只是推迟了错误,因为现在重新尝试调度作业的代码,在第3次重试之后,客户端连接就会因为30秒nginx响应超时而被删除。

第一个问题:当调度作业的简单命令同时执行时,MySQL为什么会开始死锁?

第二个问题如果innodb确实会在正常情况下创建死锁,那么在预期具有更多并发用户的生产数据库中如何使用MySql呢?我漏掉了什么吗?

(来自评论)

代码语言:javascript
运行
复制
CREATE TABLE `Set` (
    `Id` int(11) NOT NULL AUTO_INCREMENT, 
    `Key` varchar(100) NOT NULL, 
    `Value` varchar(256) NOT NULL, 
    `Score` double DEFAULT NULL, 
    `ExpireAt` datetime DEFAULT NULL, 
    PRIMARY KEY (`Id`), 
    UNIQUE KEY `IX_Set_Key_Value` (`Key`,`Value`)
) ENGINE=InnoDB AUTO_INCREMENT=143 DEFAULT CHARSET=latin1
EN

回答 1

Stack Overflow用户

发布于 2018-12-19 15:49:37

第一个问题: --我不知道Hangfire,但不太可能只在CoreBackgroundJobFactory.Create中运行一个insert查询。它至少可以在另一个表上执行select,该表可以自己锁定,并且在这两个进程上的组合可以锁定自己。

第二个问题: Innodb锁定策略取决于事务隔离级别,如果您正在运行高并发环境,可以降低隔离级别:它将降低死锁概率。然而,一些副作用可能会出现,即使在我个人的经验中,我甚至没有使用READ_UNCOMMITED。您可以尝试将其添加到Hangfire数据源配置中,并查看所发生的情况。

票数 1
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/53854450

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档