java.util.concurrent.RejectedExecutionException异常
java.util.concurrent.RejectedExecutionException异常
MickyInvQ
发布于 2021-01-21 06:29:05
发布于 2021-01-21 06:29:05
文章被收录于专栏：InvQ的专栏
场景
业务一切正常，突然收到一堆告警，发现全是
java.util.concurrent.RejectedExecutionException异常报错。
具体看了下代码，里面的执行逻辑也不难，没有外部依赖都是内存多线程cpu类型计算的逻辑。
下面是报错的线程池状态变化，由上到下，按照时间增长，最后达到饱和。
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 99, active threads = 75, queued tasks = 5000, completed tasks = 1595259]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 101, active threads = 74, queued tasks = 4999, completed tasks = 1595298]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 90, active threads = 70, queued tasks = 4998, completed tasks = 1595354]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 93, active threads = 75, queued tasks = 5000, completed tasks = 1595378]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 85, active threads = 72, queued tasks = 5000, completed tasks = 1595443]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 83, active threads = 70, queued tasks = 5000, completed tasks = 1595447]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 90, active threads = 81, queued tasks = 5000, completed tasks = 1595476]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 85, active threads = 74, queued tasks = 4994, completed tasks = 1595574]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 91, active threads = 80, queued tasks = 4999, completed tasks = 1595678]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 95, active threads = 85, queued tasks = 5000, completed tasks = 1595696]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 95, active threads = 87, queued tasks = 5000, completed tasks = 1595721]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 93, active threads = 85, queued tasks = 4999, completed tasks = 1595727]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 94, active threads = 85, queued tasks = 4998, completed tasks = 1595737]
...
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 109, active threads = 96, queued tasks = 4999, completed tasks = 1596150]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 113, active threads = 102, queued tasks = 4998, completed tasks = 1596191]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 109, queued tasks = 5000, completed tasks = 1596355]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 110, queued tasks = 5000, completed tasks = 1596517]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 119, active threads = 110, queued tasks = 4999, completed tasks = 1596519]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 113, active threads = 105, queued tasks = 5000, completed tasks = 1596553]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 122, active threads = 109, queued tasks = 4999, completed tasks = 1596602]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]因为cpu4核，所以coresize = 40,maxsize = 120，都正常。
private static BlockingQueue<Runnable> executorQueue = new LinkedBlockingQueue<>(5000);
private static ExecutorService executorService = new ThreadPoolExecutor(
Runtime.getRuntime().availableProcessors() * 10,
Runtime.getRuntime().availableProcessors() * 30,
60, TimeUnit.SECONDS, executorQueue);通过对ThreadPoolExecutor类分析，引发java.util.concurrent.RejectedExecutionException主要有两种原因：
- 线程池显示的调用了shutdown()之后，再向线程池提交任务的时候，如果你配置的拒绝策略是ThreadPoolExecutor.AbortPolicy的话，这个异常就被会抛出来。
- 当你的排队策略为有界队列，并且配置的拒绝策略是ThreadPoolExecutor.AbortPolicy，当线程池的线程数量已经达到了maximumPoolSize的时候，你再向它提交任务，就会抛出ThreadPoolExecutor.AbortPolicy异常。
排查
因为全机房、只有一个节点报错，而其他机器都正常，在这个一定保证正确的前提下。 1、看了线上的请求量、并没有突增，所以排除外部因素 2、看了代码逻辑，逻辑内部并没有打印报错日志，说明不是线程执行耗时导致后面的其他线程排队 3、排除线程池是否提前关闭。并没有，因为手动没有显示关闭，另外看日志也知道里面的线程数还在变化，所以不存在关闭的说法。
考虑是否是机器本身的原因，后面经过排查，看到那个点线上的cpu使用率突然升高、系统负载突然飙升，网卡流出、流入报文数目、tcp连接数也突然为0。具体干了什么，已经让运营去看了。至少说明我们代码没有问题。
如下图，可以感受下：
在这里插入图片描述
解决
线上第一时间出现这个问题，没想2秒，看了下报错代码逻辑，然后重启了这个节点实例（当时重启完后，观察又没问题了，带着疑惑然后才去排查）
本文参与 腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。
原始发表：2021/01/18 ，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
目录
腾讯云开发者
