前往小程序，Get更优阅读体验！
立即前往
文档建议反馈控制台
首页
学习
活动
专区
工具
TVP
最新优惠活动
文章/答案/技术大牛
发布
首页
学习
活动
专区
工具
TVP最新优惠活动
返回腾讯云官网
社区首页 >专栏 >weblogic 11g StuckThreadMaxTime 问题解决 以及 线程池、数据库连接池参数调优

weblogic 11g StuckThreadMaxTime 问题解决 以及 线程池、数据库连接池参数调优

作者头像
MickyInvQ
发布于 2020-09-27 17:18:51
1.5K0
发布于 2020-09-27 17:18:51
举报
文章被收录于专栏：InvQ的专栏InvQ的专栏

问题背景

Java程序中使用原生POI 进行excel导出，数据量可能比较大。

问题日志

代码语言：javascript
复制
<2018-4-24 下午03时56分09秒 CST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "626" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@ad1721[
GET /xxx/xxx/xxx.do HTTP/1.1
Connection: keep-alive
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Referer: http://localhost:7001/zx.html
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.8
Cookie: WLS_HTTP_BRIDGE=bJZHhpZbGJMjt5qd9Mxpcgxb1FBxDZLYp4BTt7sgFTGYhN2GMbz9!-1353724554

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-21 "[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1674)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.preEmit(Saver.java:1286)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.emit(Saver.java:1185)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.emitElement(Saver.java:957)
    org.apache.xmlbeans.impl.store.Saver.processElement(Saver.java:396)
    org.apache.xmlbeans.impl.store.Saver.process(Saver.java:296)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.saveToString(Saver.java:1854)
    org.apache.xmlbeans.impl.store.Cursor._xmlText(Cursor.java:544)
    org.apache.xmlbeans.impl.store.Cursor.xmlText(Cursor.java:2426)
    ^-- Holding lock: org.apache.xmlbeans.impl.store.Locale@149d860[thin lock]
    org.apache.xmlbeans.impl.values.XmlObjectBase.xmlText(XmlObjectBase.java:1485)
    org.apache.xmlbeans.impl.values.XmlObjectBase.toString(XmlObjectBase.java:1472)
    ^-- Holding lock: org.apache.xmlbeans.impl.store.Locale@149d860[thin lock]
    org.apache.poi.xssf.usermodel.extensions.XSSFCellBorder.equals(XSSFCellBorder.java:180)
    java.util.ArrayList.indexOf(ArrayList.java:210)
    org.apache.poi.xssf.model.StylesTable.putBorder(StylesTable.java:448)
    org.apache.poi.xssf.usermodel.XSSFCellStyle.setBottomBorderColor(XSSFCellStyle.java:946)
    org.apache.poi.xssf.usermodel.XSSFCellStyle.setBottomBorderColor(XSSFCellStyle.java:935)

问题分析：

此问题是由于处理请求超时引起的，系统配置的处理时间是600s，但是该线程处理了626后，仍然没将请求释放，所以报了这个错误。如果发送该请求较多，很有可能会导致weblogic的线程阻塞，严重会引起weblogic挂起现象。

问题解决：

1）修改StuckThreadMaxTime参数，将默认的600s改成1200s，或者其它适合的值。

这里写图片描述
这里写图片描述

2）增大线程数，防止线程阻塞问题。 3）优化程序，减少处理时间。

其他解决方案（从sql优化入手）

该异常出现的原因是资源请求的时间超出了weblogic设定的600s，造成资源排队请求，如果类似的操作很多的话，那么会造成大面积的资源请求队列，从而引起weblogic无法正常提供服务，严重时引起weblogic崩溃。那么这种原因是如何导致的呢？ 首先，我们从测试服务器上发现，出现这种情况的原因是因为该请求的时间过长，于是从该请求的数据处理过程入手进行分析，发现该请求的sql语句，在sql/plus下执行时间过长，如下： 如表大概10W数据，在sql/plus下执行时间超长，造成请求weblogic反应时间超出默认值，从而引起资源排队请求的问题，引起服务器不稳定运行。那么出现了这种问题，怎么解决呢？我们的解决方法是对该sql语句进行优化处理： 1:对INFO_SIGN,PPMC等字段建立规范表，从数据库中进行查询，尽量减少in的使用 2:对<>等操作符不使用,使用> or <等方式来代替 3:尽量减少排序order by，rownum的使用，只在关键时刻进行使用，其他时刻能够不使用的就不进行使用。 通过以上方式来减少资源请求时间，从而减少以上异常的发生，来保证服务器的正常运行。

线程池、数据库连接池调优

线程池

修改config.xml中的<self-tuning-thread-pool-size-min>

如下：

代码语言：javascript
复制
  <server>
    <name>AdminServer</name>
    <reverse-dns-allowed>false</reverse-dns-allowed>
    <native-io-enabled>true</native-io-enabled>
    <thread-pool-percent-socket-readers>33</thread-pool-percent-socket-readers>
    <ssl>
      <login-timeout-millis>25000</login-timeout-millis>
    </ssl>
    <max-open-sock-count>-1</max-open-sock-count>
    <stuck-thread-max-time>1200</stuck-thread-max-time>
    <stuck-thread-timer-interval>60</stuck-thread-timer-interval>
    <self-tuning-thread-pool-size-min>50</self-tuning-thread-pool-size-min>
    <self-tuning-thread-pool-size-max>300</self-tuning-thread-pool-size-max>
    <listen-address></listen-address>
    <accept-backlog>300</accept-backlog>
    <login-timeout-millis>5000</login-timeout-millis>
    <low-memory-time-interval>3600</low-memory-time-interval>
    <low-memory-sample-size>10</low-memory-sample-size>
    <low-memory-granularity-level>5</low-memory-granularity-level>
    <low-memory-gc-threshold>5</low-memory-gc-threshold>
  </server>
数据库连接池：

在你的domin下，config下，jdbc下，对应的数据源xml配置中 增加

代码语言：javascript
复制
 <jdbc-connection-pool-params>
    <initial-capacity>2</initial-capacity>
    <max-capacity>15</max-capacity>
    <capacity-increment>1</capacity-increment>
    <test-table-name>SQL SELECT 1</test-table-name>
    <statement-cache-size>10</statement-cache-size>
    <statement-cache-type>LRU</statement-cache-type>
  </jdbc-connection-pool-params>

在控制台中如下操作：

这里写图片描述
这里写图片描述

——————————————日志分隔符————————————————————————

完整相关日志如下：

代码语言：javascript
复制
Tue Apr 24 16:04:44 2018
BEA JRockit(R) R27.6.5-32_o-121899-1.6.0_14-20091001-2107-windows-ia32

"Main Thread" id=1 idx=0x4 tid=5492 prio=5 alive, in native, waiting
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x03E4FFE8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:849)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x03E4FFE8[fat lock]
    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:465)
    at weblogic/Server.main(Server.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"(Signal Handler)" id=2 idx=0x8 tid=7444 prio=5 alive, in native, daemon

"(GC Main Thread)" id=3 idx=0xc tid=9156 prio=5 alive, in native, daemon

"(GC Worker Thread 1)" id=? idx=0x10 tid=8700 prio=5 alive, in native, daemon

"(GC Worker Thread 2)" id=? idx=0x14 tid=9708 prio=5 alive, in native, daemon

"(GC Worker Thread 3)" id=? idx=0x18 tid=1280 prio=5 alive, in native, daemon

"(GC Worker Thread 4)" id=? idx=0x1c tid=7756 prio=5 alive, in native, daemon

"(Code Generation Thread 1)" id=4 idx=0x20 tid=8888 prio=5 alive, in native, native_waiting, daemon

"(Code Optimization Thread 1)" id=5 idx=0x24 tid=492 prio=5 alive, in native, native_waiting, daemon

"(VM Periodic Task)" id=6 idx=0x28 tid=4764 prio=10 alive, in native, daemon

"(Attach Listener)" id=7 idx=0x2c tid=4772 prio=5 alive, in native, daemon

"Finalizer" id=8 idx=0x30 tid=9244 prio=8 alive, in native, native_waiting, daemon
    at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
    at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
    at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Reference Handler" id=9 idx=0x34 tid=1936 prio=10 alive, in native, native_waiting, daemon
    at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
    at java/lang/ref/Reference.access$100(Reference.java:11)
    at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"(Sensor Event Thread)" id=10 idx=0x38 tid=8836 prio=5 alive, in native, daemon

"JDWP Transport Listener: dt_Socket" id=11 idx=0x3c tid=8400 prio=10 alive, in native, daemon

"JDWP Event Helper Thread" id=12 idx=0x40 tid=452 prio=10 alive, in native, native_waiting, daemon

"Timer-0" id=15 idx=0x44 tid=9472 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x03E39DE8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at java/util/TimerThread.mainLoop(Timer.java:483)
    ^-- Lock released while waiting: java/util/TaskQueue@0x03E39DE8[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Timer-1" id=16 idx=0x48 tid=6184 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x02C81B80[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/util/TimerThread.mainLoop(Timer.java:509)
    ^-- Lock released while waiting: java/util/TaskQueue@0x02C81B80[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=17 idx=0x4c tid=9948 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x02C8BAC0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x02C8BAC0[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"weblogic.time.TimeEventGenerator" id=18 idx=0x50 tid=9340 prio=9 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x02FF7D70[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
    ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x02FF7D70[fat lock]
    at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"JMAPI event thread" id=19 idx=0x54 tid=8376 prio=5 alive, in native, native_waiting, daemon

"weblogic.timers.TimerThread" id=20 idx=0x58 tid=1064 prio=9 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/timers/internal/TimerThread@0x030065D0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:267)
    ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0x030065D0[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
代码语言：javascript
复制
"Thread-7" id=24 idx=0x60 tid=1112 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x02E3B590
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x64 tid=1844 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x68 tid=484 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x6c tid=932 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=28 idx=0x70 tid=5992 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"VDE Transaction Processor Thread" id=31 idx=0x74 tid=3728 prio=2 alive, in native, waiting, daemon
    -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x03F42670[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
    ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x03F42670[fat lock]
    at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"DoSManager" id=33 idx=0x7c tid=9032 prio=6 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"LDAPConnThread-0 ldap://10.20.8.67:13001" id=34 idx=0x80 tid=8316 prio=5 alive, in native, daemon
    at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
    at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)
    at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
    at java/net/SocketInputStream.read(SocketInputStream.java:129)
    at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)
    at java/io/BufferedInputStream.read(BufferedInputStream.java:237)
    ^-- Holding lock: java/io/BufferedInputStream@0x04343BF0[thin lock]
    at netscape/ldap/ber/stream/BERElement.getElement(BERElement.java:101)
    at netscape/ldap/LDAPConnThread.run(LDAPConnThread.java:538)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-11" id=35 idx=0x84 tid=1972 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x042C5FC8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-12" id=36 idx=0x88 tid=5996 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x04DF2408
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=37 idx=0x8c tid=6408 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x052046C8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x052046C8[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ConnectionStateManager-0" id=38 idx=0x90 tid=7476 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x046F7BE8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
    at org/apache/curator/framework/state/ConnectionStateManager.processEvents(ConnectionStateManager.java:175)
    at org/apache/curator/framework/state/ConnectionStateManager.access$000(ConnectionStateManager.java:42)
    at org/apache/curator/framework/state/ConnectionStateManager$1.call(ConnectionStateManager.java:108)
    at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303)
    at java/util/concurrent/FutureTask.run(FutureTask.java:138)
    at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (10.20.8.46:2181)" id=39 idx=0x94 tid=3140 prio=5 alive, in native, daemon
    at sun/nio/ch/PollArrayWrapper.poll0(JIJ)I(Native Method)
    at sun/nio/ch/PollArrayWrapper.poll(PollArrayWrapper.java:74)
    at sun/nio/ch/WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:63)
    at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    ^-- Holding lock: sun/nio/ch/Util$1@0x0478EFE8[thin lock]
    ^-- Holding lock: java/util/Collections$UnmodifiableSet@0x0478EF60[thin lock]
    ^-- Holding lock: sun/nio/ch/WindowsSelectorImpl@0x0478E6E8[thin lock]
    at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:80)
    at org/apache/zookeeper/ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    at org/apache/zookeeper/ClientCnxn$SendThread.run(ClientCnxn.java:1081)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'-EventThread" id=40 idx=0x98 tid=8732 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x03DC06A0
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/apache/zookeeper/ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"CuratorFramework-0" id=41 idx=0x9c tid=7176 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x046E6908
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:160)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:39)
    at org/apache/curator/framework/imps/CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:652)
    at org/apache/curator/framework/imps/CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:55)
    at org/apache/curator/framework/imps/CuratorFrameworkImpl$3.call(CuratorFrameworkImpl.java:243)
    at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303)
    at java/util/concurrent/FutureTask.run(FutureTask.java:138)
    at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-13" id=42 idx=0xa0 tid=10248 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: org/apache/commons/httpclient/util/IdleConnectionTimeoutThread@0x03DC9830[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/apache/commons/httpclient/util/IdleConnectionTimeoutThread.run(IdleConnectionTimeoutThread.java:108)
    ^-- Lock released while waiting: org/apache/commons/httpclient/util/IdleConnectionTimeoutThread@0x03DC9830[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-14" id=43 idx=0xa4 tid=10356 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x091D05C0
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:63)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:33)
    at kafka/utils/IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
    at kafka/utils/IteratorTemplate.hasNext(IteratorTemplate.scala:58)
    at com/paic/groupOperateAnalysis/service/impl/SearchEngine$1.run(SearchEngine.java:75)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"metrics-meter-tick-thread-1" id=45 idx=0xa8 tid=10372 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"metrics-meter-tick-thread-2" id=46 idx=0xac tid=10376 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ZkClient-EventThread-48-30.4.139.159:35450,30.4.139.160:35450,30.4.139.161:35450" id=48 idx=0xb4 tid=10388 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x093C0D98
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/I0Itec/zkclient/ZkEventThread.run(ZkEventThread.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-14-SendThread(30.4.139.161:35450)" id=49 idx=0xb8 tid=10392 prio=5 alive, in native, daemon
    at sun/nio/ch/PollArrayWrapper.poll0(JIJ)I(Native Method)
    at sun/nio/ch/PollArrayWrapper.poll(PollArrayWrapper.java:74)
    at sun/nio/ch/WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:63)
    at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    ^-- Holding lock: sun/nio/ch/Util$1@0x093C1800[thin lock]
    ^-- Holding lock: java/util/Collections$UnmodifiableSet@0x093C17F0[thin lock]
    ^-- Holding lock: sun/nio/ch/WindowsSelectorImpl@0x093C15E0[thin lock]
    at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:80)
    at org/apache/zookeeper/ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    at org/apache/zookeeper/ClientCnxn$SendThread.run(ClientCnxn.java:1081)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-14-EventThread" id=50 idx=0xbc tid=10396 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x0B306AA8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/apache/zookeeper/ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"kafka-consumer-scheduler-0" id=51 idx=0xc0 tid=10404 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-18" id=52 idx=0xc4 tid=10416 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x066516C0
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:63)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:33)
    at kafka/utils/IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
    at kafka/utils/IteratorTemplate.hasNext(IteratorTemplate.scala:58)
    at com/paic/groupOperateAnalysis/service/impl/SearchEngine$1.run(SearchEngine.java:75)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554155067-8b86727d_watcher_executor" id=53 idx=0xc8 tid=10420 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
    at kafka/consumer/ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:544)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ZkClient-EventThread-54-30.4.139.159:35450,30.4.139.160:35450,30.4.139.161:35450" id=54 idx=0xcc tid=10424 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x06163D10
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/I0Itec/zkclient/ZkEventThread.run(ZkEventThread.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-18-SendThread(30.4.139.159:35450)" id=55 idx=0xd0 tid=10428 prio=5 alive, in native, daemon
    at sun/nio/ch/PollArrayWrapper.poll0(JIJ)I(Native Method)
    at sun/nio/ch/PollArrayWrapper.poll(PollArrayWrapper.java:74)
    at sun/nio/ch/WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:63)
    at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    ^-- Holding lock: sun/nio/ch/Util$1@0x0664C770[thin lock]
    ^-- Holding lock: java/util/Collections$UnmodifiableSet@0x0664C760[thin lock]
    ^-- Holding lock: sun/nio/ch/WindowsSelectorImpl@0x0664C550[thin lock]
    at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:80)
    at org/apache/zookeeper/ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    at org/apache/zookeeper/ClientCnxn$SendThread.run(ClientCnxn.java:1081)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-18-EventThread" id=56 idx=0xd4 tid=10432 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x0664E310
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/apache/zookeeper/ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"kafka-consumer-scheduler-0" id=57 idx=0xd8 tid=10436 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554158815-ae234a20_watcher_executor" id=58 idx=0xdc tid=10440 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
    at kafka/consumer/ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:544)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-1" id=60 idx=0xe4 tid=10564 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3AF970[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3AF970[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-2" id=61 idx=0xe8 tid=10568 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B01A0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B01A0[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-3" id=62 idx=0xec tid=10572 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B0978[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B0978[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-4" id=63 idx=0xf0 tid=10576 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1150[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1150[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-5" id=64 idx=0xf4 tid=10580 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1928[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1928[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-6" id=65 idx=0xf8 tid=10584 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B2100[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B2100[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-7" id=66 idx=0xfc tid=10588 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B28D8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B28D8[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-8" id=67 idx=0x100 tid=10592 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x060A8A38[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x060A8A38[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-9" id=68 idx=0x104 tid=10596 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668B9A0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668B9A0[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-10" id=69 idx=0x108 tid=10600 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668C4B0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668C4B0[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_QuartzSchedulerThread" id=70 idx=0x10c tid=10604 prio=5 alive, in native, waiting
    -- Waiting for notification on: java/lang/Object@0x06798AF8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/core/QuartzSchedulerThread.run(QuartzSchedulerThread.java:245)
    ^-- Lock released while waiting: java/lang/Object@0x06798AF8[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Timer-2" id=71 idx=0x110 tid=10608 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x04931AE0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/util/TimerThread.mainLoop(Timer.java:509)
    ^-- Lock released while waiting: java/util/TaskQueue@0x04931AE0[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=83 idx=0x140 tid=10740 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x0B2D7AC8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0B2D7AC8[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=86 idx=0x14c tid=8948 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x0F235B30[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0F235B30[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554158815-ae234a20-leader-finder-thread" id=87 idx=0x150 tid=10124 prio=5 alive, in native, parked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x0664F210
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at kafka/consumer/ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:61)
    at kafka/utils/ShutdownableThread.run(ShutdownableThread.scala:60)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554155067-8b86727d-leader-finder-thread" id=88 idx=0x154 tid=9944 prio=5 alive, in native, parked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x030F3BA8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at kafka/consumer/ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:61)
    at kafka/utils/ShutdownableThread.run(ShutdownableThread.scala:60)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=89 idx=0x158 tid=10460 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr$2@0x03E62930[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Thread.join(Thread.java:1143)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr$2@0x03E62930[fat lock]
    at java/lang/Thread.join(Thread.java:1196)
    at java/lang/ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
    at java/lang/ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
    at java/lang/Shutdown.runHooks(Shutdown.java:79)
    at java/lang/Shutdown.sequence(Shutdown.java:123)
    at java/lang/Shutdown.exit(Shutdown.java:168)
    ^-- Holding lock: java/lang/Class@0x02C28598[thin lock]
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-1" id=14 idx=0x15c tid=4976 prio=5 alive, in native, native_blocked
    at jrockit/vm/RNI.generateVirtualCode(Ljava/lang/Object;II)I(Native Method)
    at weblogic/application/internal/flow/HeadWorkContextFlow.unprepare(HeadWorkContextFlow.java:0)
    at weblogic/application/internal/BaseDeployment$1.previous(BaseDeployment.java:1233)
    at weblogic/application/utils/StateMachineDriver.previousState(StateMachineDriver.java:329)
    at weblogic/application/utils/StateMachineDriver.previousState(StateMachineDriver.java:309)
    at weblogic/application/internal/BaseDeployment.unprepare(BaseDeployment.java:495)
    at weblogic/application/internal/SingleModuleDeployment.unprepare(SingleModuleDeployment.java:39)
    at weblogic/application/internal/DeploymentStateChecker.unprepare(DeploymentStateChecker.java:205)
    at weblogic/deploy/internal/targetserver/AppContainerInvoker.unprepare(AppContainerInvoker.java:117)
    at weblogic/deploy/internal/targetserver/BasicDeployment.unprepare(BasicDeployment.java:287)
    at weblogic/management/deploy/internal/ConfiguredDeployments.undeployPreStandbyInternalApps(ConfiguredDeployments.java:108)
    at weblogic/management/deploy/internal/DeploymentServerService.undeployPreStandbyInternalApps(DeploymentServerService.java:167)
    at weblogic/management/deploy/internal/DeploymentPreStandbyServerService.halt(DeploymentPreStandbyServerService.java:36)
    at weblogic/t3/srvr/ServerServicesManager.haltInternal(ServerServicesManager.java:504)
    at weblogic/t3/srvr/ServerServicesManager.halt(ServerServicesManager.java:336)
    at weblogic/t3/srvr/T3Srvr.shutdown(T3Srvr.java:986)
    at weblogic/t3/srvr/T3Srvr.forceShutdown(T3Srvr.java:892)
    at weblogic/t3/srvr/T3Srvr$2.run(T3Srvr.java:905)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-24" id=90 idx=0x170 tid=9840 prio=5 alive, in native, daemon
    at bea/jmapi/DiagnosticCommandImpl.execute(Ljava/lang/String;Ljava/io/FileDescriptor;)V(Native Method)
    at bea/jmapi/DiagnosticCommandImpl.execute(DiagnosticCommandImpl.java:50)
    at com/bea/jvm/DiagnosticCommand.execute(DiagnosticCommand.java:242)
    at com/bea/jvm/DiagnosticCommand$Command.execute(DiagnosticCommand.java:394)
    at bea/jmapi/ThreadSystemImpl.getThreadStackDump(ThreadSystemImpl.java:98)
    at weblogic/platform/JRockitVM.threadDump(JRockitVM.java:61)
    at weblogic/t3/srvr/T3Srvr.logThreadDump(T3Srvr.java:280)
    at weblogic/t3/srvr/ServerLifeCycleTimerThread.run(ServerLifeCycleTimerThread.java:77)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=91 idx=0x174 tid=6896 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=92 idx=0x178 tid=11124 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=93 idx=0x17c tid=10316 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=94 idx=0x180 tid=10924 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=95 idx=0x184 tid=11208 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=96 idx=0x188 tid=9872 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=97 idx=0x18c tid=8372 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=98 idx=0x190 tid=7252 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=99 idx=0x194 tid=10636 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=100 idx=0x198 tid=8736 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=101 idx=0x19c tid=5012 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=102 idx=0x1a0 tid=7468 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=103 idx=0x1a4 tid=10484 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=104 idx=0x1a8 tid=8852 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=105 idx=0x1ac tid=10904 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=106 idx=0x1b0 tid=7472 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=107 idx=0x1b4 tid=6240 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=108 idx=0x1b8 tid=9548 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=109 idx=0x1bc tid=11216 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=110 idx=0x1c0 tid=10400 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=111 idx=0x1c4 tid=3328 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=112 idx=0x1c8 tid=11228 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=113 idx=0x1cc tid=7524 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=114 idx=0x1d0 tid=10260 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=115 idx=0x1d4 tid=10648 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=116 idx=0x1d8 tid=8788 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

Blocked lock chains
===================
Chain 2:
"SIGINT handler" id=92 idx=0x178 tid=11124 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 3:
"SIGINT handler" id=93 idx=0x17c tid=10316 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 4:
"SIGINT handler" id=94 idx=0x180 tid=10924 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 5:
"SIGINT handler" id=95 idx=0x184 tid=11208 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 6:
"SIGINT handler" id=96 idx=0x188 tid=9872 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 7:
"SIGINT handler" id=97 idx=0x18c tid=8372 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 8:
"SIGINT handler" id=98 idx=0x190 tid=7252 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 9:
"SIGINT handler" id=99 idx=0x194 tid=10636 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 10:
"SIGINT handler" id=100 idx=0x198 tid=8736 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 11:
"SIGINT handler" id=101 idx=0x19c tid=5012 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 12:
"SIGINT handler" id=102 idx=0x1a0 tid=7468 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 13:
"SIGINT handler" id=103 idx=0x1a4 tid=10484 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 14:
"SIGINT handler" id=104 idx=0x1a8 tid=8852 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 15:
"SIGINT handler" id=105 idx=0x1ac tid=10904 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 16:
"SIGINT handler" id=106 idx=0x1b0 tid=7472 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 17:
"SIGINT handler" id=107 idx=0x1b4 tid=6240 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 18:
"SIGINT handler" id=108 idx=0x1b8 tid=9548 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 19:
"SIGINT handler" id=109 idx=0x1bc tid=11216 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 20:
"SIGINT handler" id=110 idx=0x1c0 tid=10400 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 21:
"SIGINT handler" id=111 idx=0x1c4 tid=3328 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 22:
"SIGINT handler" id=112 idx=0x1c8 tid=11228 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 23:
"SIGINT handler" id=113 idx=0x1cc tid=7524 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 24:
"SIGINT handler" id=114 idx=0x1d0 tid=10260 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 25:
"SIGINT handler" id=115 idx=0x1d4 tid=10648 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 26:
"SIGINT handler" id=116 idx=0x1d8 tid=8788 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Open lock chains
================
Chain 1:
"SIGINT handler" id=91 idx=0x174 tid=6896 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 (waiting on notification)

===== END OF THREAD DUMP ===============

>
本文参与 腾讯云自媒体分享计划，分享自作者个人站点/博客。
原始发表：2018-04-24 ，如有侵权请联系 cloudcommunity@tencent.com 删除
sql
数据库

本文分享自 作者个人站点/博客 前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ，欢迎热爱写作的你一起参与！

sql
数据库
评论
登录后参与评论
0 条评论
热度
最新
登录 后参与评论
推荐阅读
LV.
文章
0
获赞
0
目录
  • 问题背景
  • 问题日志
  • 问题分析：
  • 问题解决：
  • 其他解决方案（从sql优化入手）
  • 线程池、数据库连接池调优
    • 线程池
      • 数据库连接池：
      相关产品与服务
      数据库
      云数据库为企业提供了完善的关系型数据库、非关系型数据库、分析型数据库和数据库生态工具。您可以通过产品选择和组合搭建，轻松实现高可靠、高可用性、高性能等数据库需求。云数据库服务也可大幅减少您的运维工作量，更专注于业务发展，让企业一站式享受数据上云及分布式架构的技术红利！
      产品介绍
      精选特惠 用云无忧
      领券

      腾讯云开发者

      扫码关注腾讯云开发者

      扫码关注腾讯云开发者

      领取腾讯云代金券

      热门产品

      热门推荐

      更多推荐

      Copyright © 2013 - 2024 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

      深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

      腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

      问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档

      Copyright © 2013 - 2024 Tencent Cloud.

      All Rights Reserved. 腾讯云 版权所有

      登录 后参与评论