先是凌晨的时候中间件连数据库出了问题：
org.apache.commons.dbcp.sqlNestedException: Cannot create PoolableConnectionFactory (IO 错误: The Network Adapter could not establish the connection)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1244) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:882) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at com.ic.core.util.IcDataSourceManager.getConnection(IcDataSourceManager.java:81) ~[eic-core-1.0.6.jar:na]
然后出现连接不够用的情况：
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[STUCK] ExecuteThread: '61' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",out of which "1" are available and "0" are unavailable.>
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",out of which "1" are available and "0" are unavailable.>
weblogic状态开始告警。
上午10点钟的时候用jstack看到有事务在回滚，按理说事务回滚结束之后连接池会回收连接的，但等了两个小时看还是这个状态。
"[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1ad1c53000 nid=0x516e waiting for monitor entry [0x00007f1a9d6d5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
- waiting to lock <0x000000070d83a480> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
at weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
at weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
原文链接：https://www.f2er.com/oracle/207252.htmlorg.apache.commons.dbcp.sqlNestedException: Cannot create PoolableConnectionFactory (IO 错误: The Network Adapter could not establish the connection)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1244) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:882) ~[com.springsource.org.apache.commons.dbcp-1.2.2.osgi.jar:na]
at com.ic.core.util.IcDataSourceManager.getConnection(IcDataSourceManager.java:81) ~[eic-core-1.0.6.jar:na]
然后出现连接不够用的情况：
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[STUCK] ExecuteThread: '61' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",out of which "1" are available and "0" are unavailable.>
####<2017-10-24 上午12时25分28秒 CST> <Info> <Common> <zc-sc-app04> <sys_app04_8001> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1508775928273> <BEA-000628> <Created "1" resources for pool "CTDataSource",out of which "1" are available and "0" are unavailable.>
weblogic状态开始告警。
上午10点钟的时候用jstack看到有事务在回滚，按理说事务回滚结束之后连接池会回收连接的，但等了两个小时看还是这个状态。
"[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f1ad1c53000 nid=0x516e waiting for monitor entry [0x00007f1a9d6d5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
- waiting to lock <0x000000070d83a480> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
at weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
at weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
检查了一下gv$session,并未发现有活动的连接，且当前的Weblogic连接都没有执行的sql语句，我认为这是僵尸的线程，只能重启。
后来确定这是一个bug.
Many Stuck Threads on JDBC Connection Rollback (文档 ID 1584478.1)
SYMPTOMS
- There are lots of messages like the following in the server log,and the server response is slow:
<WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "611" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@1f12dec2",which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>
<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "631" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@20cb3eea",which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3896)
weblogic.jdbc.wrapper.Connection.forcedCleanup(Connection.java:153)
weblogic.common.resourcepool.ResourcePoolImpl.timeoutInactiveResources(ResourcePoolImpl.java:1953)
weblogic.common.resourcepool.ResourcePoolImpl.access$1500(ResourcePoolImpl.java:41)
weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2675)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>
CAUSE
The issue depends on an uncontrollable timing sequence to occur,and also may involve a requirement that the datasource has been disabled or flushed (whether internally or manually) prevIoUsly. The basic symptom is of a hanging thread,trying to do a rollback on a connection whose socket is in an externally-induced condition (maybe due to a firewall) that will relegate any standard JDBC call to hang indefinitely in a socket read. This issue is addressed byBug 10169939.
