对于我们的应用程序的一次安装,我们一直看到生产中的问题被用户报告为“系统变慢”或“请求永不返回”.最后,服务器必须重新启动.我们有几个这样的事件,每晚重启服务器似乎是一个解决方法.
我们的应用程序大量使用动态类加载(存储在数据库中的.jar文件作为blob)和反射.
环境细节：
> Java 1.7.021
>操作系统：Linux(2.6.32-504.16.2.el6.x86_64)
> JBoss EAP 6.2
> Appdynamics正在使用
>内存和gc设置：-XX：PermSize = 256m -XX：MaxPermSize = 2560m -Xms2048m -Xmx10240m -server -XX：UseParallelGC -XX：UseParallelOldGC
我们切换到了
> Java 1.7.080
> -XX：UseG1GC
但看起来我们仍然面临着这个问题.
我们在logfiles,heapdumps,threadumps和gc日志中看到的内容如下所示
>我们没有看到任何OutOfMemory错误
> Heap和PermGend的内存消耗似乎没问题(堆和perm gen未完全使用)
>使用Java 1.7.021和parallelGC,我们看到了几个长时间运行(5分钟)的Full GC – 在切换到G1GC后这似乎不再发生(这是我们的预期)
>我们在日志中看不到任何与CodeCache相关的警告
我们看到的是
>对于某些事件,heapdump中DelegatingClassLoaders的数量约为5000,而其他事件则约为10万.
>在用户初始报告系统变慢之后,状态为RUNNABLE且正在处理“getDeclaredConstructors0”的线程数正在增加(从开头只有两个到大约50个.线程似乎“挂起”多个小时的相同位置(它们处于RUNNABLE状态).
有关该线程组的摘录,请参见下文：
"http-xxx:8443-141" daemon prio=10 tid=0x00007efe9412f000 nid=0x64fb runnable [0x00007efe1ba8a000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2413)
at java.lang.Class.getConstructor0(Class.java:2723)
at java.lang.Class.newInstance0(Class.java:345)
at java.lang.Class.newInstance(Class.java:327)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
[application code]
"http-xxx:8443-138" daemon prio=10 tid=0x00007efe9412a000 nid=0x64ed runnable [0x00007efe1c397000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2413)
at java.lang.Class.getConstructor0(Class.java:2723)
at java.lang.Class.newInstance0(Class.java:345)
at java.lang.Class.newInstance(Class.java:327)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.mvel2.PropertyAccessor.set(PropertyAccessor.java:350)
at org.mvel2.PropertyAccessor.set(PropertyAccessor.java:134)
at org.mvel2.MVEL.setProperty(MVEL.java:1088)
[application code]
更新到Java 1.7.80 / G1Gc后的情况似乎相似.不幸的是没有可用的线程,只是日志中的wicket警告)
2016-03-10 15:00:39,009 WARN [http-xxx:7443-23] org.apache.wicket.page.PageAccessSynchronizer: Thread 'http-xxx:7443-23' Failed to acquire lock to page with id '28',attempted for 3 minutes out of allowed 3 minutes. The thread that holds the lock has name 'http-xxx:7443-18'. 2016-03-10 15:00:39,017 WARN [http-xxx:7443-23] org.apache.wicket.page.PageAccessSynchronizer: "http-xxx:7443-18" daemon prio=5 tid=8950 state=RUNNABLE org.apache.wicket.util.lang.Threads$ThreadDump at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) at java.lang.Class.getConstructor0(Class.java:2895) at java.lang.Class.newInstance(Class.java:354) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) at java.security.AccessController.doPrivileged(Native Method) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113) at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331) at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1376) at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:493) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) [application code] 2016-03-10 15:04:14,640 WARN [http-xxx:7443-16] org.apache.wicket.page.PageAccessSynchronizer: Thread 'http-xxx:7443-16' Failed to acquire lock to page with id '11',attempted for 3 minutes out of allowed 3 minutes. The thread that holds the lock has name 'http-xxx:7443-38'. 2016-03-10 15:04:14,642 WARN [http-xxx:7443-16] org.apache.wicket.page.PageAccessSynchronizer: "http-xxx:7443-38" daemon prio=5 tid=9088 state=RUNNABLE org.apache.wicket.util.lang.Threads$ThreadDump at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595) at java.lang.Class.getConstructor0(Class.java:2895) at java.lang.Class.newInstance(Class.java:354) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) at java.security.AccessController.doPrivileged(Native Method) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) [application code]
我们目前无法重现这一点(仍在努力).但也许社区中的某些人已经看到了类似的东西,并且知道什么可以帮助我们重现或解决问题.
我们目前的一个猜测是,这与本机内存消耗有关(因为像http://www.ibm.com/developerworks/java/library/j-nativememory-linux/这样的信息)但我们在日志中没有看到这方面的任何提示(没有OutOfMemory错误,没有来自linux管理员的报告说系统内存不足)
