我在构建服务器上有一个间歇性问题,其中构建中的
Java进程无法终止,并且似乎继续运行(使用100％的cpu)永远(我已经看到它在周末运行了2天需要约10分钟). kill -9 pid似乎是停止进程的唯一方法.
我已经尝试在进程上调用kill -QUIT pid,但它似乎没有产生任何到STDOUT的堆栈跟踪(也许它没有响应信号？).没有-F强制选项的jstack似乎无法连接到正在运行的JVM,但是使用force选项会产生下面包含的输出.
不幸的是,即使有堆栈跟踪,我看不到任何明显的进一步调查路径.
据我所知,它显示了两个“BLOCKED”线程,它们运行了Object.wait(它们的堆栈似乎只包含核心Java代码,我们没有),第三个是“IN_VM”,没有堆栈输出.
我应该采取哪些步骤来收集有关问题原因的更多信息(或更好的是,我如何解决它)？
- $/opt/jdk1.6.0_29/bin/jstack -l -F 5546
- Attaching to process ID 5546,please wait...
- Debugger attached successfully.
- Server compiler detected.
- JVM version is 20.4-b02
- Deadlock Detection:
- No deadlocks found.
- Finding object size using Printezis bits and skipping over...
- Thread 5555: (state = BLOCKED)
- Locked ownable synchronizers:
- - None
- Thread 5554: (state = BLOCKED)
- - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- - java.lang.ref.ReferenceQueue.remove(long) @bci=44,line=118 (Interpreted frame)
- - java.lang.ref.ReferenceQueue.remove() @bci=2,line=134 (Interpreted frame)
- - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3,line=159 (Interpreted frame)
- Locked ownable synchronizers:
- - None
- Thread 5553: (state = BLOCKED)
- - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- - java.lang.Object.wait() @bci=2,line=485 (Interpreted frame)
- - java.lang.ref.Reference$ReferenceHandler.run() @bci=46,line=116 (Interpreted frame)
- Locked ownable synchronizers:
- - None
- Thread 5548: (state = IN_VM)
- Locked ownable synchronizers:
- - None
(Java版本1.6.0更新29,在Scientific Linux版本6.0上运行)
更新：
运行strace -f -p 894生成一个看似无尽的流…
- [pid 900] sched_yield() = 0
- [pid 900] sched_yield() = 0
- ...
然后当Ctrl-Cd
- Process 894 detached
- ...
- Process 900 detached
- ...
- Process 909 detached
jmap -histo 894不连接但jmap -F -histo 894返回…
- Attaching to process ID 894,please wait...
- Debugger attached successfully.
- Server compiler detected.
- JVM version is 20.4-b02
- Iterating over heap. This may take a while...
- Finding object size using Printezis bits and skipping over...
- Finding object size using Printezis bits and skipping over...
- Object Histogram:
- num #instances #bytes Class description
- --------------------------------------------------------------------------
- 1: 11356 1551744 * MethodKlass
- 2: 11356 1435944 * ConstMethodKlass
- 3: 914 973488 * ConstantPoolKlass
- 4: 6717 849032 char[]
- 5: 16987 820072 * SymbolKlass
- 6: 2305 686048 byte[]
- 7: 914 672792 * InstanceKlassKlass
- 8: 857 650312 * ConstantPoolCacheKlass
- 9: 5243 167776 java.lang.String
- 10: 1046 108784 java.lang.Class
- 11: 1400 87576 short[]
- 12: 1556 84040 * System ObjArray
- 13: 1037 64584 int[]
- 14: 103 60152 * ObjArrayKlassKlass
- 15: 622 54736 java.lang.reflect.Method
- 16: 1102 49760 java.lang.Object[]
- 17: 937 37480 java.util.TreeMap$Entry
- 18: 332 27960 java.util.HashMap$Entry[]
- 19: 579 27792 java.nio.HeapByteBuffer
- 20: 578 27744 java.nio.HeapCharBuffer
- 21: 1021 24504 java.lang.StringBuilder
- 22: 1158 24176 java.lang.Class[]
- 23: 721 23072 java.util.HashMap$Entry
- 24: 434 20832 java.util.TreeMap
- 25: 689 18936 java.lang.String[]
- 26: 238 17440 java.lang.reflect.Method[]
- 27: 29 16800 * MethodDataKlass
- 28: 204 14688 java.lang.reflect.Field
- 29: 330 13200 java.util.LinkedHashMap$Entry
- 30: 264 12672 java.util.HashMap
- ...
- 585: 1 16 java.util.LinkedHashSet
- 586: 1 16 sun.rmi.runtime.NewThreadAction$2
- 587: 1 16 java.util.Hashtable$EmptyIterator
- 588: 1 16 java.util.Collections$EmptySet
- Total : 79700 8894800
- Heap traversal took 1.288 seconds.
