Fixes are available
8.5.5.6: WebSphere Application Server V8.5.5 Fix Pack 6
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
8.5.5.7: WebSphere Application Server V8.5.5 Fix Pack 7
8.5.5.8: WebSphere Application Server V8.5.5 Fix Pack 8
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
8.5.5.9: WebSphere Application Server V8.5.5 Fix Pack 9
8.5.5.10: WebSphere Application Server V8.5.5 Fix Pack 10
8.5.5.11: WebSphere Application Server V8.5.5 Fix Pack 11
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
APAR status
Closed as program error.
Error description
Deadlock occurs during heavy logging when the log rotation is happening very frequently. The java stack in the deadlock state is as follow: Deadlock detected !!! --------------------- Thread "WebContainer : 35" (0x00000000047D3600) is waiting for: sys_mon_t:0x00007F651D582810 infl_mon_t: 0x00007F651D582888: com/ibm/ejs/ras/ManagerAdmin@0x000000000CE6AEB8 which is owned by: Thread "Default : 1" (0x0000000003E2E200) which is waiting for: sys_mon_t:0x00007F6523831A80 infl_mon_t: 0x00007F6523831AF8: com/ibm/ejs/ras/WrappingFileOutputStream@0x0000000029F408B0 which is owned by: Thread "WebContainer : 35" (0x00000000047D3600) "WebContainer : 35" J9VMThread:0x00000000047D3600, j9thread_t:0x00007F653263A6E0, java/lang/Thread:0x0000000022ADCB48, state:B, prio=5 (native thread ID:0x3644, native priority:0x5, native policy:UNKNOWN) (native stack address range from:0x00007F6519E92000, to:0x00007F6519ED3000, size:0x41000) Blocked on: com/ibm/ejs/ras/ManagerAdmin@0x000000000CE6AEB8 Owned by: "Default : 1" (J9VMThread:0x0000000003E2E200, java/lang/Thread:0x00000000172435E0) Java callstack: at com/ibm/ejs/ras/RasHelper .getHeaderAsProperties(RasHelper.java:1 988) at com/ibm/ejs/ras/TraceLogger.printHeader(TraceLogger.java:431 (Compiled Code)) (entered lock: com/ibm/ejs/ras/TraceLogger@0x0000000029F40880, entry count: 3) at com/ibm/ejs/ras/TraceLogger.logRolled(TraceLogger.java:297) (entered lock: com/ibm/ejs/ras/TraceLogger@0x0000000029F40880, entry count: 2) at com/ibm/ejs/ras/WrappingFileOutputStream .switchFiles(WrappingFileOutputStream.java:689(Compiled Code)) (entered lock: com/ibm/ejs/ras/WrappingFileOutputStream@ 0x0000000029F408B0,entry count: 3) at com/ibm/ejs/ras/WrappingFileOutputStream.switchFiles2 (WrappingFileOutpu t Stream.java:611(Compiled Code)) (entered lock: com/ibm/ejs/ras/WrappingFileOutputStream@ 0x0000000029F408B0, entry count: 2) <thread stack continues> "Default : 1" J9VMThread:0x0000000003E2E200, j9thread_t:0x00007F653441D690, java/lang/Thread:0x00000000172435E0, state:B, prio=5 (native thread ID:0x13EF, native priority:0x5, native policy:UNKNOWN) (native stack address range from:0x00007F653381D000, to:0x00007F653385E000, size:0x41000) Blocked on: com/ibm/ejs/ras/WrappingFileOutputStream@0x0000000029F408B0 Owned by: "WebContainer : 35" (J9VMThread:0x00000000047D3600, java/lang/Thread:0x0000000022ADCB48) Java callstack: at com/ibm/ejs/ras/ManagerAdmin.closeServerTraceFile (ManagerAdmin.java:1407 ) (entered lock: com/ibm/ejs/ras/ManagerAdmin@ 0x000000000CE6AEB8, entry count: 3) at com/ibm/ejs/ras/ManagerAdmin.removeServerTraceFileListener (ManagerAdmin. java:1361) (entered lock: com/ibm/ejs/ras/ManagerAdmin@ 0x000000000CE6AEB8, entry count: 2) at com/ibm/ejs/ras/ManagerAdmin.processTraceStateChanged (ManagerAdmin.java: 1464) (entered lock: com/ibm/ejs/ras/ManagerAdmin@ 0x000000000CE6AEB8, entry count: 1) at com/ibm/ejs/ras/ManagerAdmin.setTraceState(ManagerAdmin.java:693 )
Local fix
Avoid setting trace specification once the log is getting rotated.
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server V8.0 and V8.5 * **************************************************************** * PROBLEM DESCRIPTION: Deadlock may happen when heavy * * logging is enabled , and file * * rollover is happening very frequently. * **************************************************************** * RECOMMENDATION: * **************************************************************** Deadlock may happen when heavy logging is enabled , and file rollover is happening very frequently. The deadlock was caused by a call for checking if the process running is under thinclient or server, for creating a header for a new log file. Checking this above condition happens in a synchronized block which was causing the deadlock. The java stack in the deadlock state is as follow: Deadlock detected !!! --------------------- Thread "WebContainer : 35" (0x00000000047D3600) is waiting for: sys_mon_t:0x00007F651D582810 infl_mon_t: 0x00007F651D582888: com/ibm/ejs/ras/ManagerAdmin@0x000000000CE6AEB8 which is owned by: Thread "Default : 1" (0x0000000003E2E200) which is waiting for: sys_mon_t:0x00007F6523831A80 infl_mon_t: 0x00007F6523831AF8: com/ibm/ejs/ras/WrappingFileOutputStream@0x0000000029F408B0 which is owned by: Thread "WebContainer : 35" (0x00000000047D3600)
Problem conclusion
Synchronized block has been removed but in a way that keeps the method thread safe. Also, a mechanism has been derived in which this check of thin client and server are not required until the process restarts. The deadlock will no longer appear in this case. The fix for this APAR is currently targeted for inclusion in fix packs 8.0.0.11 and 8.5.5.6. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PI28814
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
800
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2014-10-31
Closed date
2015-02-18
Last modified date
2015-02-18
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R800 PSY
UP
R850 PSY
UP
Document Information
Modified date:
28 April 2022