IBM Support

PI28814: DEADLOCK CAN OCCUR WHEN ONE THREAD IS TRYING TO ROTATE LOG FILE AND ANOTHER IS SETTING A NEW TRACE SPECIFICATION

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • Deadlock occurs during heavy logging when the log rotation is
    happening very frequently. The java stack in the deadlock
    state is as follow:
    
    Deadlock detected !!!
    ---------------------
    Thread "WebContainer : 35" (0x00000000047D3600)
    is waiting for:
    sys_mon_t:0x00007F651D582810 infl_mon_t: 0x00007F651D582888:
    com/ibm/ejs/ras/ManagerAdmin@0x000000000CE6AEB8
    which is owned by:
    Thread "Default : 1" (0x0000000003E2E200)
    which is waiting for:
    sys_mon_t:0x00007F6523831A80 infl_mon_t: 0x00007F6523831AF8:
    com/ibm/ejs/ras/WrappingFileOutputStream@0x0000000029F408B0
    which is owned by:
    Thread "WebContainer : 35" (0x00000000047D3600)
    
    
    "WebContainer : 35" J9VMThread:0x00000000047D3600,
    j9thread_t:0x00007F653263A6E0,
    java/lang/Thread:0x0000000022ADCB48,
    state:B, prio=5
    (native thread ID:0x3644, native priority:0x5, native
    policy:UNKNOWN)
    (native stack address range from:0x00007F6519E92000,
    to:0x00007F6519ED3000, size:0x41000)
    Blocked on: com/ibm/ejs/ras/ManagerAdmin@0x000000000CE6AEB8
    Owned by:
    "Default : 1" (J9VMThread:0x0000000003E2E200,
    java/lang/Thread:0x00000000172435E0)
    Java callstack:
    at com/ibm/ejs/ras/RasHelper
    .getHeaderAsProperties(RasHelper.java:1 988)
    at com/ibm/ejs/ras/TraceLogger.printHeader(TraceLogger.java:431
    (Compiled Code))
    (entered lock:
    com/ibm/ejs/ras/TraceLogger@0x0000000029F40880, entry count: 3)
    at com/ibm/ejs/ras/TraceLogger.logRolled(TraceLogger.java:297)
    (entered lock:
    com/ibm/ejs/ras/TraceLogger@0x0000000029F40880, entry count: 2)
    at com/ibm/ejs/ras/WrappingFileOutputStream
    .switchFiles(WrappingFileOutputStream.java:689(Compiled Code))
    (entered lock: com/ibm/ejs/ras/WrappingFileOutputStream@
    0x0000000029F408B0,entry count: 3)
    at com/ibm/ejs/ras/WrappingFileOutputStream.switchFiles2
    (WrappingFileOutpu t Stream.java:611(Compiled Code))
    (entered lock: com/ibm/ejs/ras/WrappingFileOutputStream@
    0x0000000029F408B0, entry count: 2)
    <thread stack continues>
    
    
    "Default : 1" J9VMThread:0x0000000003E2E200,
    j9thread_t:0x00007F653441D690,
    java/lang/Thread:0x00000000172435E0,
    state:B, prio=5
    (native thread ID:0x13EF, native priority:0x5, native
    policy:UNKNOWN)
    (native stack address range from:0x00007F653381D000,
    to:0x00007F653385E000, size:0x41000)
    Blocked on:
    com/ibm/ejs/ras/WrappingFileOutputStream@0x0000000029F408B0
    Owned by: "WebContainer : 35" (J9VMThread:0x00000000047D3600,
    java/lang/Thread:0x0000000022ADCB48)
    Java callstack:
    at com/ibm/ejs/ras/ManagerAdmin.closeServerTraceFile
    (ManagerAdmin.java:1407 )
    (entered lock: com/ibm/ejs/ras/ManagerAdmin@
    0x000000000CE6AEB8, entry count: 3)
    at com/ibm/ejs/ras/ManagerAdmin.removeServerTraceFileListener
    (ManagerAdmin. java:1361)
    (entered lock: com/ibm/ejs/ras/ManagerAdmin@
    0x000000000CE6AEB8, entry count: 2)
    at com/ibm/ejs/ras/ManagerAdmin.processTraceStateChanged
    (ManagerAdmin.java: 1464)
    (entered lock: com/ibm/ejs/ras/ManagerAdmin@
    0x000000000CE6AEB8, entry count: 1)
    at
    com/ibm/ejs/ras/ManagerAdmin.setTraceState(ManagerAdmin.java:693
    )
    

Local fix

  • Avoid setting trace specification once the log is getting
    rotated.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V8.0 and V8.5                        *
    ****************************************************************
    * PROBLEM DESCRIPTION: Deadlock may happen when heavy          *
    *                      logging is enabled , and file           *
    *                      rollover is happening very frequently.  *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Deadlock may happen when heavy logging is enabled , and file
    rollover is happening very frequently. The deadlock was caused
    by a call for checking if the process running is under
    thinclient or server, for creating a header for a new log
    file. Checking this above condition happens in a synchronized
    block which was causing the deadlock.
    The java stack in the deadlock state is as follow:
    Deadlock detected !!!
    ---------------------
    Thread "WebContainer : 35" (0x00000000047D3600)
    is waiting for:
    sys_mon_t:0x00007F651D582810 infl_mon_t: 0x00007F651D582888:
    com/ibm/ejs/ras/ManagerAdmin@0x000000000CE6AEB8
    which is owned by:
    Thread "Default : 1" (0x0000000003E2E200)
    which is waiting for:
    sys_mon_t:0x00007F6523831A80 infl_mon_t: 0x00007F6523831AF8:
    com/ibm/ejs/ras/WrappingFileOutputStream@0x0000000029F408B0
    which is owned by:
    Thread "WebContainer : 35" (0x00000000047D3600)
    

Problem conclusion

  • Synchronized block has been removed but in a way that keeps
    the method thread safe. Also, a mechanism has been derived in
    which this check of thin client and server are not required
    until the process restarts. The deadlock will no longer appear
    in this case.
    The fix for this APAR is currently targeted for inclusion in
    fix packs 8.0.0.11 and 8.5.5.6.  Please refer to the
    Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PI28814

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    800

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2014-10-31

  • Closed date

    2015-02-18

  • Last modified date

    2015-02-18

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R800 PSY

       UP

  • R850 PSY

       UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 April 2022