PM37960: DEADLOCK OCCURS DURING DEFERRED EJB INITIALIZATION UNDER RARE CIRCUMSTANCES.

Fixes are available

APAR status

Closed as program error.

Error description

An app was updated/redeployed without quiescing the server. A
deadlock was encountered which ended up causing the
servant to timeout with ABENDSEC3 SEC3 RSN04130004.
This can occur under the following circumstances
  A) the server has been configured to initialize all EJBs at
     application startup time
     or
     The server has a startup bean deployed in it
     or
     the server has an MDB deployed in it
  B) an app deployment is in progress
  C) inbound remote EJB requests are coming in to the server
  D) an EJB being invoked just so happens to overlap the EJB
     wrapper for an EJB just initializing

Javacore from the failure contains:
1LKDEADLOCK    Deadlock detected !!!

NULL           ---------------------
NULL
2LKDEADLOCKTHR  Thread "WebSphere:ORB.thread.pool t=007bd3e8"
(0x5B3DB100)
3LKDEADLOCKWTR    is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x572541C0 infl_mon_t: 0x57254200:
4LKDEADLOCKOBJ
com/ibm/ejs/csi/J2EENameImpl@53711A30/53711A3C:
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "WebSphere t=007bb718" (0x5B414E00)
3LKDEADLOCKWTR    which is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x57703C4C infl_mon_t: 0x57703C8C:
4LKDEADLOCKOBJ
com/ibm/ejs/util/cache/WrapperBucket@36A5B778/36A5B784:
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "WebSphere:ORB.thread.pool t=007bd3e8"
(0x5B3DB100)

Call stack for "WebSphere:ORB.thread.pool t=007bd3e8"

======================================================
"WebSphere:ORB.thread.pool t=007bd3e8" (TID:0x5B3DB100,
sys_thread_t:0x5B3DBAF0, state:B, native ID:0x3B5E9000) prio=5
at com/ibm/ejs/container/HomeRecord.getHomeAndInitialize(
    HomeRecord.java:467)
at com/ibm/ejs/container/EJSContainer.getHomeWrapperCommon(
    EJSContainer.java:1273)
at com/ibm/ejs/container/EJSContainer.getHomeInstance(
    EJSContainer.java:1182)
at com/ibm/ejs/container/EJSContainer.startBean(
    EJSContainer.java:1168)
at com/ibm/ws/runtime/component/EJBContainerImpl.startBean(
    EJBContainerImpl.java:3484)
at com/ibm/ws/runtime/component/EJBContainerImpl.
    startAndBindHome(EJBContainerImpl.java:4761)
at com/ibm/ws/runtime/component/EJBContainerImpl.
    initializeDeferredEJB(EJBContainerImpl.java:4546)
at com/ibm/ejs/container/HomeOfHomes.getHome(
    HomeOfHomes.java:345)
at com/ibm/ejs/container/HomeOfHomes.internalCreateWrapper(
    HomeOfHomes.java:481)
at com/ibm/ejs/container/EJSContainer.createWrapper(
    EJSContainer.java:4396(Compiled Code))
at com/ibm/ejs/container/WrapperManager.faultOnKey(
    WrapperManager.java:528(Compiled Code))
at com/ibm/ejs/util/cache/Cache.findAndFault(Cache.java:497(
    Compiled Code))
at com/ibm/ejs/container/WrapperManager.keyToObject(
    WrapperManager.java:481(Compiled Code))
at com/ibm/ejs/oa/EJSOAImpl.keyToObject(EJSOAImpl.java:553(
    Compiled Code))
at com/ibm/ejs/oa/EJSRootOAImpl.keyToObject(
    EJSRootOAImpl.java:271(Compiled Code))
at com/ibm/ws390/orb/CommonBridge.invoke(Bytecode
    PC:777(Compiled Code))
at com/ibm/ws390/orb/ORBEJSBridge.invoke(Bytecode PC:6(
    Compiled Code))
at sun/reflect/GeneratedMethodAccessor234.invoke(Bytecode PC:24(
    Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(
    DelegatingMethodAccessorImpl.java:43(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:618(
    Compiled Code))
at com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.
    ORBEJSBridgeInvokeParmSetter(
    ORBEJSBridgeInvoke.java:159(Compiled Code))
at com/ibm/ws390/orb/CommonBridge.nativeRunApplicationThread(
    Native Method)
at com/ibm/ws390/orb/CommonBridge.runApplicationThread(
    Bytecode PC:7)
at com/ibm/ws/util/ThreadPool$ZOSWorker.run(
    ThreadPool.java:1697)

Call stack for "WebSphere t=007bb718"
=====================================
"WebSphere t=007bb718" (TID:0x5B414E00, sys_thread_t:0x5B412730,
state:B, native ID:0x3B629D00) prio=5
at com/ibm/ejs/util/cache/Cache.findAndFault(Cache.java:463(
    Compiled Code))
at com/ibm/ejs/container/WrapperManager.getWrapper(
    WrapperManager.java:277(Compiled Code))
at com/ibm/ejs/container/EJSHome.getWrapper(EJSHome.java:776)
at com/ibm/ejs/container/EJSContainer.loadBeanMetaData(
    EJSContainer.java:1697(Compiled Code))
at com/ibm/ejs/container/HomeOfHomes.loadBeanMetaData(
    HomeOfHomes.java:641)
at com/ibm/ejs/container/HomeRecord.getHomeAndInitialize(
    HomeRecord.java:479)
at com/ibm/ejs/container/EJSContainer.getHomeWrapperCommon(
    EJSContainer.java:1273)
at com/ibm/ejs/container/EJSContainer.getHomeInstance(
    EJSContainer.java:1182)
at com/ibm/ejs/container/EJSContainer.startBean(
    EJSContainer.java:1168)
at com/ibm/ws/runtime/component/EJBContainerImpl.startBean(
    EJBContainerImpl.java:3484)
at com/ibm/ws/runtime/component/EJBContainerImpl.install(
    EJBContainerImpl.java:2888(Compiled Code))
at com/ibm/ws/runtime/component/EJBContainerImpl.start(
    EJBContainerImpl.java:3730)
at com/ibm/ws/runtime/component/ApplicationMgrImpl.start(
    ApplicationMgrImpl.java:1250(Compiled Code))
at com/ibm/ws/runtime/component/DeployedApplicationImpl.
    fireDeployedObjectStart(DeployedApplicationImpl.java:1140)
at com/ibm/ws/runtime/component/DeployedModuleImpl.start(
    DeployedModuleImpl.java:572(Compiled Code))
at com/ibm/ws/runtime/component/DeployedApplicationImpl.start(
    DeployedApplicationImpl.java:819(Compiled Code))
at com/ibm/ws/runtime/component/ApplicationMgrImpl.
    startApplication(
    ApplicationMgrImpl.java:921(Compiled Code))
at com/ibm/ws/runtime/component/ApplicationMgrImpl$1.run(
    ApplicationMgrImpl.java:1437)
at com/ibm/ws/security/auth/ContextManagerImpl.runAs(
    ContextManagerImpl.java:4158(Compiled Code))
at com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem(
    ContextManagerImpl.java:4255(Compiled Code))
at com/ibm/ws/security/core/SecurityContext.runAsSystem(
    SecurityContext.java:245)
at com/ibm/ws/runtime/component/ApplicationMgrImpl.
    startApplication(
    ApplicationMgrImpl.java:1442(Compiled Code))
at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(
    NativeMethodAccessorImpl.java:79(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(
    DelegatingMethodAccessorImpl.java:43(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:618(
    Compiled Code))
at sun/reflect/misc/Trampoline.invoke(MethodUtil.java:68)
at sun/reflect/GeneratedMethodAccessor238.invoke(Bytecode PC:40)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(
    DelegatingMethodAccessorImpl.java:43(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:618(
    Compiled Code))
at sun/reflect/misc/MethodUtil.invoke(MethodUtil.java:271)
at javax/management/modelmbean/RequiredModelMBean.invokeMethod(
    RequiredModelMBean.java:1092)
at javax/management/modelmbean/RequiredModelMBean.invoke(
    RequiredModelMBean.java:973)
at com/sun/jmx/mbeanserver/DynamicMetaDataImpl.invoke(
    DynamicMetaDataImpl.java:231)
at com/sun/jmx/mbeanserver/MetaDataImpl.invoke(
    MetaDataImpl.java:238)
at com/sun/jmx/interceptor/DefaultMBeanServerInterceptor.invoke(
    DefaultMBeanServerInterceptor.java:833)
at com/sun/jmx/mbeanserver/JmxMBeanServer.invoke(
    JmxMBeanServer.java:802)
at com/ibm/ws390/management/connector/corba/CorbaConnectorImpl.
    invoke(CorbaConnectorImpl.java:705)
at com/ibm/ws390/management/connector/corba/
    _CorbaConnectorImplBase.
    _invoke(_CorbaConnectorImplBase.java:406)
at com/ibm/ws390/orb/CommonBridge.invoke(
    Bytecode PC:1337(Compiled Code))
at com/ibm/ws390/orb/ORBEJSBridge.invoke(Bytecode PC:6(
    Compiled Code))
at sun/reflect/GeneratedMethodAccessor234.invoke(
    Bytecode PC:24(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(
    DelegatingMethodAccessorImpl.java:43(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:618(
    Compiled Code))
at com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.
    ORBEJSBridgeInvokeParmSetter(
    ORBEJSBridgeInvoke.java:159(Compiled Code))

Local fix

quiesce workload before attempting to deploy a new version of
the application.

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server with Enterprise Java Beans (EJBs)    *
****************************************************************
* PROBLEM DESCRIPTION: Application start with EJBs can         *
*                      deadlock if the first request to an     *
*                      EJB is made simultaneously.             *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Remote EJB request processing acquires a hash table bucket
lock; if the remote request is the first call to that EJB,
then an application-wide lock is acquired.  During application
start of non-deferred EJBs, such as MDBs or startup beans, the
application-wide lock is acquired and then the hash table
bucket lock is acquired.  If these two events happen
concurrently, a deadlock occurs.

Problem conclusion

The application start code path was changed to acquire the
hash table bucket lock before the application-wide lock, which
prevents the deadlock.

APAR PM37960 is currently targeted for inclusion in Service
Level (Fix Pack) 6.1.0.41 of WebSphere Application Server
V6.1.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM37960
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-04-28
Closed date
2011-05-10
Last modified date
2011-12-02

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM38277

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R610 PSY UK73054
UP11/11/03 P F111

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
27 October 2021

Tips

PM37960: DEADLOCK OCCURS DURING DEFERRED EJB INITIALIZATION UNDER RARE CIRCUMSTANCES.

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R610 PSY UK73054

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?