PK76104: DEADLOCK DETECTED IN NOTIFICATIONSERVICE.HANDLENOTIFICATION WITH "JAVA/LANG/OUTOFMEMORYERROR":"FAILED TO FORK OS THREAD"

Fixes are available

APAR status

Closed as program error.

Error description

****************************************************************
* USERS AFFECTED:  WebSphere Application Server V6.1 for z/OS  *
*                  6.1.0.13                                    *
****************************************************************
* PROBLEM DESCRIPTION:                                         *

A Javacore created at the time of the abend:

 "java/lang/OutOfMemoryError":"Failed to fork OS thread"

shows threads in a deadlock condition:

1LKDEADLOCK    Deadlock detected !!!
2LKDEADLOCKTHR  Thread "Non-deferrable Alarm : 0" (0x594D6200)
3LKDEADLOCKWTR    is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x66693BCC infl_mon_t: 0x66693C0C:
4LKDEADLOCKOBJ      java/util/HashMap@45FC3870/45FC387C:
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "SoapConnectorThreadPool : 9118"
                    (0x6B0B5E00)
3LKDEADLOCKWTR    which is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x666948D0 infl_mon_t: 0x66694910:
4LKDEADLOCKOBJ      java/util/ArrayList@45FC38F0/45FC38FC:
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "Non-deferrable Alarm : 0" (0x594D6200)

The "Failed to fork OS thread" indicates that this is a NATIVE
OutOfMemory that occurs when no more threads can be created in
the address space.  This happens because the deadlock condition
creates a bottleneck.

This specific deadlock occurs when one of the threads has
encountered an FFDC incident.

The two deadlocked stacktraces are as follows:

3XMTHREADINFO      "Non-deferrable Alarm : 0" (TID:0x594D6200,
sys_thread_t:0x594D1238, state:B, native ID:0x3ECF0DE0) prio=5
4XESTACKTRACE          at
com/ibm/ws/management/event/NotificationService.getControlJVMLis
tener(NotificationService.java:554)
4XESTACKTRACE          at
com/ibm/ws/management/event/NotificationService.handleNotificati
on(NotificationService.java:286)
4XESTACKTRACE          at
com/ibm/ws/management/event/WsNotifDelegator.handleNotification(
WsNotifDelegator.java:56)
4XESTACKTRACE          at
com/ibm/ws/management/event/MBeanServerDelegateListener.handleNo
tification(MBeanServerDelegateListener.java:138)
4XESTACKTRACE          at
com/sun/jmx/interceptor/DefaultMBeanServerInterceptor$ListenerWr
apper.handleNotification(DefaultMBeanServerInterceptor.java:1670
(Compiled Code))
4XESTACKTRACE          at
javax/management/NotificationBroadcasterSupport.handleNotificati
on(NotificationBroadcasterSupport.java(Compiled Code))
4XESTACKTRACE          at
javax/management/NotificationBroadcasterSupport.sendNotification
(NotificationBroadcasterSupport.java:192(Compiled Code))
4XESTACKTRACE          at
javax/management/modelmbean/RequiredModelMBean.sendNotification(
RequiredModelMBean.java:2032)
4XESTACKTRACE          at
com/ibm/websphere/management/RuntimeCollaborator.sendNotificatio
n(RuntimeCollaborator.java:101)
4XESTACKTRACE          at
com/ibm/ws/runtime/mbean/RasLoggingServiceMBean.sendEvent(RasLog
gingServiceMBean.java:162)
4XESTACKTRACE          at
com/ibm/ws/runtime/mbean/RasLoggingServiceMBean.processEvent(Ras
LoggingServiceMBean.java:122)
4XESTACKTRACE          at
com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.jav
a:40(Compiled Code))
4XESTACKTRACE          at
java/util/logging/Logger.log(Logger.java:469(Compiled Code))
4XESTACKTRACE          at
com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1602)
4XESTACKTRACE          at
com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1530)
4XESTACKTRACE          at
com/ibm/ejs/ras/Tr.fireMessageEvent(Tr.java:1462)
4XESTACKTRACE          at
com/ibm/ejs/ras/Tr.warning(Tr.java:1154)
4XESTACKTRACE          at
com/ibm/ws/dcs/vri/common/nls/AbstractNLSEvent.invokeNLSTrace(Ab
stractNLSEvent.java:209)
4XESTACKTRACE          at
com/ibm/ws/dcs/vri/transportAdapter/rmmImpl/ptpDiscovery/Discove
ryServerMgr.alarm(DiscoveryServerMgr.java:1238)
4XESTACKTRACE          at
com/ibm/ws/dcs/utils/AlarmImpl$AlarmTask.execute(AlarmImpl.java:
103(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/dcs/utils/AlarmImpl$AlarmTask.executeWithLock(AlarmIm
pl.java(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/dcs/utils/AlarmImpl$AlarmTask.run(AlarmImpl.java:82(C
ompiled Code))
4XESTACKTRACE          at
com/ibm/ws/dcs/utils/AlarmImpl$AlarmTask.alarm(AlarmImpl.java:73
(Compiled Code))
4XESTACKTRACE          at
com/ibm/ejs/util/am/_Alarm.run(_Alarm.java:90(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1469)


3XMTHREADINFO      "SoapConnectorThreadPool : 9118"
(TID:0x6B0B5E00, sys_thread_t:0x6B038968,
                     state:B, native ID:0x3F22E5F0) prio=5
4XESTACKTRACE          at
com/ibm/ws/management/event/NotificationService.handleNotificati
on(NotificationService.java:284)
4XESTACKTRACE          at
com/ibm/ws/management/event/WsNotifDelegator.handleNotification(
WsNotifDelegator.java:56)
4XESTACKTRACE          at
com/ibm/ws/management/event/MBeanServerDelegateListener.handleNo
tification(MBeanServerDelegateListener.java:138)
4XESTACKTRACE          at
com/sun/jmx/interceptor/DefaultMBeanServerInterceptor$ListenerWr
apper.handleNotification(DefaultMBeanServerInterceptor.java:1670
(Compiled Code))
4XESTACKTRACE          at
javax/management/NotificationBroadcasterSupport.handleNotificati
on(NotificationBroadcasterSupport.java(Compiled Code))
4XESTACKTRACE          at
javax/management/NotificationBroadcasterSupport.sendNotification
(NotificationBroadcasterSupport.java:192(Compiled Code))
4XESTACKTRACE          at
javax/management/modelmbean/RequiredModelMBean.sendNotification(
RequiredModelMBean.java:2032)
4XESTACKTRACE          at
com/ibm/websphere/management/RuntimeCollaborator.sendNotificatio
n(RuntimeCollaborator.java:101)
4XESTACKTRACE          at
com/ibm/ws/runtime/mbean/RasLoggingServiceMBean.sendEvent(RasLog
gingServiceMBean.java:162)
4XESTACKTRACE          at
com/ibm/ws/runtime/mbean/RasLoggingServiceMBean.processEvent(Ras
LoggingServiceMBean.java:126)
4XESTACKTRACE          at
com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.jav
a:40(Compiled Code))
4XESTACKTRACE          at
java/util/logging/Logger.log(Logger.java:469(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/logging/WsLogger.deliverOrBuffer(WsLogger.java:240)
4XESTACKTRACE          at
com/ibm/ws/logging/WsLogger.log(WsLogger.java:217)
4XESTACKTRACE          at
com/ibm/ws/logging/WsLogger.logp(WsLogger.java:404)
4XESTACKTRACE          at
com/ibm/ws/ffdc/IncidentStreamImpl.initialize(IncidentStreamImpl
.java:720)
4XESTACKTRACE          at
com/ibm/ws/ffdc/IncidentStreamManager.getIncidentStream(Incident
StreamManager.java:80)
4XESTACKTRACE          at
com/ibm/ws/ffdc/DiagnosticEngine.processIncident(DiagnosticEngin
e.java:100)
4XESTACKTRACE          at
com/ibm/ws/ffdc/FFDCFilter.filterEngine(FFDCFilter.java:453)
4XESTACKTRACE          at
com/ibm/ws/ffdc/FFDCFilter.processException(FFDCFilter.java:201)
4XESTACKTRACE          at
com/ibm/ws/management/event/DownstreamServerManager$ServerUpdate
OpInfo.waitForRequestToComplete(DownstreamServerManager.java:484
)
4XESTACKTRACE          at
com/ibm/ws/management/event/DownstreamServerManager.updateServer
Listeners(DownstreamServerManager.java:316)
4XESTACKTRACE          at
com/ibm/ws/management/event/DownstreamServerManager.unsetFilter(
DownstreamServerManager.java:200)
4XESTACKTRACE          at
com/ibm/ws/management/event/NotificationService.removeNotificati
onListener(NotificationService.java:455)
4XESTACKTRACE          at
com/ibm/ws/management/event/NotificationService.replaceUpstreamS
erverSender(NotificationService.java:421)
4XESTACKTRACE          at
com/ibm/ws/management/event/NotificationService.addNotificationL
istener(NotificationService.java:175)
4XESTACKTRACE          at
com/ibm/ws/management/AdminServiceImpl.addNotificationListener(A
dminServiceImpl.java:1456)
4XESTACKTRACE          at
com/ibm/ws/management/connector/AdminServiceDelegator.addNotific
ationListener(AdminServiceDelegator.java:218)
4XESTACKTRACE          at
sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE          at
sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessor
Impl.java:79(Compiled Code))
4XESTACKTRACE          at
sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
AccessorImpl.java:43(Compiled Code))
4XESTACKTRACE          at
java/lang/reflect/Method.invoke(Method.java:603(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/management/connector/soap/SOAPConnector.invoke(SOAPCo
nnector.java:351(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/management/connector/soap/SOAPConnector.service(SOAPC
onnector.java:205(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/management/connector/soap/SOAPConnection.handleReques
t(SOAPConnection.java:55(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/http/HttpConnection.readAndHandleRequest(HttpConnecti
on.java:680(Compiled Code))
4XESTACKTRACE          at
com/ibm/ws/http/HttpConnection.run(HttpConnection.java:484(Compi
led Code))
4XESTACKTRACE          at
com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1469(Compi
led Code))
****************************************************************

Local fix

****************************************************************
* RECOMMENDATION:  There is no local fix/workaround for this   *
* problem.  Please acquire the APAR as appropriate.            *
*                                                              *
*                                                              *
****************************************************************

Problem summary

****************************************************************
* USERS AFFECTED:  All users of WebSphere Application Server   *
*                  V6.1.0 for z/OS                             *
****************************************************************
* PROBLEM DESCRIPTION: In certain circumstances, the           *
*                      nodeagent can lock up and log an out    *
*                      of memory error, due to a deadlock in   *
*                      the code.                               *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
The problem is due to a deadlock situation in the
notification services code. Under certain situations, two
different threads each end up waiting for a resource that the
other thread has. This can lead to a java OutOfMemoryError in
the node agent.

In one case where this error happened, the nodeagent log
showed the following messages:

"An unusual connectivity state occured with member
cell\xxnod1e\xxnod1e, details: alarm(): Closing the connection
because members did not manage to connect."

"BBOO0011W The function
ZIOPChannelBridge::outbound_connect(ORB_Request *, BOSS_IOR
*)+713 received CORBA system exception CORBA::COMM_FAILURE.
Error code is C9C26A16."

An analysis of the resulting javacore file clearly showed a
deadlock, with both deadlocked threads stuck in
NotificationServices.java

Problem conclusion

The problem was resolved by changing a synchronization
statement, so it is no longer possible in notification
services code for separate threads to lock two different
objects in different order.

APAR PK76104 is currently targeted for inclusion in Service
Level (Fix Pack) 6.1.0.25 of WebSphere Application Server V6.1
for z/OS.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PK76104
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2008-11-20
Closed date
2009-01-27
Last modified date
2009-07-01

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PK90230

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R610 PSY UK47214
UP09/06/22 P F906

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
29 December 2021

Tips

PK76104: DEADLOCK DETECTED IN NOTIFICATIONSERVICE.HANDLENOTIFICATION WITH "JAVA/LANG/OUTOFMEMORYERROR":"FAILED TO FORK OS THREAD"

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R610 PSY UK47214

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?