PM02163: HIGH NUMBER OF INVALIDATION REQUESTS CAN CAUSE A DEADLOCK IN THE SERVER.

Fixes are available

APAR status

Closed as program error.

Error description

When a high number of invalidation requests are being processed
in a multi-Servant environment, a deadlock can occur which
causes the server to become unresponsive and hang.
.
The customer was running a Portal server with 5 Servants. Portal
pages were being accessed which also cause invalidations to flow
.

When an item is invalidated in the cache, the invalidation
message is sent from the originating SR to the CR, which proxies
it to ALL the SRs (excluding the origin SR) synchronously.
If there are 5 servants, the flow is ... SR1 --> CR --> (SR2,
SR3, SR4, SR5). If the call from one of the target SRs does not
return, the original SR1 call is also hung.
.
ORB threads are used in the SR to send invalidations to the CR,
as well as receive the proxied calls from other servants.
If there is a lot of activity, the ORB thread used by the CR to
proxy the invalidations to the other SRs can seem to be hung
because the SR does not return. This can happen, if the
DRSServantProxy call gets stuck in the SR waiting on locks
owned by the other ORB threads sending invalidation requests to
the other servants.
.
The CR might show the majority of its threads trying to proxy
invalidation requests to the other servants:
com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest
Native Method
com/ibm/ws390/orb/ClientDelegate.invoke
(Compiled Code)
org/omg/CORBA/portable/ObjectImpl._invoke
ObjectImpl.java(Inlined Compiled Code)
com/ibm/ws/drs/ws390/proxy/servant/_DRSServantProxyStub.updateEn
tryProp             (Compiled Code)
com/ibm/ws/drs/ws390/DRSServantProxyWrapper.updateEntryProp
DRSServantProxyWrapper.java(Inlined Compiled Code)
com/ibm/ws/cache/drs/ws390/DynacacheDRSControllerDataXfer.update
EntryProp           DynacacheDRSControllerDataXfer.java(Compiled
Code)
com/ibm/ws/drs/ws390/proxy/controller/DRSControllerProxyHelper.u
pdateEntryProp      DRSControllerProxyHelper.java(Compiled Code)
com/ibm/ws/drs/ws390/proxy/controller/DRSControllerProxyImpl.upd
ateEntryProp        DRSControllerProxyImpl.java(Inlined Compiled
Code)
com/ibm/ws/drs/ws390/proxy/controller/_DRSControllerProxyImpl_Ti
e.updateEntryProp   _DRSControllerProxyImpl_Tie.java(Compiled
Code)
com/ibm/ws/drs/ws390/proxy/controller/_DRSControllerProxyImpl_Ti
e._invoke           _DRSControllerProxyImpl_Tie.java(Compiled
Code)
sun/reflect/GeneratedMethodAccessor8.invoke
(Compiled Code)
sun/reflect/DelegatingMethodAccessorImpl.invoke
DelegatingMethodAccessorImpl.java(Compiled Code)
java/lang/reflect/Method.invoke
Method.java(Compiled Code)
com/ibm/ws390/orb/RIRProtectedProxyHandler.invoke
(Compiled Code)
$Proxy0._invoke
(Compiled Code)
com/ibm/ws390/orb/CommonBridge.invoke
(Compiled Code)
com/ibm/ws390/orb/ORBEJSBridge.invoke
ORBEJSBridge.java(Inlined Compiled Code)
com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo
keParmSetter
.

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  IBM WebSphere Application Server V6.0.1     *
*                  users of dynacache with multiple servant    *
*                  regions.                                    *
****************************************************************
* PROBLEM DESCRIPTION: A deadlock occurs and the server        *
*                      hangs when a dynacache user requests    *
*                      a high number of invalidations and      *
*                      there are multiple servant regions.     *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Data Replication Services (DRS) uses proxy calls to flow
updates between the control region (CR) and servant regions
(SR).  These proxy calls run on Object Request Broker (ORB)
threads.  The SRs call back into the CR and a deadlock occurs
because all the ORB threads are in use and a thread is not
available to handle a proxied call from the SR.

Problem conclusion

Dynacache has been modified to batch invalidations on the SR
and immediately return so the ORB thread is freed up.  The
invalidations are then processed on a daemon thread at
periodic intervals.

APAR PM02163 is currently targeted for inclusion in Service
Level (Fix Pack) 6.0.2.41 of WebSphere Application Server
V6.0.1.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM02163
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
601
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2009-11-25
Closed date
2010-01-20
Last modified date
2010-05-04

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM02686 PM02782 PM02852 PM02854

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R601 PSY UK55832
UP10/04/16 P F004

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.0.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 December 2021

Tips

PM02163: HIGH NUMBER OF INVALIDATION REQUESTS CAN CAUSE A DEADLOCK IN THE SERVER.

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R601 PSY UK55832

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?