PI71240: SIB MESSAGE DEADLOCK AFTER SERVANT IS RESTARED DUE TO ABENDEC3

Fixes are available

APAR status

Closed as program error.

Error description

A hang issue happened causing the servant region to abend with
an EC3 and cause a new servant region to restart (normal
behavior after an EC3 timeout). After the problem that was
causing the hang was solved, the servant region didn't
processing any work.  This servant runs an application  which
is an MDB that consumes messages from a SIB queue. This SIB
queue is hosted in a message engine that was running in this
same application server (in the
ADJUNCT  region).

Javacores show a deadlock:

Deadlock detected !!!
---------------------
Thread "JFAP Proxy Event Queue Listener" (0x00000000264FB400)
  is waiting for:

java/util/concurrent/locks/ReentrantLock$NonfairSync@0x000000008
2FE8358
  which is owned by:
Thread "Consumer w0sr01.000-PDC_BX : 221746"
(0x00000000266A1100)
  which is waiting for:
    sys_mon_t:0x00000048402FE098 infl_mon_t: 0x00000048402FE110:

com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration@0x00000
00082FE8
3F8
  which is owned by:
Thread "JFAP Proxy Event Queue Listener" (0x00000000264FB400)

These two threads have this traceback:

"JFAP Proxy Event Queue Listener" J9VMThread:0x00000000264FB400,
j9thread_t:0x0000004809C2C170,
java/lang/Thread:0x0000000082DA70E8,
state:P, prio=5
(java/lang/Thread getId:0x8B, isDaemon:true)
(native thread ID:0x21DD5800, native priority:0x5, native
policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001)
CPU usage total: 4538.684000000 secs
Parked on:
java/util/concurrent/locks/ReentrantLock$NonfairSync@0x000000008
2FE8358
Owned by: "Consumer w0sr01.000-PDC_BX : 221746"
(J9VMThread:0x00000000266A1100,
java/lang/Thread:0x00000000841D2618)
Heap bytes allocated since last GC cycle=0 (0x0)
Java callstack:
  at sun/misc/Unsafe.park(Native Method)
  at java/util/concurrent/locks/LockSupport.
park(LockSupport.java:182(Compiled Code))
  at java/util/concurrent/locks/AbstractQueuedSynchronizer.
parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822(Compil
ed Code))
  atjava/util/concurrent/locks/AbstractQueuedSynchronizer.
acquireQueued(AbstractQueuedSynchronizer.java:853(Compiled
Code))
  at java/util/concurrent/locks/AbstractQueuedSynchronizer.
acquire(AbstractQueuedSynchronizer.java:1189(Compiled Code))
  at java/util/concurrent/locks/ReentrantLock$NonfairSync.
lock(ReentrantLock.java:197(Compiled Code))
  at java/util/concurrent/locks/ReentrantLock.
lock(ReentrantLock.java:273(Compiled Code))
  at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
put(JSLocalConsumerPoint.java:753(Compiled Code))
  at com/ibm/ws/sib/processor/impl/ConsumerDispatcher.
giveMessageToConsumer(ConsumerDispatcher.java:855(Compiled
Code))
<truncated stack trace >

"Consumer w0sr01.000-PDC_BX : 221746"
J9VMThread:0x00000000266A1100,
j9thread_t:0x0000004809D90550,
java/lang/Thread:0x00000000841D2618,
state:B, prio=5
java/lang/Thread getId:0x71F6C, isDaemon:true)
(native thread ID:0x21D4F800, native priority:0x5, native
policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201)
CPU usage total: 3.470000000 secs
Blocked on:
com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration@0x00000
00082FE8
3F8 Owned by: "JFAP Proxy Event Queue Listener"
(J9VMThread:0x00000000264FB400,
java/lang/Thread:0x0000000082DA70E8)
Heap bytes allocated since last GC cycle=0 (0x0)
Java callstack:
  at
com/ibm/ws/sib/processor/impl/AbstractLockedMessageEnumeration.
addNewMessage(AbstractLockedMessageEnumeration.java:470(Compiled
Code))
  at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
lockMessages(JSLocalConsumerPoint.java:2966(Compiled Code))
  at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
processQueuedMsgs(JSL ocalConsumerPoint.java:2652(Compiled
Code))
  at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
runAsynchConsumer(JSLocalConsumerPoint.java:3318(Compiled Code))
    (entered lock: com/ibm/ws/sib/processor/impl/
AsynchConsumer@0x0000000082FE83D8, entry count: 1)
  at
com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint$AsynchThread
.run(JSLocalConsumerPoint.java:3637(Compiled Code))
  at com/ibm/ws/util/ThreadPool$Worker.
run(ThreadPool.java:1881(CompiledCode))

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V8.0, V8.5, and V9.0                 *
****************************************************************
* PROBLEM DESCRIPTION: Deadlock in                             *
*                      com/ibm/ws/sib/processor/impl/JSLockedM *
*                      es                                      *
*                      sageEnumeration                         *
****************************************************************
* RECOMMENDATION:  Very rare, if seen apply the fix and        *
*                  restart                                     *
*                  the appserver.                              *
****************************************************************
Deadlock in
com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration
A dump needs to be taken and the thread stacks analyzed to
determne of the deadlock is the same.

Problem conclusion

The fix is to correctly apply the locking hierarchy in
jsLocalConsumerPoint.CleanupBifurcatedConsumer().

APAR PI71240 is currently targeted for inclusion in
Fix Packs  8.0.0.13, 8.5.5.12 and 9.0.0.3. of WebSphere
Application Server.

Please refer to the Recommended Updates page for delivery
information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

In addition, please refer to URL:
http://www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack PTF information.

Temporary fix

Comments

APAR Information

APAR number
PI71240
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-10-25
Closed date
2017-01-10
Last modified date
2017-01-10

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R850 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
04 May 2022

Tips

PI71240: SIB MESSAGE DEADLOCK AFTER SERVANT IS RESTARED DUE TO ABENDEC3

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R850 PSY

Document Information

Share your feedback

Need support?