PK91465: Deadlock in JMSConnectionHandle.markManagedConnectionAsStale

Fixes are available

APAR status

Closed as program error.

Error description

A deadlock was seen when a JMSException was thrown by
com/ibm/ejs/jms/JMSSessionHandle.enlist when an MDB was using
a listener port to process messages from a WebSphere MQ queue.
A javacore of the server experiencing the deadlock showed a
thread from the MessageListenerThreadPool with this stack trace:
"MessageListenerThreadPool : 2"
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:199(Compiled Code))
    at com/ibm/mq/jms/MQQueueAgentThread.waitForEnd
     (MQQueueAgentThread.java:293)
    at com/ibm/mq/jms/MQQueueAgent.removeBrowser
     (MQQueueAgent.java:1285)
    at com/ibm/mq/jms/MQConnectionBrowser.close
     (MQConnectionBrowser.java:607)
    at com/ibm/mq/jms/MQConnectionConsumer.close
     (MQConnectionConsumer.java:464)
    at com/ibm/ejs/jms/listener/MDBListenerImpl.internalStop
     (MDBListenerImpl.java:910)
    at com/ibm/ejs/jms/listener/JMSExceptionListener.onException
     (JMSExceptionListener.java:118)
    at com/ibm/ejs/jms/JMSManagedConnection.onException
     (JMSManagedConnection.java:1262)
    at com/ibm/ejs/jms/JMSManagedConnection.cleanup
     (JMSManagedConnection.java:955(Compiled Code))
    at com/ibm/ejs/j2c/MCWrapper.cleanup(MCWrapper.java:1455
     (Compiled Code))
    at com/ibm/ejs/j2c/FreePool.cleanupAndDestroyMCWrapper
     (FreePool.java:703(Compiled Code))
    at com/ibm/ejs/j2c/FreePool.returnToFreePool
     (FreePool.java:476(Compiled Code))
    at com/ibm/ejs/j2c/PoolManager.release
     (PoolManager.java:1780(Compiled Code))
    at com/ibm/ejs/j2c/MCWrapper.releaseToPoolManager
     (MCWrapper.java:2287(Compiled Code))
    at com/ibm/ejs/j2c/MCWrapper.connectionErrorOccurred
     (MCWrapper.java:2400)
    at com/ibm/ejs/j2c/ConnectionEventListener.
     connectionErrorOccurred(ConnectionEventListener.java:447)
    at com/ibm/ejs/jms/JMSManagedConnection.
     connectionErrorOccurred(JMSManagedConnection.java:1364)
    at com/ibm/ejs/jms/JMSConnectionHandle.
     markManagedConnectionAsStale(JMSConnectionHandle.java:876)
    at com/ibm/ejs/jms/JMSSessionHandle.
     markManagedObjectsAsStale(JMSSessionHandle.java:1445)
    at com/ibm/ejs/jms/JMSSessionHandle.enlist
     (JMSSessionHandle.java:1009(Compiled Code))
    at com/ibm/ejs/jms/listener/ServerSession.onMessage
     (ServerSession.java:651(Compiled Code))
    ...

While all the other threads from the MessageListenerThreadPool
had stack traces like this:
"MessageListenerThreadPool : 1"
    at com/ibm/ejs/jms/JMSConnectionHandle.
     markManagedConnectionAsStale(JMSConnectionHandle.java:872)
    at com/ibm/ejs/jms/JMSSessionHandle.
     markManagedObjectsAsStale(JMSSessionHandle.java:1445)
    at com/ibm/ejs/jms/JMSSessionHandle.enlist
     (JMSSessionHandle.java:1009(Compiled Code))
    at com/ibm/ejs/jms/listener/ServerSession.onMessage
     (ServerSession.java:651(Compiled Code))
    ...

And the stack trace for the MQQueueAgent thread was this:
"MQQueueAgent ..."
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:199(Compiled Code))
    at com/ibm/ejs/jms/listener/ServerSessionPool.
     getServerSession(ServerSessionPool.java:348(Compiled Code))
    at com/ibm/mq/jms/MQConnectionConsumer$
     MQConnectionConsumerMRH.deliverNow
     (MQConnectionConsumer.java:207 (Compiled Code))
    at com/ibm/mq/jms/MQConnectionConsumer$
     MQConnectionConsumerMRH.endDeliver
     (MQConnectionConsumer.java:114 (Compiled Code))
    at com/ibm/mq/jms/MQConnectionConsumer$
     MQConnectionConsumerMRH.handleMessageReference
     (MQConnectionConsumer.java:130 (Compiled Code))
    at com/ibm/mq/jms/MQConnectionBrowser.deliver
     (MQConnectionBrowser.java:701(Compiled Code))
    at com/ibm/mq/jms/MQQueueAgentThread.deliverMsgRef
     (MQQueueAgentThread.java:909(Compiled Code))
    at com/ibm/mq/jms/MQQueueAgentThread.processMessage
     (MQQueueAgentThread.java:701(Compiled Code))
    at com/ibm/mq/jms/MQQueueAgentThread.run
     (MQQueueAgentThread.java:1761(Compiled Code))
    ...

Local fix

No workaround is currently available, though in WebSphere
Application Server 7.0, users are recommended to replace their
WebSphere MQ Listener Ports with Activation Specifications.

Problem summary

****************************************************************
* USERS AFFECTED:  Users of IBM WebSphere Application Server   *
*                  Version 6.0, Version 6.1 or Version 7.0     *
*                  using WebSphere MQ as a messaging provider  *
****************************************************************
* PROBLEM DESCRIPTION: Listener port stops consuming after a   *
*                      messaging provider throws an            *
*                      exception in transaction                *
*                      enlistment of a server session          *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
A javacore of the server shows a thread from the
MessageListenerThreadPool waiting at:
com/ibm/mq/jms/MQQueueAgentThread.waitForEnd

While all of the other threads from the
MessageListenerThreadPool are in:
com/ibm/ejs/jms/JMSConnectionHandle.markManagedConnectionAsStale

And the MQQueueAgent thread is waiting at:
com/ibm/ejs/jms/listener/ServerSessionPool.getServerSession

WSVR0605W log entries are written to the JVM logs stating that
threads in theMessageListenerThreadPool pool may have hung.

The deadlock blocks all threads attempting further JMS
messaging, and might prevent the application server from
shuting down.

This problem only occurs if there are the same number or more
messages on the queue that is being consumed from than the
Maximum sessions setting on the listener port.

Problem conclusion

The fix for this APAR ensures that only one message listener
thread attempts to clean up the managed connection so that the
remaining message listener threads can process messages,
allowing the MQQueueAgent thread to shutdown and thus
preventing the deadlock from occurring.

The fix for this APAR is currently targeted for inclusion in
fix pack 6.0.2.41, 6.1.0.29 and 7.0.0.7.  Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PK91465
Reported component name
PLAT MSG COM
Reported component ID
620600101
Reported release
200
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2009-07-15
Closed date
2009-07-31
Last modified date
2010-03-10

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
PLAT MSG COM
Fixed component ID
620600101

Applicable component levels

R200 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
29 December 2021

Tips

PK91465: Deadlock in JMSConnectionHandle.markManagedConnectionAsStale

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R200 PSY

Document Information

Share your feedback

Need support?