PK68367: MESSAGES MAY REMAIN IN LOCKED STATE AFTER DATABASE TRANSACTION LOGS BECOME FULL. A MESSAGING ENGINE RESTART IS REQUIRED.

Fixes are available

APAR status

Closed as program error.

Error description

When a system is under load and the transaction logs of the
database used for the datastore of the messaging engine become
full, messages may become stuck in LOCKED status on queues.

The following error is seen in the JVM logs of the application
server hosting the messaging engine, for each message which
becomes stuck in LOCKED status:
CWSIP0002E: An internal messaging error occurred
   in com.ibm.ws.sib.processor.impl.store.items.
   MessageItem.registerMessageEventListener,
   1:1561:1.210.1.2, 10

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider for *
*                  WebSphere Application Server V6             *
****************************************************************
* PROBLEM DESCRIPTION: When the transaction logs for the       *
*                      database providing the message store    *
*                      for a messaging engine become full,     *
*                      because the system is under load,       *
*                      operations may fail and the             *
*                      maximum failed deliveries may be        *
*                      reached for MDBs consuming messages.    *
*                                                              *
*                      This causes the messaging engine to     *
*                      attempt to move the message to the      *
*                      exception destination. However, this    *
*                      may also fail due to the database       *
*                      transaction logs being full. If this    *
*                      fails with the exception types & stacks *
*                      listed in this APAR, the logic          *
*                      to retry this operation is bypassed     *
*                      and the message may remain in LOCKED    *
*                      state on the queue even after           *
*                      database transaction log space          *
*                      becomes available again.                *
*                                                              *
*                      The messaging engine requires a         *
*                      restart for these messages to be        *
*                      correct processed.                      *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
One of the following two summarized exception stacks would be
seen in FFDCs on the server hosting the messaging engine if
this issue occurs:

com.ibm.ws.sib.processor.exceptions.SIMPErrorException:
 com.ibm.websphere.sib.exception.SIErrorException:
   CWSIP0002E: An internal messaging error occurred in
   com.ibm.ws.sib.processor.impl.store.items.MessageItem.
   registerMessageEventListener, 1:1548:1.210.1.2, 10
 at com.ibm.ws.sib.processor.impl.PtoPInputHandler.
                      handleMessage(PtoPInputHandler.java:633)
 at com.ibm.ws.sib.processor.impl.
    ExceptionDestinationHandlerImpl.handleUndeliverableMessage
                    (ExceptionDestinationHandlerImpl.java:734)
 at com.ibm.ws.sib.processor.impl.
    ExceptionDestinationHandlerImpl.handleUndeliverableMessage
                    (ExceptionDestinationHandlerImpl.java:382)
 at com.ibm.ws.sib.processor.impl.BaseDestinationHandler.
  handleUndeliverableMessage(BaseDestinationHandler.java:1902)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher.
                eventPreUnlocked(ConsumerDispatcher.java:2368)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher.
            messageEventOccurred(ConsumerDispatcher.java:2157)
 at com.ibm.ws.sib.processor.impl.store.items.MessageItem.
                              unlockMsg(MessageItem.java:2300)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher.
         eventPostRollbackRemove(ConsumerDispatcher.java:2298)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher.
            messageEventOccurred(ConsumerDispatcher.java:2153)
 at com.ibm.ws.sib.processor.impl.store.items.
    MessageItem.eventPostRollbackRemove(MessageItem.java:1107)
 at com.ibm.ws.sib.msgstore.task.
     AbstractRemoveTask.postAbort(AbstractRemoveTask.java:214)
 at com.ibm.ws.sib.msgstore.task.TaskList.
                               postComplete(TaskList.java:386)
 at com.ibm.ws.sib.msgstore.transactions.XidParticipant.
                             rollback(XidParticipant.java:700)
 at com.ibm.ws.sib.msgstore.transactions.XidManager.
                                  rollback(XidManager.java:323)
 at com.ibm.ws.sib.msgstore.transactions.
                                MSDelegatingXAResource.rollback
                              (MSDelegatingXAResource.java:615)
 at com.ibm.ws.sib.comms.server.clientsupport.
                           StaticCATXATransaction.rcvXARollback
                              (StaticCATXATransaction.java:984)



com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E:
  The data source has produced an unexpected exception:
  com.ibm.db2.jcc.b.SqlException: DB2 SQL error:
  SQLCODE: -964, SQLSTATE: 57011, SQLERRMC: null
 at com.ibm.ws.sib.msgstore.persistence.impl.
                          PersistentMessageStoreImpl.updateEntry
                          (PersistentMessageStoreImpl.java:1310)
 at com.ibm.ws.sib.msgstore.persistence.impl.
                           UniqueKeyGeneratorImpl.getUniqueValue
                               (UniqueKeyGeneratorImpl.java:150)
 at com.ibm.ws.sib.msgstore.impl.MessageStoreImpl.
                       getUniqueValue(MessageStoreImpl.java:858)
 at com.ibm.ws.sib.msgstore.cache.links.ItemStreamLink.addItem
                                       (ItemStreamLink.java:282)
 at com.ibm.ws.sib.msgstore.ItemStream.addItem
                                           (ItemStream.java:105)
 at com.ibm.ws.sib.msgstore.ItemStream.addItem
                                           (ItemStream.java:136)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher.
                       storeMessage(ConsumerDispatcher.java:656)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher.
                       internalPut(ConsumerDispatcher.java:1290)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher.put
                                   (ConsumerDispatcher.java:843)
 at com.ibm.ws.sib.processor.impl.PtoPInputHandler.
                        handleMessage(PtoPInputHandler.java:560)
 at com.ibm.ws.sib.processor.impl.
      ExceptionDestinationHandlerImpl.handleUndeliverableMessage
                      (ExceptionDestinationHandlerImpl.java:734)
 at com.ibm.ws.sib.processor.impl.
      ExceptionDestinationHandlerImpl.handleUndeliverableMessage
                      (ExceptionDestinationHandlerImpl.java:382)
 at com.ibm.ws.sib.processor.impl.BaseDestinationHandler.
    handleUndeliverableMessage(BaseDestinationHandler.java:1902)
 at com.ibm.ws.sib.processor.impl.ConsumerDispatcher$
                               ExceptionDestinationRetryHandler.
                             alarm(ConsumerDispatcher.java:3162)

Problem conclusion

This APAR resolves the problem, by correcting the handling of
these exceptions so messages will be moved to the exception
destination when database transaction log space becomes
available, rather than remaining stuck in LOCKED state until a
restart of the messaging engine.

The fix for this APAR is currently targeted for inclusion in
fixpack 6.1.0.21.  Please refer to the Recommended Updates
page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PK68367
Reported component name
PLAT MSG COM
Reported component ID
620600101
Reported release
200
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2008-06-30
Closed date
2008-08-22
Last modified date
2008-09-25

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
PLAT MSG COM
Fixed component ID
620600101

Applicable component levels

R200 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 December 2021

Tips

PK68367: MESSAGES MAY REMAIN IN LOCKED STATE AFTER DATABASE TRANSACTION LOGS BECOME FULL. A MESSAGING ENGINE RESTART IS REQUIRED.

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R200 PSY

Document Information

Share your feedback

Need support?