PK83346: SOME MESSAGES SENT TO A DESTINATION IN A TRANSACTION ARE LOST, AFTER AN INDIVIDUAL MDB APPLICATION HAS BEEN RESTARTED

Fixes are available

APAR status

Closed as program error.

Error description

This issue affects JMS message-driven beans running on a
server that does not host a messaging engine, and where the
application consumes messages from one destination in a bus
and sends messages to another destination in the same bus.

The issue only occurs after an individual application
containing message-driven beans is stopped, while that
application is actively consuming messages.

It does not occur as a result of stopping an entire
application server, or as a result of stopping individual
applications on an idle system.

It does not occur for applications that send messages to a
destination in a different bus to the bus from which
the messages were consumed, or applications that connect to
a different messaging engine in the same bus to send messages.

The following FFDC report is seen on the server hosting the
message-driven bean application:
Exception = javax.transaction.xa.XAException
Source = com.ibm.ejs.j2c.XATransactionWrapper.end
probeid = 417
Stack Dump = javax.transaction.xa.XAException at
com.ibm.ws.sib.comms.client.OptimizedSIXAResourceProxy.end
(OptimizedSIXAResourceProxy.java:589)
...

The following FFDC report is seen on the server hosting the
messaging engine to which the message-driven bean application
connects:
Exception = com.ibm.websphere.sib.exception.SIErrorException
Source = com.ibm.ws.sib.comms.server.clientsupport.
ServerTransportReceiveListener.dataReceived
probeid = 3-004-0013
Stack Dump = com.ibm.websphere.sib.exception.SIErrorException:
CWSIC2059E: An internal error occurred. An attempt was made to
get an object from the map with id 1, but there is no item in
the map with this key.
at com.ibm.ws.sib.comms.server.IdToObjectMap.get
(IdToObjectMap.java:144)
at com.ibm.ws.sib.comms.server.IdToTransactionTable.get
(IdToTransactionTable.java:264)
at com.ibm.ws.sib.comms.server.IdToTransactionTable.get
(IdToTransactionTable.java:320)
at com.ibm.ws.sib.comms.server.clientsupport.
StaticCATProducer.sendSessMsg(StaticCATProducer.java:521)
at com.ibm.ws.sib.comms.server.clientsupport.
StaticCATProducer.rcvSendSessMsgNoReply
(StaticCATProducer.java:444)

Local fix

Change the code so the transaction can successfuly disassociate
itself from a connection to the messaging engine.

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider     *
*                  for IBM WebSphere Application Server        *
*                  Version 6.1 and Version 7.0                 *
****************************************************************
* PROBLEM DESCRIPTION: An FFDC containing the following        *
*                      exception is logged on the server       *
*                      hosting a messaging engine:             *
*                      com.ibm.websphere.sib.exception.        *
*                      SIErrorException:                       *
*                      CWSIC2059E: An internal error           *
*                      occurred. An attempt was made to        *
*                      get an object from the map with id      *
*                      <POSITIVE_INTEGER>, but there is no     *
*                      item in the map with this key.          *
*                                                              *
*                      The stack within the exception          *
*                      contains the following method:          *
*                      com.ibm.ws.sib.comms.server.            *
*                      clientsupport.StaticCATProducer.        *
*                      sendSessMsg()                           *
*                                                              *
*                      For each FFDC reported containing       *
*                      this exception, a message might         *
*                      have been lost.                         *
*                                                              *
*                      The sending application (running in a   *
*                      remote server) does not receive any     *
*                      exception to inform it of the issue,    *
*                      and the other actions performed         *
*                      within the same transaction complete    *
*                      successfully.                           *
*                                                              *
*                      The issue only occurs after the         *
*                      endpoint for a JMS message-driven bean  *
*                      (MDB) is stopped while it is actively   *
*                      consuming messages, and that MDB        *
*                      sends messages to the same bus it       *
*                      consumes messages from.                 *
*                                                              *
*                      Only the server on which the            *
*                      endpoint was stopped is affected.       *
*                      However, any application that sends     *
*                      messages to the bus using the           *
*                      same connection factory as the          *
*                      MDB code could be affected in that      *
*                      server once the issue has occurred.     *
*                                                              *
*                      The issue does not occur after a        *
*                      whole application server is restarted.  *
****************************************************************
* RECOMMENDATION:  Review the FFDC logs of servers hosting     *
*                  messaging engines for instances of FFDCs    *
*                  containing CWSIC2059E exceptions.           *
*                                                              *
*                  If FFDC reports containing CWSIC2059E       *
*                  exceptions are found, restart all servers   *
*                  in the cell.                                *
*                                                              *
*                  Follow the advice given in the              *
*                  circumvention section below to avoid any    *
*                  additional exposure to the issue.           *
****************************************************************
The stopping of the endpoint for a JMS MDB (connected to a
messaging engine via a network) might cause a JMS Session to
be left in a pool in an invalid state - associated with
a transaction that no longer exists.
The exact circumstances in which can occur are described below.

Once an invalid session is in the pool, applications might
obtain the session from the pool and attempt to use it to
send messages.
As the session is associated with a nonexistent transaction,
sends using the session fail. CWSIC2059E exceptions
are logged by the messaging engine each time a send fails.

However, sending of messages over a network to a messaging
engine within a transaction is a one-way operation in
WebSphere Application Server V6.1 and V7.0.
If the send fails, the messaging engine marks the transaction
as rollback-only and the error is reported during transaction
completion.
As in this case the messaging engine cannot find a transaction
to mark rollback-only, it has no mechanism by which to
inform the application that the problem has occurred.
This results in the application's transaction completing, even
though the send of a message that was intended to be part of
that transaction failed - so the message is lost.

The following sequence of events cause an invalid session to
enter the pool:
- A message is delivered to a JMS message-driven bean under a
  transaction, using a JMS activation specification (within
  the Service Integration Bus resource adapter).
- The onMessage method of the message-driven bean gets a JMS
  session from the pool (connected to the same bus as the
  message was consumed from) to send a message.
- The transaction manager enlists the XAResource associated
  with the JMS Session into the global transaction.
- An optimization is used, because the same network
  connection is shared by the JMS Session and the connection
  used to consume the message.
  This optimization causes the XAResource of the JMS Session to
  join the existing transaction branch of the global
  transaction, rather than a new transaction branch being
  created.
- The endpoint for the message-driven bean is stopped, causing
  cleanup of the connection used by the resource adapter to
  consume the message.
- The cleanup code incorrectly discards some state required
  by the XAResource of the JMS Session that was joined
  to the same transaction branch.
- The transaction for the message being consumed when the
  message-driven bean is rolled back, which should cause the
  session to be dissociated from the transaction.
  However, an exception occurs (an XAException with XAER_PROTO
  thrown by OptimizedSIXAResourceProxy.end) due to the invalid
  cleanup that occurred when the endpoint was stopped.
  This prevents the XAResource of the session from being
  dissociated from the transaction. However, the session
  returns to the pool for re-used by other threads -
  resulting in the CWSIC2059E exceptions on the messaging
  engine, and the loss of messages.

Problem conclusion

The fix for this APAR corrects the cleanup code that is
invoked when an MDB endpoint is stopped. This prevents it from
discarding state required by JMS sessions joined to transaction
branches of in-flight work for the MDB.

The fix for this APAR is currently targeted for inclusion in
fix pack 6.1.0.27 and 7.0.0.5.  Please refer to the Recommended
Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PK83346
Reported component name
PLAT MSG COM
Reported component ID
620600101
Reported release
200
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2009-03-25
Closed date
2009-04-20
Last modified date
2010-06-01

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
PLAT MSG COM
Fixed component ID
620600101

Applicable component levels

R200 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
29 December 2021

Tips

PK83346: SOME MESSAGES SENT TO A DESTINATION IN A TRANSACTION ARE LOST, AFTER AN INDIVIDUAL MDB APPLICATION HAS BEEN RESTARTED

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R200 PSY

Document Information

Share your feedback

Need support?